From huq2090 at gmail.com Sat Sep 1 15:34:01 2018 From: huq2090 at gmail.com (Fazlul Huq) Date: Sat, 1 Sep 2018 15:34:01 -0500 Subject: [petsc-users] Execution time of the code Message-ID: Hello PETSc Developers, I know that to know the execution time of code I have to run code with -log_view and I also read chapter 13 (Profiling) of the user-manual, but I can't figure out what is the total execution time of my code from the information printed on the screen. I got Time (sec) there but there exist sub-categories like Max, Max/Min, Avg and Total. My simple question (and may be stupid question!) is, what is the total execution time of the code here? (Following is the part of the information printed on the screen) jimmy at dave:~/projects/petsc/src/ksp/ksp/examples/tutorials$ mpiexec -n 4 ./ex10 -f0 medium -f1 arco6 -ksp_gmres_classicalgramschmidt -log_view -mat_type baij -matload_block_size 3 -pc_type bjacobi -options_left Number of iterations = 19 Residual norm 1.088292e-05 Number of iterations = 59 Residual norm 3.871022e-02 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./ex10 on a arch-linux2-c-debug named dave with 4 processors, by jimmy Sat Sep 1 13:41:14 2018 Using Petsc Release Version 3.9.2, unknown Max Max/Min Avg Total Time (sec): 6.196e-01 1.00002 6.196e-01 Objects: 1.060e+02 1.00000 1.060e+02 Flop: 2.361e+08 1.00684 2.353e+08 9.413e+08 Flop/sec: 3.811e+08 1.00682 3.798e+08 1.519e+09 MPI Messages: 2.360e+02 1.34857 2.061e+02 8.245e+02 MPI Message Lengths: 1.256e+07 2.24620 4.071e+04 3.357e+07 MPI Reductions: 2.160e+02 1.00000 Thanks. Sincerely, Huq -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Sep 1 16:02:49 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sat, 1 Sep 2018 21:02:49 +0000 Subject: [petsc-users] Execution time of the code In-Reply-To: References: Message-ID: <0AA0C26B-58A1-4015-9694-A1D3E68DDD57@anl.gov> 6.196e-01 But that is a relatively meaningless number. What is more important is the time in the various events of the code that is printed further down. Like SNESSolve(), KSPSolve(), ... Barry > On Sep 1, 2018, at 3:34 PM, Fazlul Huq wrote: > > Hello PETSc Developers, > > I know that to know the execution time of code I have to run code with -log_view and I also read chapter 13 (Profiling) of the user-manual, but I can't figure out what is the total execution time of my code from the information printed on the screen. I got Time (sec) there but there exist sub-categories like Max, Max/Min, Avg and Total. > My simple question (and may be stupid question!) is, what is the total execution time of the code here? > (Following is the part of the information printed on the screen) > > jimmy at dave:~/projects/petsc/src/ksp/ksp/examples/tutorials$ mpiexec -n 4 ./ex10 -f0 medium -f1 arco6 -ksp_gmres_classicalgramschmidt -log_view -mat_type baij -matload_block_size 3 -pc_type bjacobi -options_left > Number of iterations = 19 > Residual norm 1.088292e-05 > Number of iterations = 59 > Residual norm 3.871022e-02 > ************************************************************************************************************************ > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** > ************************************************************************************************************************ > > ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- > > ./ex10 on a arch-linux2-c-debug named dave with 4 processors, by jimmy Sat Sep 1 13:41:14 2018 > Using Petsc Release Version 3.9.2, unknown > > Max Max/Min Avg Total > Time (sec): 6.196e-01 1.00002 6.196e-01 > Objects: 1.060e+02 1.00000 1.060e+02 > Flop: 2.361e+08 1.00684 2.353e+08 9.413e+08 > Flop/sec: 3.811e+08 1.00682 3.798e+08 1.519e+09 > MPI Messages: 2.360e+02 1.34857 2.061e+02 8.245e+02 > MPI Message Lengths: 1.256e+07 2.24620 4.071e+04 3.357e+07 > MPI Reductions: 2.160e+02 1.00000 > > Thanks. > > Sincerely, > Huq From kenneth.c.hall at duke.edu Tue Sep 4 09:07:53 2018 From: kenneth.c.hall at duke.edu (Kenneth Hall, SC.D.) Date: Tue, 4 Sep 2018 14:07:53 +0000 Subject: [petsc-users] Back tracking/preconditioning with matrix free solvers Message-ID: Hi All, I am solving a nonlinear problem using a matrix free approach with Petsc. My code is written in Fortran90. I have read the documentation, and have had some success, but I am having problems with two features. 1. The code runs fine with no preconditioning. But when I using the -user_precond option, my preconditioner never gets called by Petsc, even though I make the appropriate calls to PCSetType and PCShellSetApply. 2. I am using the GMRES solver, which defaults to the L2-norm minimization line search (SNESLINESEARCHL2). I would like to use backtracking (SNESLINESEARCHBT). I have specified backtracking using SNESLineSearchSetType, but I then get the following error: [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: MatMFFDSetBase() has not been called, this is often caused by forgetting to call MatAssemblyBegin/End on the first Mat in the SNES compute function [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.3, Jul, 02, 2018 [0]PETSC ERROR: mustang on a arch-darwin-c-debug-double-real named Macintosh-11.local by hall Tue Sep 4 07:29:25 2018 [0]PETSC ERROR: Configure options --with-scalar-type=real [0]PETSC ERROR: #1 MatMult_MFFD() line 314 in /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/mat/impls/mffd/mffd.c [0]PETSC ERROR: #2 MatMult() line 2311 in /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/mat/interface/matrix.c [0]PETSC ERROR: #3 SNESLineSearchApply_BT() line 130 in /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/snes/linesearch/impls/bt/linesearchbt.c [0]PETSC ERROR: #4 SNESLineSearchApply() line 645 in /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/snes/linesearch/interface/linesearch.c [0]PETSC ERROR: #5 SNESSolve_NGMRES() line 262 in /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/snes/impls/ngmres/snesngmres.c [0]PETSC ERROR: #6 SNESSolve() line 4314 in /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/snes/interface/snes.c So it appears that Petsc is trying to do a matrix multiply. But this is a matrix free method. How do I fix this? The relevant code is attached below. Any help with these two issues is greatly appreciated. Code follows signature. I have omitted non-relavent portions. Kenneth Hall ============================================================= module petsc_krylov use petsc use data_module use solution_module implicit none #include #include private public :: gmres2 SNES :: snes ! nonlinear solver PC :: pc ! preconditioner context KSP :: ksp ! Krylov subspace method context SNESLineSearch :: linesearch Vec :: x,r ! solution, residual vectors PetscErrorCode :: ierr PetscInt :: n integer :: neq integer :: icount,level=0 contains . . . subroutine gmres2 implicit none ! !.... declare local variables PetscInt :: its ! iterations for convergence PetscMPIInt :: rank PetscBool :: snes_mf,snes_mf_set,user_precond ! !.... set problem size call get_equation_count n = neq ! !.... initialliize petsc call PetscInitialize(PETSC_NULL_CHARACTER,ierr) ; if (ierr .ne. 0) then ; print*,'Unable to initialize PETSc' ; endif ! !.... Create nonlinear solver context call SNESCreate(PETSC_COMM_WORLD,snes,ierr); CHKERRA(ierr) call SNESSetType(snes,SNESNGMRES,ierr); CHKERRA(ierr) call SNESSetTolerances(snes, 0.0, 0.0, 0.0, 20, 10000,ierr); CHKERRQ(ierr) ! !.... Create vectors for solution and nonlinear function call VecCreateSeq(PETSC_COMM_SELF,n,x,ierr); CHKERRA(ierr) call VecDuplicate(x,r,ierr); CHKERRA(ierr) ! !.... Set function evaluation routine and vector call SNESSetFunction(snes,r,FormFunction,0,ierr); CHKERRA(ierr) ! !.... Customize nonlinear solver; set runtime options snes_mf = PETSC_FALSE; call PetscOptionsGetBool(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,"-snes_mf",snes_mf,snes_mf_set,ierr); CHKERRA(ierr) if (snes_mf .eqv. PETSC_TRUE) then call SNESGetKSP(snes,ksp,ierr); CHKERRA(ierr) call KSPGetPC(ksp,pc,ierr); CHKERRA(ierr) call SNESGetLineSearch(snes,linesearch,ierr); CHKERRA(ierr) call SNESLineSearchSetType(linesearch,SNESLINESEARCHBT, ierr); CHKERRA(ierr) user_precond = PETSC_FALSE; call PetscOptionsHasName(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,"-user_precond",user_precond,ierr); CHKERRA(ierr) if (user_precond .eqv. PETSC_TRUE) then call PCSetType(pc,PCSHELL,ierr); CHKERRA(ierr) call PCShellSetApply(pc,MatrixFreePreconditioner,ierr); CHKERRA(ierr) else call PCSetType(pc,PCNONE,ierr); CHKERRA(ierr) end if end if ! !.... set snes options call SNESSetFromOptions(snes,ierr); CHKERRA(ierr) ! !.... initialize solution call load_x_from_u(x) ! !.... solve call SNESSolve(snes,PETSC_NULL_VEC,x,ierr) ! !.... Free work space. All PETSc objects should be destroyed when they are no longer needed. call VecDestroy(x,ierr) call VecDestroy(r,ierr) call SNESDestroy(snes,ierr) call PetscFinalize(ierr) end ! !==================================================================================================== !==================================================================================================== ! subroutine MatrixFreePreconditioner(x1,y1) implicit none ! !.... declare passed variables Vec :: x1,y1 . . . . end subroutine ! !==================================================================================================== !==================================================================================================== ! subroutine FormFunction(snes,x1,f1,dummy,ierr) implicit none ! !.... declare passed variables SNES :: snes Vec :: x1,f1,x2 PetscErrorCode :: ierr integer :: dummy(*) . . . end subroutine FormFunction end module From knepley at gmail.com Tue Sep 4 09:41:32 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 4 Sep 2018 10:41:32 -0400 Subject: [petsc-users] Back tracking/preconditioning with matrix free solvers In-Reply-To: References: Message-ID: On Tue, Sep 4, 2018 at 10:12 AM Kenneth Hall, SC.D. wrote: > Hi All, > > I am solving a nonlinear problem using a matrix free approach with Petsc. > My code is written in Fortran90. > I have read the documentation, and have had some success, but I am having > problems with two features. > > 1. The code runs fine with no preconditioning. But when I using the > -user_precond option, my preconditioner never gets called by Petsc, even > though I make the appropriate calls to PCSetType and PCShellSetApply. > Lets do this first. I think your PC choice is getting overwritten by options. Can you run with -snes_view and send the output when trying to use your MF PC? Thanks, Matt > 2. I am using the GMRES solver, which defaults to the L2-norm minimization > line search (SNESLINESEARCHL2). I would like to use backtracking > (SNESLINESEARCHBT). I have specified backtracking using > SNESLineSearchSetType, but I then get the following error: > > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: MatMFFDSetBase() has not been called, this is often caused > by forgetting to call > MatAssemblyBegin/End on the first Mat in the SNES compute > function > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.3, Jul, 02, 2018 > [0]PETSC ERROR: mustang on a arch-darwin-c-debug-double-real named > Macintosh-11.local by hall Tue Sep 4 07:29:25 2018 > [0]PETSC ERROR: Configure options --with-scalar-type=real > [0]PETSC ERROR: #1 MatMult_MFFD() line 314 in > /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/mat/impls/mffd/mffd.c > [0]PETSC ERROR: #2 MatMult() line 2311 in > /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/mat/interface/matrix.c > [0]PETSC ERROR: #3 SNESLineSearchApply_BT() line 130 in > /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/snes/linesearch/impls/bt/linesearchbt.c > [0]PETSC ERROR: #4 SNESLineSearchApply() line 645 in > /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/snes/linesearch/interface/linesearch.c > [0]PETSC ERROR: #5 SNESSolve_NGMRES() line 262 in > /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/snes/impls/ngmres/snesngmres.c > [0]PETSC ERROR: #6 SNESSolve() line 4314 in > /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/snes/interface/snes.c > > So it appears that Petsc is trying to do a matrix multiply. But this is a > matrix free method. How do I fix this? > > The relevant code is attached below. Any help with these two issues is > greatly appreciated. > > Code follows signature. I have omitted non-relavent portions. > Kenneth Hall > > > > ============================================================= > > > module petsc_krylov > use petsc > use data_module > use solution_module > implicit none > #include > #include > private > public :: gmres2 > SNES :: snes ! nonlinear solver > PC :: pc ! preconditioner > context > KSP :: ksp ! Krylov subspace > method context > SNESLineSearch :: linesearch > Vec :: x,r ! solution, residual > vectors > PetscErrorCode :: ierr > PetscInt :: n > integer :: neq > integer :: icount,level=0 > contains > > . > . > . > > subroutine gmres2 > implicit none > ! > !.... declare local variables > PetscInt :: its ! iterations for > convergence > PetscMPIInt :: rank > PetscBool :: snes_mf,snes_mf_set,user_precond > ! > !.... set problem size > call get_equation_count > n = neq > ! > !.... initialliize petsc > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) ; if (ierr .ne. 0) > then ; print*,'Unable to initialize PETSc' ; endif > ! > !.... Create nonlinear solver context > call SNESCreate(PETSC_COMM_WORLD,snes,ierr); CHKERRA(ierr) > call SNESSetType(snes,SNESNGMRES,ierr); CHKERRA(ierr) > call SNESSetTolerances(snes, 0.0, 0.0, 0.0, 20, 10000,ierr); > CHKERRQ(ierr) > ! > !.... Create vectors for solution and nonlinear function > call VecCreateSeq(PETSC_COMM_SELF,n,x,ierr); CHKERRA(ierr) > call VecDuplicate(x,r,ierr); CHKERRA(ierr) > ! > !.... Set function evaluation routine and vector > call SNESSetFunction(snes,r,FormFunction,0,ierr); CHKERRA(ierr) > ! > !.... Customize nonlinear solver; set runtime options > snes_mf = PETSC_FALSE; > call > PetscOptionsGetBool(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,"-snes_mf",snes_mf,snes_mf_set,ierr); > CHKERRA(ierr) > if (snes_mf .eqv. PETSC_TRUE) then > call SNESGetKSP(snes,ksp,ierr); CHKERRA(ierr) > call KSPGetPC(ksp,pc,ierr); CHKERRA(ierr) > call SNESGetLineSearch(snes,linesearch,ierr); CHKERRA(ierr) > call SNESLineSearchSetType(linesearch,SNESLINESEARCHBT, ierr); > CHKERRA(ierr) > user_precond = PETSC_FALSE; > call > PetscOptionsHasName(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,"-user_precond",user_precond,ierr); > CHKERRA(ierr) > if (user_precond .eqv. PETSC_TRUE) then > call PCSetType(pc,PCSHELL,ierr); CHKERRA(ierr) > call PCShellSetApply(pc,MatrixFreePreconditioner,ierr); > CHKERRA(ierr) > else > call PCSetType(pc,PCNONE,ierr); CHKERRA(ierr) > end if > end if > ! > !.... set snes options > call SNESSetFromOptions(snes,ierr); CHKERRA(ierr) > ! > !.... initialize solution > call load_x_from_u(x) > ! > !.... solve > call SNESSolve(snes,PETSC_NULL_VEC,x,ierr) > ! > !.... Free work space. All PETSc objects should be destroyed when they > are no longer needed. > call VecDestroy(x,ierr) > call VecDestroy(r,ierr) > call SNESDestroy(snes,ierr) > call PetscFinalize(ierr) > end > ! > > !==================================================================================================== > > !==================================================================================================== > ! > subroutine MatrixFreePreconditioner(x1,y1) > implicit none > ! > !.... declare passed variables > Vec :: x1,y1 . > . > . > . > > end subroutine > ! > > !==================================================================================================== > > !==================================================================================================== > ! > subroutine FormFunction(snes,x1,f1,dummy,ierr) > implicit none > ! > !.... declare passed variables > SNES :: snes > Vec :: x1,f1,x2 > PetscErrorCode :: ierr > integer :: dummy(*) > . > . > . > end subroutine FormFunction > > > > end module > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ling.Zou at inl.gov Tue Sep 4 09:55:19 2018 From: Ling.Zou at inl.gov (Ling Zou) Date: Tue, 4 Sep 2018 14:55:19 +0000 Subject: [petsc-users] Problem with SNES convergence In-Reply-To: <469723814.1317957.1535693197280@mail.yahoo.com> References: <1371398831.496454.1535553418412.ref@mail.yahoo.com> <1371398831.496454.1535553418412@mail.yahoo.com> <387994F6-79FF-4DE4-8375-B24F63B0F332@anl.gov> <1135178798.714456.1535603850997@mail.yahoo.com> <7337A99E-C3BE-48EF-BC95-4A1736227F8E@mcs.anl.gov>, <469723814.1317957.1535693197280@mail.yahoo.com> Message-ID: If it is indeed 'zero pivot' issue, you could add an extra option to your simulation '-pc_factor_shift_type NONZERO -pc_factor_shift_amount 1.0e-10', and see if it helps. http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot PETSc: Documentation: FAQ - mcs.anl.gov www.mcs.anl.gov Execution PETSc executables are SO big and take SO long to link. We find this annoying as well. On most machines PETSc can use shared libraries, so executables should be much smaller, run ./configure with the additional option --with-shared-libraries (this is the default). ________________________________ From: Rahul Samala Sent: Thursday, August 30, 2018 11:26:37 PM To: Fande Kong; Smith, Barry F.; Ling Zou Cc: PETSc users list Subject: Re: [petsc-users] Problem with SNES convergence Thank you Barry for the suggestion, I will definitely look in to TS implementation. I have implemented my own version of reduced time step size using the SNESGetConvergedReason, but encountered the same problem after another 30 time-steps. So, I am experimenting with different set of initial conditions. Ling, thank you for the code, I will see if my understanding is different from yours. -- Rahul. On Thursday, August 30, 2018, 11:25:35 PM GMT+5:30, Smith, Barry F. wrote: > On Aug 30, 2018, at 12:49 PM, Fande Kong > wrote: > > Hi Barry, > > I haven't had time to look into TS so far. But it is definitely interesting. One simple question would like this: If I have a simple loop for time steppers, and each step SNES is called. How hard to convert my code to use TS? You would need to look at the examples in ts/examples/tutorials but essentially you need to provide a slightly different version of FormFunction() and FormJacobian() and call TSSolve() etc instead of having your own loop. Barry > > Any suggestion? Where should I start from? > > > Fande, > > On Thu, Aug 30, 2018 at 11:29 AM Smith, Barry F. > wrote: > > Note also that the PETSc TS component has a large variety of timesteppers with automatic adaptivity which adjust the time-step for accuracy and convergence. Depending on the exact needs of your time-stepper it might be easier in the long run to use PETSc's time-steppers rather than write your own. > > Barry > > > > On Aug 30, 2018, at 10:46 AM, Ling Zou > wrote: > > > > Rahul, please see the logic I used to reduce time step size. > > Hope this helps you. > > > > Ling > > > > for (timestep = 1; timestep <= N; timestep++) // loop over time steps > > { > > // before trying to solve a time step, 1) it is still not successful; 2) we are not giving up; 3) haven't failed yet > > bool give_up = false; bool success = false; bool experienced_fail_this_time_step = false; > > // save solutions and old solutions into u_copy and u_old_copy, in case a time step fails and we restart from this saved solutions > > VecCopy(u, u_copy); VecCopy(u_old, u_old_copy); > > > > while ((!give_up) && (!success)) // as long as not successful and not giving up, we solve again with smaller time step > > { > > if (time_step_size < dt_min) { give_up = true; break; } // ok, bad luck, give up due to time step size smaller than a preset value > > if (experienced_fail_this_time_step) { // get the vectors from backups if this is a re-try, i.e., already failed with a larger time step > > VecCopy(u_old_copy, u); VecCopy(u_old_copy, u_old); > > } > > > > try { > > SNESSolve(snes, NULL, u); > > SNESGetConvergedReason(snes, &snes_converged_reason); > > > > if (snes_converged_reason > 0) success = true; // yes, snes converged > > else { // no, snes did not converge > > cutTimeStepSize(); // e.g., dt / 2 > > experienced_fail_this_time_step = true; > > } > > } > > catch (int err) { // in case your own pieces of code throws an exception > > std::cout << "An exception occurred." << std::endl; > > success = false; > > cutTimeStepSize(); // e.g., dt / 2 > > experienced_fail_this_time_step = true; > > } > > } > > > > if (success) { > > // output, print, whatever > > // duplicate current solution to old solution in preparing next time step > > VecCopy(u, u_old); > > // you can increase time step size here, e.g. * 2 > > increaseTimeStepSize(); > > } > > > > if (give_up) { > > simulationFailed = true; > > std::cerr << "Simulation failed.\n"; > > //exit(1);// dont exit(1) now, just break the for-loop, let PETSc clean its workspace. > > break; > > } > > } > > > > From: Rahul Samala > > > Sent: Wednesday, August 29, 2018 10:37:30 PM > > To: Ling Zou; Smith, Barry F. > > Cc: PETSc Users List > > Subject: Re: [petsc-users] Problem with SNES convergence > > > > Thank you Ling, I would definitely like to look at your code for reducing timestep size. > > Thank you Barry for your inputs. > > > > -- > > Rahul. > > > > On Wednesday, August 29, 2018, 9:02:00 PM GMT+5:30, Smith, Barry F. > wrote: > > > > > > > > Current time (before start of timestep) 52.5048, iter=5380 Timestep=864.000000 > > 0 SNES Function norm 1.650467412595e+05 > > 0 KSP preconditioned resid norm 3.979123221160e+03 true resid norm 1.650467412595e+05 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP preconditioned resid norm 9.178246525982e-11 true resid norm 7.006473307032e-09 ||r(i)||/||b|| 4.245144892632e-14 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > 1 SNES Function norm 6.722712947273e+02 > > Linear solve did not converge due to DIVERGED_NANORINF iterations 0 > > Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 1 > > > > This usually is an indicator that the LU (ILU) factorization has hit a zero pivot (hence the linear solve has a divide by zero so gives the DIVERGED_NANORINF flag). > > > > You can/should call SNESGetConvergedReason() immediately after each SNESSolve(), if the result is negative that means something has failed in the nonlinear solve and you can try cutting the time-step and trying again. > > > > Good luck, > > > > Barry > > > > > > > On Aug 29, 2018, at 10:11 AM, Ling Zou > wrote: > > > > > > 1) My experience is that this kind of bug or sudden death (everything is fine till suddenly something is broken) is very difficult to debug/fix. I looked at your txt files and could not give any quick comments. Maybe PETSc developers have better idea on this. > > > 2) I do have successful experience on reducing time step size when PETSc fails solving or my own piece of code throws an exception. If you are interested, I can share them. > > > > > > -Ling > > > From: petsc-users > on behalf of Rahul Samala > > > > Sent: Wednesday, August 29, 2018 8:36:58 AM > > > To: PETSc Users List > > > Subject: [petsc-users] Problem with SNES convergence > > > > > > Hello PetSc users, > > > > > > 1) I have problem with SNES convergence. I call SNESSolve in a time loop and use the inbuilt Jacobian feature. The code works fine for about 5380 time steps after which it breaks down. The solution till that point looks fine. I have used newtonls of type l2. (newtontr and others aren't working). Since I have used inbuilt Jacobian feature and the code worked for about 5000 time steps I don't understand the reason for failure, is it an incorrect function evaluation? Attached are the outputs with -pc_type lu and ilu along with -snes_linesearch_type l2 -snes_converged_reason -snes_monitor -snes_view -ksp_converged_reason -ksp_monitor_true_residual > > > > > > 2) How to get hold of failure signal, like Nonlinear solve DIVERGED_MAX_IT or DIVERGED_LINEAR_SOLVE so that whenever it occurs I can use a reduced time step and see if the code converges. > > > > > > Thank you, > > > Rahul. > > > > > > output_ilu.txt > > > > > > > > > output_ilu.txt > > > > > > > > > > > > output_lu.txt > > > > > > > > > output_lu.txt > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Sep 4 13:56:45 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 4 Sep 2018 18:56:45 +0000 Subject: [petsc-users] Back tracking/preconditioning with matrix free solvers In-Reply-To: References: Message-ID: <28DD508B-1DE0-47E8-9AF7-37CDA0B2B261@anl.gov> > On Sep 4, 2018, at 9:07 AM, Kenneth Hall, SC.D. wrote: > > Hi All, > > I am solving a nonlinear problem using a matrix free approach with Petsc. My code is written in Fortran90. > I have read the documentation, and have had some success, but I am having problems with two features. > > 1. The code runs fine with no preconditioning. But when I using the -user_precond option, my preconditioner never gets called by Petsc, even though I make the appropriate calls to PCSetType and PCShellSetApply. > > 2. I am using the GMRES solver, which defaults to the L2-norm minimization line search (SNESLINESEARCHL2). I would like to use backtracking (SNESLINESEARCHBT). I have specified backtracking using SNESLineSearchSetType, but I then get the following error: > > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: MatMFFDSetBase() has not been called, this is often caused by forgetting to call > MatAssemblyBegin/End on the first Mat in the SNES compute function > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.3, Jul, 02, 2018 > [0]PETSC ERROR: mustang on a arch-darwin-c-debug-double-real named Macintosh-11.local by hall Tue Sep 4 07:29:25 2018 > [0]PETSC ERROR: Configure options --with-scalar-type=real > [0]PETSC ERROR: #1 MatMult_MFFD() line 314 in /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/mat/impls/mffd/mffd.c > [0]PETSC ERROR: #2 MatMult() line 2311 in /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/mat/interface/matrix.c > [0]PETSC ERROR: #3 SNESLineSearchApply_BT() line 130 in /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/snes/linesearch/impls/bt/linesearchbt.c > [0]PETSC ERROR: #4 SNESLineSearchApply() line 645 in /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/snes/linesearch/interface/linesearch.c > [0]PETSC ERROR: #5 SNESSolve_NGMRES() line 262 in /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/snes/impls/ngmres/snesngmres.c > [0]PETSC ERROR: #6 SNESSolve() line 4314 in /Users/hall/Documents/Fortran_Codes/Packages/petsc-3.9.3/src/snes/interface/snes.c > > So it appears that Petsc is trying to do a matrix multiply. But this is a matrix free method. The BT (in fact several line searches) uses matrix vector products to determine if "sufficient decrease" in the function takes place, hence it is not appropriate for NGMRES if you cannot apply matrix-vector products. All you can do is try the various line searches and find out which ones work "natively" with NGMRES. Barry > How do I fix this? > > The relevant code is attached below. Any help with these two issues is greatly appreciated. > > Code follows signature. I have omitted non-relavent portions. > Kenneth Hall > > > > ============================================================= > > > module petsc_krylov > use petsc > use data_module > use solution_module > implicit none > #include > #include > private > public :: gmres2 > SNES :: snes ! nonlinear solver > PC :: pc ! preconditioner context > KSP :: ksp ! Krylov subspace method context > SNESLineSearch :: linesearch > Vec :: x,r ! solution, residual vectors > PetscErrorCode :: ierr > PetscInt :: n > integer :: neq > integer :: icount,level=0 > contains > > . > . > . > > subroutine gmres2 > implicit none > ! > !.... declare local variables > PetscInt :: its ! iterations for convergence > PetscMPIInt :: rank > PetscBool :: snes_mf,snes_mf_set,user_precond > ! > !.... set problem size > call get_equation_count > n = neq > ! > !.... initialliize petsc > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) ; if (ierr .ne. 0) then ; print*,'Unable to initialize PETSc' ; endif > ! > !.... Create nonlinear solver context > call SNESCreate(PETSC_COMM_WORLD,snes,ierr); CHKERRA(ierr) > call SNESSetType(snes,SNESNGMRES,ierr); CHKERRA(ierr) > call SNESSetTolerances(snes, 0.0, 0.0, 0.0, 20, 10000,ierr); CHKERRQ(ierr) > ! > !.... Create vectors for solution and nonlinear function > call VecCreateSeq(PETSC_COMM_SELF,n,x,ierr); CHKERRA(ierr) > call VecDuplicate(x,r,ierr); CHKERRA(ierr) > ! > !.... Set function evaluation routine and vector > call SNESSetFunction(snes,r,FormFunction,0,ierr); CHKERRA(ierr) > ! > !.... Customize nonlinear solver; set runtime options > snes_mf = PETSC_FALSE; > call PetscOptionsGetBool(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,"-snes_mf",snes_mf,snes_mf_set,ierr); CHKERRA(ierr) > if (snes_mf .eqv. PETSC_TRUE) then > call SNESGetKSP(snes,ksp,ierr); CHKERRA(ierr) > call KSPGetPC(ksp,pc,ierr); CHKERRA(ierr) > call SNESGetLineSearch(snes,linesearch,ierr); CHKERRA(ierr) > call SNESLineSearchSetType(linesearch,SNESLINESEARCHBT, ierr); CHKERRA(ierr) > user_precond = PETSC_FALSE; > call PetscOptionsHasName(PETSC_NULL_OPTIONS,PETSC_NULL_CHARACTER,"-user_precond",user_precond,ierr); CHKERRA(ierr) > if (user_precond .eqv. PETSC_TRUE) then > call PCSetType(pc,PCSHELL,ierr); CHKERRA(ierr) > call PCShellSetApply(pc,MatrixFreePreconditioner,ierr); CHKERRA(ierr) > else > call PCSetType(pc,PCNONE,ierr); CHKERRA(ierr) > end if > end if > ! > !.... set snes options > call SNESSetFromOptions(snes,ierr); CHKERRA(ierr) > ! > !.... initialize solution > call load_x_from_u(x) > ! > !.... solve > call SNESSolve(snes,PETSC_NULL_VEC,x,ierr) > ! > !.... Free work space. All PETSc objects should be destroyed when they are no longer needed. > call VecDestroy(x,ierr) > call VecDestroy(r,ierr) > call SNESDestroy(snes,ierr) > call PetscFinalize(ierr) > end > ! > !==================================================================================================== > !==================================================================================================== > ! > subroutine MatrixFreePreconditioner(x1,y1) > implicit none > ! > !.... declare passed variables > Vec :: x1,y1 . > . > . > . > > end subroutine > ! > !==================================================================================================== > !==================================================================================================== > ! > subroutine FormFunction(snes,x1,f1,dummy,ierr) > implicit none > ! > !.... declare passed variables > SNES :: snes > Vec :: x1,f1,x2 > PetscErrorCode :: ierr > integer :: dummy(*) > . > . > . > end subroutine FormFunction > > > > end module > > > > From u.rochan at gmail.com Tue Sep 4 18:34:15 2018 From: u.rochan at gmail.com (Rochan Upadhyay) Date: Tue, 4 Sep 2018 18:34:15 -0500 Subject: [petsc-users] Does PetscFE framework allow use of Hdiv and Hcurl elements ? Message-ID: Dear Petsc Users and Developers, Upon looking at the function listings and their descriptions in the manual suggests that the only finite element that can be automatically used is the continuous Lagrange. Is it possible to "easily" use vector valued Hcurl and Hdiv elements (like RT, NED, BDM and other more exotic types) ? It seems that there are two methods for generating finite elements : (1) Compute nodal basis functions from orthogonal polynomials with DOFs defined as pointwise evaluations (PETSCSPACEPOLYNOMIAL) (2) Compute basis functions at quadrature points yourself and input them directly (PETSCSPACEPOINT). Even if you input the trial and test basis functions this way, how would we input the various types of Piola transforms needed to map real to reference space. It seems in the implementation at "EvaluateFieldJets" only the covariant Piola transform (using invJ) is hard coded. Also I cannot find a good documentation of Petsc FE capabilities (unlike the superb presentations describing the DM routines). For me, it is preferable to use Petsc directly for FEM as other packages like fenics, libmesh etc.are too bulky, require too much effort to learn and not so easy to experiment with. Regards, Rochan -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 4 19:40:19 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 4 Sep 2018 20:40:19 -0400 Subject: [petsc-users] Does PetscFE framework allow use of Hdiv and Hcurl elements ? In-Reply-To: References: Message-ID: On Tue, Sep 4, 2018 at 7:34 PM Rochan Upadhyay wrote: > Dear Petsc Users and Developers, > > Upon looking at the function listings and their descriptions in the manual > suggests that the only finite element that can be automatically used is the > continuous Lagrange. Is it possible to "easily" use vector valued Hcurl and > Hdiv elements (like RT, NED, BDM and other more exotic types) ? > The short answer is no. > It seems that there are two methods for generating finite elements : (1) > Compute nodal basis functions from orthogonal polynomials with DOFs defined > as pointwise evaluations (PETSCSPACEPOLYNOMIAL) (2) Compute basis functions > at quadrature points yourself and input them directly (PETSCSPACEPOINT). > Even if you input the trial and test basis functions this way, how would we > input the various types of Piola transforms needed to map real to reference > space. It seems in the implementation at "EvaluateFieldJets" only the > covariant Piola transform (using invJ) is hard coded. > True. Toby has a branch that fixes this and will support Hdiv and Hcurl. However, I do not know how many weeks away we are on this project since the semester has now started. > Also I cannot find a good documentation of Petsc FE capabilities (unlike > the superb presentations describing the DM routines). > Yes, this is somewhat on purpose. PetscFE/FV exist principally to generate examples for me, and to help us understand how discretizations should interact with meshes and solvers. We have not yet decided whether PETSc will officially support discretizations, or just use something like Firedrake or Libmesh. However, I think pushing on this has made those packages better, so its not a waste of time. So, I am willing to explain anything you want on the list, and try and translate those explanations to documentation. Also, we accept documentation Pull Requests (even a few words long), and Patrick Sanan did a nice presentation at PETSc 2018 on how to easily submit them through the web interface. I have since tried it and its easy. Thanks, Matt > For me, it is preferable to use Petsc directly for FEM as other packages > like fenics, libmesh etc.are too bulky, require too much effort to learn > and not so easy to experiment with. > > Regards, > Rochan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Wed Sep 5 02:27:34 2018 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 5 Sep 2018 15:27:34 +0800 Subject: [petsc-users] Problem compiling with 64bit PETSc In-Reply-To: <14BE4673-3EE5-48AC-96F1-288FCDA6517D@mcs.anl.gov> References: <2C69D0C3-5D83-4AD5-9B54-783A746DF0D9@anl.gov> <27433b40-13a9-0b4c-18b1-141b251c068f@gmail.com> <14BE4673-3EE5-48AC-96F1-288FCDA6517D@mcs.anl.gov> Message-ID: <81f3d266-59cd-4d5d-389e-039b6f1b5f72@gmail.com> On 31/8/2018 10:43 AM, Smith, Barry F. wrote: > >> On Aug 30, 2018, at 9:40 PM, TAY wee-beng wrote: >> >> >> On 31/8/2018 10:38 AM, Smith, Barry F. wrote: >>> PetscReal is by default real(8) you can leave those alone >>> >>> Any integer you pass to a PETSc routine needs to be declared as PetscInt (not integer) otherwise the 64 bit indices stuff won't work. >>> >>> Barry >>> >> Hi, >> >> ok, I got it. Btw, is it advisable to change all integer in my code to PetscInt? >> >> Will it cause any conflict or waste a lot of memory? >> >> Or should I only change those related to PETSc? > That is up to you. Since you probably pass the values between PETSc and non-PETSc part of the code it is probably easier just to make all the integer PetscInt instead. No performance difference that you can measure by keeping a few integer around. > > Barry Hi, For some small parts of the code, it is preferred to use integer instead. Btw, to force variable as integer, I can use int(aa). However, I tried to force variable as PetscInt using PetscInt(aa) but it can't work. Is there any way I can make it work? Thanks. >> Thanks! >>>> On Aug 30, 2018, at 9:35 PM, TAY wee-beng wrote: >>>> >>>> >>>> On 31/8/2018 10:21 AM, Matthew Knepley wrote: >>>>> On Thu, Aug 30, 2018 at 10:17 PM TAY wee-beng wrote: >>>>> Hi, >>>>> >>>>> Due to my increase grid size, I have to go 64bit. I compiled the 64bit >>>>> PETSc w/o error. However, when I tried to compile my code using the >>>>> 64bit PETSc, I got the error below. May I know why is this so? >>>>> >>>>> What changes should I make? >>>>> >>>>> Is it possible that you did not declare some inputs as PetscInt, so the interface check is failing? >>>>> >>>>> Matt >>>> Hi, >>>> >>>> I'm using the standard >>>> >>>> integer :: >>>> >>>> real(8) :: >>>> >>>> for some variables. For some others relating to PETSc, I use PetscInt. >>>> >>>> Should I change all to PetscInt and PetscReal? >>>> >>>> Currently, I use real(8) for all real values. If I change all to PetscReal, will PetscReal be real or real(8) by default? >>>> >>>> Thanks! >>>>> >>>>> [tsltaywb at nus02 ibm3d_IIB_mpi]$ make -f makefile_2018 >>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>>>> -g -ip -ipo -O3 -c -fPIC -save kinefunc.F90 >>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>>>> -g -ip -ipo -O3 -c -fPIC -save -w >>>>> -I/home/users/nus/tsltaywb/propeller/lib/petsc-3.9.3_intel_2018_64bit_rel/include >>>>> -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include >>>>> global.F90 >>>>> global.F90(979): error #6285: There is no matching specific subroutine >>>>> for this generic subroutine call. [DMDACREATE3D] >>>>> call >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>> -----^ >>>>> global.F90(989): error #6285: There is no matching specific subroutine >>>>> for this generic subroutine call. [DMDACREATE3D] >>>>> call >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>> ---------^ >>>>> global.F90(997): error #6285: There is no matching specific subroutine >>>>> for this generic subroutine call. [DMDACREATE3D] >>>>> call >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>> ---------^ >>>>> global.F90(1005): error #6285: There is no matching specific subroutine >>>>> for this generic subroutine call. [DMDACREATE3D] >>>>> call >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>> ---------^ >>>>> global.F90(1013): error #6285: There is no matching specific subroutine >>>>> for this generic subroutine call. [DMDACREATE3D] >>>>> call >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>> ---------^ >>>>> global.F90(1021): error #6285: There is no matching specific subroutine >>>>> for this generic subroutine call. [DMDACREATE3D] >>>>> call >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>> ---------^ >>>>> global.F90(1029): error #6285: There is no matching specific subroutine >>>>> for this generic subroutine call. [DMDACREATE3D] >>>>> call >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>> ---------^ >>>>> compilation aborted for global.F90 (code 1) >>>>> >>>>> -- >>>>> Thank you very much. >>>>> >>>>> Yours sincerely, >>>>> >>>>> ================================================ >>>>> TAY Wee-Beng (Zheng Weiming) ??? >>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>> ================================================ >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ From edoardo.alinovi at gmail.com Wed Sep 5 03:50:07 2018 From: edoardo.alinovi at gmail.com (Edoardo alinovi) Date: Wed, 5 Sep 2018 10:50:07 +0200 Subject: [petsc-users] PCHYPRE performance Message-ID: Dear users, I am doing some testing of my FV code, using petsc for the linear systems solution. Actually, I am running a small problem (10k cells) in parallel with 4 processors. The solution is ok, but I have a question about hypre preconditioning performance. I am using PCHYPRE with its default settings for the solution of pressure-correction equation. What I am getting is that the multigrid preconditioning is significantly slower than block jacoby in solving the equation even if the iteration needed to converge are ten time the AMG's one (50 vs 3-4). Is this related to the small size of the problem? Many thanks for the advices, Edo ------ Edoardo Alinovi, Ph.D. DICCA, Scuola Politecnica Universita' di Genova 1, via Montallegro 16145 Genova, Italy email: edoardo.alinovi at dicca.unige.it Tel: +39 010 353 2540 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 5 05:25:59 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 5 Sep 2018 06:25:59 -0400 Subject: [petsc-users] Problem compiling with 64bit PETSc In-Reply-To: <81f3d266-59cd-4d5d-389e-039b6f1b5f72@gmail.com> References: <2C69D0C3-5D83-4AD5-9B54-783A746DF0D9@anl.gov> <27433b40-13a9-0b4c-18b1-141b251c068f@gmail.com> <14BE4673-3EE5-48AC-96F1-288FCDA6517D@mcs.anl.gov> <81f3d266-59cd-4d5d-389e-039b6f1b5f72@gmail.com> Message-ID: On Wed, Sep 5, 2018 at 3:27 AM TAY wee-beng wrote: > > On 31/8/2018 10:43 AM, Smith, Barry F. wrote: > > > >> On Aug 30, 2018, at 9:40 PM, TAY wee-beng wrote: > >> > >> > >> On 31/8/2018 10:38 AM, Smith, Barry F. wrote: > >>> PetscReal is by default real(8) you can leave those alone > >>> > >>> Any integer you pass to a PETSc routine needs to be declared as > PetscInt (not integer) otherwise the 64 bit indices stuff won't work. > >>> > >>> Barry > >>> > >> Hi, > >> > >> ok, I got it. Btw, is it advisable to change all integer in my code to > PetscInt? > >> > >> Will it cause any conflict or waste a lot of memory? > >> > >> Or should I only change those related to PETSc? > > That is up to you. Since you probably pass the values between PETSc > and non-PETSc part of the code it is probably easier just to make all the > integer PetscInt instead. No performance difference that you can measure by > keeping a few integer around. > > > > Barry > Hi, > > For some small parts of the code, it is preferred to use integer > instead. Btw, to force variable as integer, I can use int(aa). However, > I tried to force variable as PetscInt using PetscInt(aa) but it can't work. > > Is there any way I can make it work? > I think you just define a PetscInt variable and use assignment. Matt > Thanks. > >> Thanks! > >>>> On Aug 30, 2018, at 9:35 PM, TAY wee-beng wrote: > >>>> > >>>> > >>>> On 31/8/2018 10:21 AM, Matthew Knepley wrote: > >>>>> On Thu, Aug 30, 2018 at 10:17 PM TAY wee-beng > wrote: > >>>>> Hi, > >>>>> > >>>>> Due to my increase grid size, I have to go 64bit. I compiled the > 64bit > >>>>> PETSc w/o error. However, when I tried to compile my code using the > >>>>> 64bit PETSc, I got the error below. May I know why is this so? > >>>>> > >>>>> What changes should I make? > >>>>> > >>>>> Is it possible that you did not declare some inputs as PetscInt, so > the interface check is failing? > >>>>> > >>>>> Matt > >>>> Hi, > >>>> > >>>> I'm using the standard > >>>> > >>>> integer :: > >>>> > >>>> real(8) :: > >>>> > >>>> for some variables. For some others relating to PETSc, I use PetscInt. > >>>> > >>>> Should I change all to PetscInt and PetscReal? > >>>> > >>>> Currently, I use real(8) for all real values. If I change all to > PetscReal, will PetscReal be real or real(8) by default? > >>>> > >>>> Thanks! > >>>>> > >>>>> [tsltaywb at nus02 ibm3d_IIB_mpi]$ make -f makefile_2018 > >>>>> > /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 > >>>>> -g -ip -ipo -O3 -c -fPIC -save kinefunc.F90 > >>>>> > /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 > >>>>> -g -ip -ipo -O3 -c -fPIC -save -w > >>>>> > -I/home/users/nus/tsltaywb/propeller/lib/petsc-3.9.3_intel_2018_64bit_rel/include > >>>>> > -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include > >>>>> global.F90 > >>>>> global.F90(979): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. [DMDACREATE3D] > >>>>> call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> -----^ > >>>>> global.F90(989): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. [DMDACREATE3D] > >>>>> call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> ---------^ > >>>>> global.F90(997): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. [DMDACREATE3D] > >>>>> call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> ---------^ > >>>>> global.F90(1005): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. [DMDACREATE3D] > >>>>> call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> ---------^ > >>>>> global.F90(1013): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. [DMDACREATE3D] > >>>>> call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> ---------^ > >>>>> global.F90(1021): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. [DMDACREATE3D] > >>>>> call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> ---------^ > >>>>> global.F90(1029): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. [DMDACREATE3D] > >>>>> call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> ---------^ > >>>>> compilation aborted for global.F90 (code 1) > >>>>> > >>>>> -- > >>>>> Thank you very much. > >>>>> > >>>>> Yours sincerely, > >>>>> > >>>>> ================================================ > >>>>> TAY Wee-Beng (Zheng Weiming) ??? > >>>>> Personal research webpage: http://tayweebeng.wixsite.com/website > >>>>> Youtube research showcase: > https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > >>>>> linkedin: www.linkedin.com/in/tay-weebeng > >>>>> ================================================ > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >>>>> -- Norbert Wiener > >>>>> > >>>>> https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 5 05:39:37 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 5 Sep 2018 06:39:37 -0400 Subject: [petsc-users] PCHYPRE performance In-Reply-To: References: Message-ID: On Wed, Sep 5, 2018 at 4:50 AM Edoardo alinovi wrote: > Dear users, > > I am doing some testing of my FV code, using petsc for the linear systems > solution. Actually, I am running a small problem (10k cells) in parallel > with 4 processors. The solution is ok, but I have a question about hypre > preconditioning performance. I am using PCHYPRE with its default settings > for the solution of pressure-correction equation. What I am getting is that > the multigrid preconditioning is significantly slower than block jacoby in > solving the equation even if the iteration needed to converge are ten time > the AMG's one (50 vs 3-4). Is this related to the small size of the problem? > It could be the setup time. You could try ML or GAMG to compare, but that is a little strange. Thanks, Matt > Many thanks for the advices, > > Edo > > ------ > > Edoardo Alinovi, Ph.D. > > DICCA, Scuola Politecnica > Universita' di Genova > 1, via Montallegro > 16145 Genova, Italy > > email: edoardo.alinovi at dicca.unige.it > Tel: +39 010 353 2540 > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srossi at email.unc.edu Wed Sep 5 10:46:18 2018 From: srossi at email.unc.edu (Rossi, Simone) Date: Wed, 5 Sep 2018 15:46:18 +0000 Subject: [petsc-users] FIELDSPLIT fields In-Reply-To: <7B57F78B-1888-411F-B510-F2083367463F@email.unc.edu> References: <7B57F78B-1888-411F-B510-F2083367463F@email.unc.edu> Message-ID: <444902CB-087B-40EE-8D1E-ADFEA598A388@ad.unc.edu> I?m trying to setup GASM, but I?m probably misunderstanding something. If I have this mesh 0 ?? 1 ?? 2 ?? 3 ?? 4 subdomain 1 | subdomain 2 I create an interior (no overlap) and an outer (with overlap) IS for both subdomains. In my naive understanding iis1 = {0, 1} ois1 = {0, 1, 2} and iis2 = {3, 4} ois2 = {2, 3, 4} but then the node at the interface (node 2) does not belong to any interior IS. Should node 2 belong to both interior IS? Or should it belong only to one of the domains? Thanks, Simone On Aug 15, 2018, at 22:11, Griffith, Boyce Eugene wrote: > > >> On Aug 15, 2018, at 10:07 PM, Smith, Barry F. wrote: >> >> >> Yes you can have "overlapping fields" with FIELDSPLIT but I don't think you can use FIELDSPLIT for your case. You seem to have a geometric decomposition into regions. ASM and GASM are intended for such decompositions. Fieldsplit is for multiple fields that each live across the entire domain. > > Basically there is one field the lives on the entire domain, and another field that lives only on a subdomain. > > Perhaps we could do GASM for the geometric split and FIELDSPLIT within the subdomain with the two fields. > >> Barry >> >> >>> On Aug 15, 2018, at 7:42 PM, Griffith, Boyce Eugene wrote: >>> >>> Is it permissible to have overlapping fields in FIELDSPLIT? We are specifically thinking about how to handle DOFs living on the interface between two regions. >>> >>> Thanks! >>> >>> ? Boyce >> > From bsmith at mcs.anl.gov Wed Sep 5 10:54:16 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 5 Sep 2018 15:54:16 +0000 Subject: [petsc-users] FIELDSPLIT fields In-Reply-To: <444902CB-087B-40EE-8D1E-ADFEA598A388@ad.unc.edu> References: <7B57F78B-1888-411F-B510-F2083367463F@email.unc.edu> <444902CB-087B-40EE-8D1E-ADFEA598A388@ad.unc.edu> Message-ID: <02D0641F-B9D8-4E48-8EBC-209AFA50CA77@mcs.anl.gov> 2 should belong to one of the subdomains, either one is fine. Barry > On Sep 5, 2018, at 10:46 AM, Rossi, Simone wrote: > > I?m trying to setup GASM, but I?m probably misunderstanding something. > > If I have this mesh > > 0 ?? 1 ?? 2 ?? 3 ?? 4 > subdomain 1 | subdomain 2 > > I create an interior (no overlap) and an outer (with overlap) IS for both subdomains. > > In my naive understanding > > iis1 = {0, 1} > ois1 = {0, 1, 2} > > and > > iis2 = {3, 4} > ois2 = {2, 3, 4} > > but then the node at the interface (node 2) does not belong to any interior IS. Should node 2 belong to both interior IS? Or should it belong only to one of the domains? > > Thanks, > Simone > > > On Aug 15, 2018, at 22:11, Griffith, Boyce Eugene wrote: > >> >> >>> On Aug 15, 2018, at 10:07 PM, Smith, Barry F. wrote: >>> >>> >>> Yes you can have "overlapping fields" with FIELDSPLIT but I don't think you can use FIELDSPLIT for your case. You seem to have a geometric decomposition into regions. ASM and GASM are intended for such decompositions. Fieldsplit is for multiple fields that each live across the entire domain. >> >> Basically there is one field the lives on the entire domain, and another field that lives only on a subdomain. >> >> Perhaps we could do GASM for the geometric split and FIELDSPLIT within the subdomain with the two fields. >> >>> Barry >>> >>> >>>> On Aug 15, 2018, at 7:42 PM, Griffith, Boyce Eugene wrote: >>>> >>>> Is it permissible to have overlapping fields in FIELDSPLIT? We are specifically thinking about how to handle DOFs living on the interface between two regions. >>>> >>>> Thanks! >>>> >>>> ? Boyce >>> >> > From bsmith at mcs.anl.gov Wed Sep 5 11:31:07 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 5 Sep 2018 16:31:07 +0000 Subject: [petsc-users] PCHYPRE performance In-Reply-To: References: Message-ID: > On Sep 5, 2018, at 3:50 AM, Edoardo alinovi wrote: > > Dear users, > > I am doing some testing of my FV code, using petsc for the linear systems solution. Actually, I am running a small problem (10k cells) in parallel with 4 processors. The solution is ok, but I have a question about hypre preconditioning performance. I am using PCHYPRE with its default settings for the solution of pressure-correction equation. What I am getting is that the multigrid preconditioning is significantly slower than block jacoby in solving the equation even if the iteration needed to converge are ten time the AMG's one (50 vs 3-4). Is this related to the small size of the problem? Yes, compare on realistic size problems, and keep at least 10,000 degrees of freedom per processor. Run on say 100 MPI processes min. Barry > > Many thanks for the advices, > > Edo > > ------ > > Edoardo Alinovi, Ph.D. > > DICCA, Scuola Politecnica > Universita' di Genova > 1, via Montallegro > 16145 Genova, Italy > > email: edoardo.alinovi at dicca.unige.it > Tel: +39 010 353 2540 > > From fdkong.jd at gmail.com Wed Sep 5 11:37:05 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Wed, 5 Sep 2018 10:37:05 -0600 Subject: [petsc-users] FIELDSPLIT fields In-Reply-To: <02D0641F-B9D8-4E48-8EBC-209AFA50CA77@mcs.anl.gov> References: <7B57F78B-1888-411F-B510-F2083367463F@email.unc.edu> <444902CB-087B-40EE-8D1E-ADFEA598A388@ad.unc.edu> <02D0641F-B9D8-4E48-8EBC-209AFA50CA77@mcs.anl.gov> Message-ID: On Wed, Sep 5, 2018 at 9:54 AM Smith, Barry F. wrote: > > 2 should belong to one of the subdomains, either one is fine. > > Barry > > > > On Sep 5, 2018, at 10:46 AM, Rossi, Simone wrote: > > > > I?m trying to setup GASM, but I?m probably misunderstanding something. > > > > If I have this mesh > > > > 0 ?? 1 ?? 2 ?? 3 ?? 4 > > subdomain 1 | subdomain 2 > > > You may need to make a decision which subdomain ``2" belongs to. Most people just let the shared node go to the lower MPI rank. If so, in this example, ``2" belongs to the subdomain one. iis1 = {0, 1, 2} ois = {0, 1, 2, 3} iis2 = {3, 4} ois2 = {2, 3, 4} You consider seriously to use GASM, I would suggest to partition your problem (using ``hierach") in such a way that multi-rank subdomain is actually connected, otherwise you may end up having a deficient performance. Thanks, Fande, > > I create an interior (no overlap) and an outer (with overlap) IS for > both subdomains. > > > > In my naive understanding > > > > iis1 = {0, 1} > > ois1 = {0, 1, 2} > > > > and > > > > iis2 = {3, 4} > > ois2 = {2, 3, 4} > > > > but then the node at the interface (node 2) does not belong to any > interior IS. Should node 2 belong to both interior IS? Or should it belong > only to one of the domains? > > > > Thanks, > > Simone > > > > > > On Aug 15, 2018, at 22:11, Griffith, Boyce Eugene > wrote: > > > >> > >> > >>> On Aug 15, 2018, at 10:07 PM, Smith, Barry F. > wrote: > >>> > >>> > >>> Yes you can have "overlapping fields" with FIELDSPLIT but I don't > think you can use FIELDSPLIT for your case. You seem to have a geometric > decomposition into regions. ASM and GASM are intended for such > decompositions. Fieldsplit is for multiple fields that each live across the > entire domain. > >> > >> Basically there is one field the lives on the entire domain, and > another field that lives only on a subdomain. > >> > >> Perhaps we could do GASM for the geometric split and FIELDSPLIT within > the subdomain with the two fields. > >> > >>> Barry > >>> > >>> > >>>> On Aug 15, 2018, at 7:42 PM, Griffith, Boyce Eugene < > boyceg at email.unc.edu> wrote: > >>>> > >>>> Is it permissible to have overlapping fields in FIELDSPLIT? We are > specifically thinking about how to handle DOFs living on the interface > between two regions. > >>>> > >>>> Thanks! > >>>> > >>>> ? Boyce > >>> > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olk548 at mail.usask.ca Wed Sep 5 20:32:04 2018 From: olk548 at mail.usask.ca (Oleksandr Koshkarov) Date: Wed, 5 Sep 2018 19:32:04 -0600 Subject: [petsc-users] matrix-free SNES with preconditioner Message-ID: <207d4208-c5d0-a1c3-c8a5-6151731f5edb@mail.usask.ca> Hello all, I got lost a little with petsc and want to clarify something. I want to do my nonlinear solves with matrix free method, but I want a real matrix preconditioner and invert it with LU. without preconditioner it is simple: Mat Jmf; MatCreateSNESMF(snes,&Jmf); SNESSetJacobian(snes,Jmf,Jmf,MatMFFDComputeJacobian,0); But with preconditioner, I am lost: Mat Jmf,J; MatCreateSNESMF(snes,&Jmf); DMCreateMatrix(da,&J); SNESSetJacobian(snes,Jmf,J,FromJacobian,0); //And here I have problems: PetscErrorCode FormJacobian(SNES snes,Vec X,Mat Jmf,Mat J,void *ptr){ // I am constructing it only once in a while, not every nonlinear solve (this snes is inside time stepper). if (sometimes) {? MatSetValuesStencil(J,...); ... ?? } ? MatAssemblyBegin(J,MAT_FINAL_ASSEMBLY); ? MatAssemblyEnd(J,MAT_FINAL_ASSEMBLY); // assembling Jmf ? MatAssemblyBegin(Jmf,MAT_FINAL_ASSEMBLY); ? MatAssemblyEnd(Jmf,MAT_FINAL_ASSEMBLY); ? return(0); } Here I have two concerns: 1) I am changing J only once in a while, because it is expansive to do otherwise. Am I doing it correctly? During iteration when I am not changing J, will petsc know this? I do not want petsc to perform LU every time when I am not changing J. And when I will change J, will it work? (My snes is inside time stepper, and I want to construct preconditioner only say every 100 time steps) 2) What about Jmf? how can I let petsc know that it should use matrix free finite difference? Will it work if I let it as it is? (I think so, but I am not sure - my information source is ts/examples/tutorials/ex15.c, but I do not understand it fully) Thank you and best regards, Alex. From knepley at gmail.com Wed Sep 5 20:37:39 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 5 Sep 2018 21:37:39 -0400 Subject: [petsc-users] matrix-free SNES with preconditioner In-Reply-To: <207d4208-c5d0-a1c3-c8a5-6151731f5edb@mail.usask.ca> References: <207d4208-c5d0-a1c3-c8a5-6151731f5edb@mail.usask.ca> Message-ID: On Wed, Sep 5, 2018 at 9:34 PM Oleksandr Koshkarov wrote: > Hello all, > > I got lost a little with petsc and want to clarify something. > > I want to do my nonlinear solves with matrix free method, but I want a > real matrix preconditioner and invert it with LU. > > without preconditioner it is simple: > > Mat Jmf; > > MatCreateSNESMF(snes,&Jmf); > > SNESSetJacobian(snes,Jmf,Jmf,MatMFFDComputeJacobian,0); > > > But with preconditioner, I am lost: > > Mat Jmf,J; > > MatCreateSNESMF(snes,&Jmf); > > DMCreateMatrix(da,&J); > > SNESSetJacobian(snes,Jmf,J,FromJacobian,0); > > //And here I have problems: > > PetscErrorCode FormJacobian(SNES snes,Vec X,Mat Jmf,Mat J,void *ptr){ > > // I am constructing it only once in a while, not every nonlinear solve > (this snes is inside time stepper). > > if (sometimes) { MatSetValuesStencil(J,...); ... } > > MatAssemblyBegin(J,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(J,MAT_FINAL_ASSEMBLY); > > // assembling Jmf > MatAssemblyBegin(Jmf,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(Jmf,MAT_FINAL_ASSEMBLY); > > return(0); > } > > Here I have two concerns: > > 1) I am changing J only once in a while, because it is expansive to do > otherwise. Am I doing it correctly? During iteration when I am not > changing J, will petsc know this? Yes, every matrix has a state variable that is updated when values are changed. The LU looks at this state variable to decide whether to refactor. > I do not want petsc to perform LU > every time when I am not changing J. And when I will change J, will it > work? Yes, but it is easy to check in the log. Use -log_view and look at the number of calls to PCLUFactorNumeric. > (My snes is inside time stepper, and I want to construct > preconditioner only say every 100 time steps) > > 2) What about Jmf? how can I let petsc know that it should use matrix > free finite difference? Will it work if I let it as it is? (I think so, > but I am not sure - my information source is > ts/examples/tutorials/ex15.c, but I do not understand it fully) > It appears you have it correct. I always use MMS to make sure my solver is doing what I expect. Thanks, Matt > Thank you and best regards, > > Alex. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srossi at email.unc.edu Wed Sep 5 21:22:01 2018 From: srossi at email.unc.edu (Rossi, Simone) Date: Thu, 6 Sep 2018 02:22:01 +0000 Subject: [petsc-users] FIELDSPLIT fields In-Reply-To: References: <7B57F78B-1888-411F-B510-F2083367463F@email.unc.edu> <444902CB-087B-40EE-8D1E-ADFEA598A388@ad.unc.edu> <02D0641F-B9D8-4E48-8EBC-209AFA50CA77@mcs.anl.gov> Message-ID: Thanks for clarifying that. So if I have my subdomain set as 0 ?? 1 ?? 2 ?? 3 ?? 4 ?? 5 ?? 6 | subdomain 1 | subdomain 2 | subdomain 1 | Should I actually make sure that I define 3 separate subdomains? Currently, my subdomains are defined depending on a parameter of the system of PDEs I?m solving. Thanks, Simone On Sep 5, 2018, at 12:37 PM, Fande Kong > wrote: On Wed, Sep 5, 2018 at 9:54 AM Smith, Barry F. > wrote: 2 should belong to one of the subdomains, either one is fine. Barry > On Sep 5, 2018, at 10:46 AM, Rossi, Simone > wrote: > > I?m trying to setup GASM, but I?m probably misunderstanding something. > > If I have this mesh > > 0 ?? 1 ?? 2 ?? 3 ?? 4 > subdomain 1 | subdomain 2 > You may need to make a decision which subdomain ``2" belongs to. Most people just let the shared node go to the lower MPI rank. If so, in this example, ``2" belongs to the subdomain one. iis1 = {0, 1, 2} ois = {0, 1, 2, 3} iis2 = {3, 4} ois2 = {2, 3, 4} You consider seriously to use GASM, I would suggest to partition your problem (using ``hierach") in such a way that multi-rank subdomain is actually connected, otherwise you may end up having a deficient performance. Thanks, Fande, > I create an interior (no overlap) and an outer (with overlap) IS for both subdomains. > > In my naive understanding > > iis1 = {0, 1} > ois1 = {0, 1, 2} > > and > > iis2 = {3, 4} > ois2 = {2, 3, 4} > > but then the node at the interface (node 2) does not belong to any interior IS. Should node 2 belong to both interior IS? Or should it belong only to one of the domains? > > Thanks, > Simone > > > On Aug 15, 2018, at 22:11, Griffith, Boyce Eugene > wrote: > >> >> >>> On Aug 15, 2018, at 10:07 PM, Smith, Barry F. > wrote: >>> >>> >>> Yes you can have "overlapping fields" with FIELDSPLIT but I don't think you can use FIELDSPLIT for your case. You seem to have a geometric decomposition into regions. ASM and GASM are intended for such decompositions. Fieldsplit is for multiple fields that each live across the entire domain. >> >> Basically there is one field the lives on the entire domain, and another field that lives only on a subdomain. >> >> Perhaps we could do GASM for the geometric split and FIELDSPLIT within the subdomain with the two fields. >> >>> Barry >>> >>> >>>> On Aug 15, 2018, at 7:42 PM, Griffith, Boyce Eugene > wrote: >>>> >>>> Is it permissible to have overlapping fields in FIELDSPLIT? We are specifically thinking about how to handle DOFs living on the interface between two regions. >>>> >>>> Thanks! >>>> >>>> ? Boyce >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Sep 5 21:54:48 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 6 Sep 2018 02:54:48 +0000 Subject: [petsc-users] FIELDSPLIT fields In-Reply-To: References: <7B57F78B-1888-411F-B510-F2083367463F@email.unc.edu> <444902CB-087B-40EE-8D1E-ADFEA598A388@ad.unc.edu> <02D0641F-B9D8-4E48-8EBC-209AFA50CA77@mcs.anl.gov> Message-ID: > On Sep 5, 2018, at 9:22 PM, Rossi, Simone wrote: > > Thanks for clarifying that. > > So if I have my subdomain set as > > 0 ?? 1 ?? 2 ?? 3 ?? 4 ?? 5 ?? 6 > | subdomain 1 | subdomain 2 | subdomain 1 | > > Should I actually make sure that I define 3 separate subdomains? You don't have to. The degrees of freedom that define a subdomain do not have to be connected. So your first subdomain could be 0,1,2,5,6 and second subdomain 3,4. For best convergence it is better if the subdomains are connected but they need not be for correctness. If it is a struggle to make them connected then don't worry about it. Barry > Currently, my subdomains are defined depending on a parameter of the system of PDEs I?m solving. > > Thanks, > > Simone > >> On Sep 5, 2018, at 12:37 PM, Fande Kong wrote: >> >> >> >> On Wed, Sep 5, 2018 at 9:54 AM Smith, Barry F. wrote: >> >> 2 should belong to one of the subdomains, either one is fine. >> >> Barry >> >> >> > On Sep 5, 2018, at 10:46 AM, Rossi, Simone wrote: >> > >> > I?m trying to setup GASM, but I?m probably misunderstanding something. >> > >> > If I have this mesh >> > >> > 0 ?? 1 ?? 2 ?? 3 ?? 4 >> > subdomain 1 | subdomain 2 >> > >> >> You may need to make a decision which subdomain ``2" belongs to. Most people just let the shared node go to the lower MPI rank. If so, in this example, ``2" belongs to the subdomain one. >> >> iis1 = {0, 1, 2} >> ois = {0, 1, 2, 3} >> >> iis2 = {3, 4} >> ois2 = {2, 3, 4} >> >> You consider seriously to use GASM, I would suggest to partition your problem (using ``hierach") in such a way that multi-rank subdomain is actually connected, otherwise you may end up having a deficient performance. >> >> >> Thanks, >> >> Fande, >> >> >> >> > I create an interior (no overlap) and an outer (with overlap) IS for both subdomains. >> > >> > In my naive understanding >> > >> > iis1 = {0, 1} >> > ois1 = {0, 1, 2} >> > >> > and >> > >> > iis2 = {3, 4} >> > ois2 = {2, 3, 4} >> > >> > but then the node at the interface (node 2) does not belong to any interior IS. Should node 2 belong to both interior IS? Or should it belong only to one of the domains? >> > >> > Thanks, >> > Simone >> > >> > >> > On Aug 15, 2018, at 22:11, Griffith, Boyce Eugene wrote: >> > >> >> >> >> >> >>> On Aug 15, 2018, at 10:07 PM, Smith, Barry F. wrote: >> >>> >> >>> >> >>> Yes you can have "overlapping fields" with FIELDSPLIT but I don't think you can use FIELDSPLIT for your case. You seem to have a geometric decomposition into regions. ASM and GASM are intended for such decompositions. Fieldsplit is for multiple fields that each live across the entire domain. >> >> >> >> Basically there is one field the lives on the entire domain, and another field that lives only on a subdomain. >> >> >> >> Perhaps we could do GASM for the geometric split and FIELDSPLIT within the subdomain with the two fields. >> >> >> >>> Barry >> >>> >> >>> >> >>>> On Aug 15, 2018, at 7:42 PM, Griffith, Boyce Eugene wrote: >> >>>> >> >>>> Is it permissible to have overlapping fields in FIELDSPLIT? We are specifically thinking about how to handle DOFs living on the interface between two regions. >> >>>> >> >>>> Thanks! >> >>>> >> >>>> ? Boyce >> >>> >> >> >> > > From zonexo at gmail.com Wed Sep 5 22:56:40 2018 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 6 Sep 2018 11:56:40 +0800 Subject: [petsc-users] Problem compiling with 64bit PETSc In-Reply-To: References: <2C69D0C3-5D83-4AD5-9B54-783A746DF0D9@anl.gov> <27433b40-13a9-0b4c-18b1-141b251c068f@gmail.com> <14BE4673-3EE5-48AC-96F1-288FCDA6517D@mcs.anl.gov> <81f3d266-59cd-4d5d-389e-039b6f1b5f72@gmail.com> Message-ID: <5822b9ea-05fd-3cfb-b365-07d26e81d895@gmail.com> Hi, My code has some problems now after converting to 64bit indices. After debugging, I realised that I'm using: call MPI_ALLGATHER(counter,1,MPI_INTEGER,counter_global,1,MPI_INTEGER,MPI_COMM_WORLD,ierr) but now counter and counter_global are both 64bit integers. So should I change all mpi routine from MPI_INTEGER to MPI_INTEGER8? But if I switch back to using the 32bit PETSc, do I have to switch back again? In that case, does it mean I need to have 2 copies of my code - one to compile with PETSc 32, another to compile with PETSc 64? Is there an easier way? Thank you very much. Yours sincerely, ================================================ TAY Wee-Beng (Zheng Weiming) ??? Personal research webpage: http://tayweebeng.wixsite.com/website Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA linkedin: www.linkedin.com/in/tay-weebeng ================================================ On 5/9/2018 6:25 PM, Matthew Knepley wrote: > On Wed, Sep 5, 2018 at 3:27 AM TAY wee-beng > wrote: > > > On 31/8/2018 10:43 AM, Smith, Barry F. wrote: > > > >> On Aug 30, 2018, at 9:40 PM, TAY wee-beng > wrote: > >> > >> > >> On 31/8/2018 10:38 AM, Smith, Barry F. wrote: > >>>? ? PetscReal is by default real(8) you can leave those alone > >>> > >>>? ? ?Any integer you pass to a PETSc routine needs to be > declared as PetscInt (not integer) otherwise the 64 bit indices > stuff won't work. > >>> > >>>? ? ?Barry > >>> > >> Hi, > >> > >> ok, I got it. Btw, is it advisable to change all integer in my > code to PetscInt? > >> > >> Will it cause any conflict or waste a lot of memory? > >> > >> Or should I only change those related to PETSc? > >? ? ? That is up to you. Since you probably pass the values > between PETSc and non-PETSc part of the code it is probably easier > just to make all the integer PetscInt instead. No performance > difference that you can measure by keeping a few integer around. > > > >? ? ? Barry > Hi, > > For some small parts of the code, it is preferred to use integer > instead. Btw, to force variable as integer, I can use int(aa). > However, > I tried to force variable as PetscInt using PetscInt(aa) but it > can't work. > > Is there any way I can make it work? > > > I think you just define a PetscInt variable and use assignment. > > ? ?Matt > > Thanks. > >> Thanks! > >>>> On Aug 30, 2018, at 9:35 PM, TAY wee-beng > wrote: > >>>> > >>>> > >>>> On 31/8/2018 10:21 AM, Matthew Knepley wrote: > >>>>> On Thu, Aug 30, 2018 at 10:17 PM TAY wee-beng > > wrote: > >>>>> Hi, > >>>>> > >>>>> Due to my increase grid size, I have to go 64bit. I compiled > the 64bit > >>>>> PETSc w/o error. However, when I tried to compile my code > using the > >>>>> 64bit PETSc, I got the error below. May I know why is this so? > >>>>> > >>>>> What changes should I make? > >>>>> > >>>>> Is it possible that you did not declare some inputs as > PetscInt, so the interface check is failing? > >>>>> > >>>>>? ? ?Matt > >>>> Hi, > >>>> > >>>> I'm using the standard > >>>> > >>>> integer :: > >>>> > >>>> real(8) :: > >>>> > >>>> for some variables. For some others relating to PETSc, I use > PetscInt. > >>>> > >>>> Should I change all to PetscInt and PetscReal? > >>>> > >>>> Currently, I use real(8) for all real values. If I change all > to PetscReal, will PetscReal be real or real(8) by default? > >>>> > >>>> Thanks! > >>>>> > >>>>> [tsltaywb at nus02 ibm3d_IIB_mpi]$ make -f makefile_2018 > >>>>> > /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 > >>>>> -g -ip -ipo -O3 -c -fPIC? -save kinefunc.F90 > >>>>> > /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 > >>>>> -g -ip -ipo -O3 -c -fPIC? -save? -w > >>>>> > -I/home/users/nus/tsltaywb/propeller/lib/petsc-3.9.3_intel_2018_64bit_rel/include > >>>>> > -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include > >>>>> global.F90 > >>>>> global.F90(979): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. ?[DMDACREATE3D] > >>>>> call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> -----^ > >>>>> global.F90(989): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. ?[DMDACREATE3D] > >>>>>? ? ? ?call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> ---------^ > >>>>> global.F90(997): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. ?[DMDACREATE3D] > >>>>>? ? ? ?call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> ---------^ > >>>>> global.F90(1005): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. ?[DMDACREATE3D] > >>>>>? ? ? ?call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> ---------^ > >>>>> global.F90(1013): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. ?[DMDACREATE3D] > >>>>>? ? ? ?call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> ---------^ > >>>>> global.F90(1021): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. ?[DMDACREATE3D] > >>>>>? ? ? ?call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> ---------^ > >>>>> global.F90(1029): error #6285: There is no matching specific > subroutine > >>>>> for this generic subroutine call. ?[DMDACREATE3D] > >>>>>? ? ? ?call > >>>>> > DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > >>>>> ---------^ > >>>>> compilation aborted for global.F90 (code 1) > >>>>> > >>>>> -- > >>>>> Thank you very much. > >>>>> > >>>>> Yours sincerely, > >>>>> > >>>>> ================================================ > >>>>> TAY Wee-Beng (Zheng Weiming) ??? > >>>>> Personal research webpage: http://tayweebeng.wixsite.com/website > >>>>> Youtube research showcase: > https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > >>>>> linkedin: www.linkedin.com/in/tay-weebeng > > >>>>> ================================================ > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> What most experimenters take for granted before they begin > their experiments is infinitely more interesting than any results > to which their experiments lead. > >>>>> -- Norbert Wiener > >>>>> > >>>>> https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlmackie862 at gmail.com Wed Sep 5 23:01:40 2018 From: rlmackie862 at gmail.com (Randall Mackie) Date: Wed, 5 Sep 2018 21:01:40 -0700 Subject: [petsc-users] Problem compiling with 64bit PETSc In-Reply-To: <5822b9ea-05fd-3cfb-b365-07d26e81d895@gmail.com> References: <2C69D0C3-5D83-4AD5-9B54-783A746DF0D9@anl.gov> <27433b40-13a9-0b4c-18b1-141b251c068f@gmail.com> <14BE4673-3EE5-48AC-96F1-288FCDA6517D@mcs.anl.gov> <81f3d266-59cd-4d5d-389e-039b6f1b5f72@gmail.com> <5822b9ea-05fd-3cfb-b365-07d26e81d895@gmail.com> Message-ID: <4506E418-0F49-469E-8246-728553365A01@gmail.com> You can use PetscMPIInt for integers in MPI calls. Check petscsys.h for definitions of all of these. Randy > On Sep 5, 2018, at 8:56 PM, TAY wee-beng wrote: > > Hi, > > My code has some problems now after converting to 64bit indices. > > After debugging, I realised that I'm using: > > call MPI_ALLGATHER(counter,1,MPI_INTEGER,counter_global,1,MPI_INTEGER,MPI_COMM_WORLD,ierr) > > but now counter and counter_global are both 64bit integers. So should I change all mpi routine from MPI_INTEGER to MPI_INTEGER8? > > But if I switch back to using the 32bit PETSc, do I have to switch back again? In that case, does it mean I need to have 2 copies of my code - one to compile with PETSc 32, another to compile with PETSc 64? > > Is there an easier way? > Thank you very much. > > Yours sincerely, > > ================================================ > TAY Wee-Beng (Zheng Weiming) ??? > Personal research webpage: http://tayweebeng.wixsite.com/website > Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > linkedin: www.linkedin.com/in/tay-weebeng > ================================================ > On 5/9/2018 6:25 PM, Matthew Knepley wrote: >> On Wed, Sep 5, 2018 at 3:27 AM TAY wee-beng > wrote: >> >> On 31/8/2018 10:43 AM, Smith, Barry F. wrote: >> > >> >> On Aug 30, 2018, at 9:40 PM, TAY wee-beng > wrote: >> >> >> >> >> >> On 31/8/2018 10:38 AM, Smith, Barry F. wrote: >> >>> PetscReal is by default real(8) you can leave those alone >> >>> >> >>> Any integer you pass to a PETSc routine needs to be declared as PetscInt (not integer) otherwise the 64 bit indices stuff won't work. >> >>> >> >>> Barry >> >>> >> >> Hi, >> >> >> >> ok, I got it. Btw, is it advisable to change all integer in my code to PetscInt? >> >> >> >> Will it cause any conflict or waste a lot of memory? >> >> >> >> Or should I only change those related to PETSc? >> > That is up to you. Since you probably pass the values between PETSc and non-PETSc part of the code it is probably easier just to make all the integer PetscInt instead. No performance difference that you can measure by keeping a few integer around. >> > >> > Barry >> Hi, >> >> For some small parts of the code, it is preferred to use integer >> instead. Btw, to force variable as integer, I can use int(aa). However, >> I tried to force variable as PetscInt using PetscInt(aa) but it can't work. >> >> Is there any way I can make it work? >> >> I think you just define a PetscInt variable and use assignment. >> >> Matt >> >> Thanks. >> >> Thanks! >> >>>> On Aug 30, 2018, at 9:35 PM, TAY wee-beng > wrote: >> >>>> >> >>>> >> >>>> On 31/8/2018 10:21 AM, Matthew Knepley wrote: >> >>>>> On Thu, Aug 30, 2018 at 10:17 PM TAY wee-beng > wrote: >> >>>>> Hi, >> >>>>> >> >>>>> Due to my increase grid size, I have to go 64bit. I compiled the 64bit >> >>>>> PETSc w/o error. However, when I tried to compile my code using the >> >>>>> 64bit PETSc, I got the error below. May I know why is this so? >> >>>>> >> >>>>> What changes should I make? >> >>>>> >> >>>>> Is it possible that you did not declare some inputs as PetscInt, so the interface check is failing? >> >>>>> >> >>>>> Matt >> >>>> Hi, >> >>>> >> >>>> I'm using the standard >> >>>> >> >>>> integer :: >> >>>> >> >>>> real(8) :: >> >>>> >> >>>> for some variables. For some others relating to PETSc, I use PetscInt. >> >>>> >> >>>> Should I change all to PetscInt and PetscReal? >> >>>> >> >>>> Currently, I use real(8) for all real values. If I change all to PetscReal, will PetscReal be real or real(8) by default? >> >>>> >> >>>> Thanks! >> >>>>> >> >>>>> [tsltaywb at nus02 ibm3d_IIB_mpi]$ make -f makefile_2018 >> >>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >> >>>>> -g -ip -ipo -O3 -c -fPIC -save kinefunc.F90 >> >>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >> >>>>> -g -ip -ipo -O3 -c -fPIC -save -w >> >>>>> -I/home/users/nus/tsltaywb/propeller/lib/petsc-3.9.3_intel_2018_64bit_rel/include >> >>>>> -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include >> >>>>> global.F90 >> >>>>> global.F90(979): error #6285: There is no matching specific subroutine >> >>>>> for this generic subroutine call. [DMDACREATE3D] >> >>>>> call >> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >> >>>>> -----^ >> >>>>> global.F90(989): error #6285: There is no matching specific subroutine >> >>>>> for this generic subroutine call. [DMDACREATE3D] >> >>>>> call >> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >> >>>>> ---------^ >> >>>>> global.F90(997): error #6285: There is no matching specific subroutine >> >>>>> for this generic subroutine call. [DMDACREATE3D] >> >>>>> call >> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >> >>>>> ---------^ >> >>>>> global.F90(1005): error #6285: There is no matching specific subroutine >> >>>>> for this generic subroutine call. [DMDACREATE3D] >> >>>>> call >> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >> >>>>> ---------^ >> >>>>> global.F90(1013): error #6285: There is no matching specific subroutine >> >>>>> for this generic subroutine call. [DMDACREATE3D] >> >>>>> call >> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >> >>>>> ---------^ >> >>>>> global.F90(1021): error #6285: There is no matching specific subroutine >> >>>>> for this generic subroutine call. [DMDACREATE3D] >> >>>>> call >> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >> >>>>> ---------^ >> >>>>> global.F90(1029): error #6285: There is no matching specific subroutine >> >>>>> for this generic subroutine call. [DMDACREATE3D] >> >>>>> call >> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >> >>>>> ---------^ >> >>>>> compilation aborted for global.F90 (code 1) >> >>>>> >> >>>>> -- >> >>>>> Thank you very much. >> >>>>> >> >>>>> Yours sincerely, >> >>>>> >> >>>>> ================================================ >> >>>>> TAY Wee-Beng (Zheng Weiming) ??? >> >>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >> >>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >> >>>>> linkedin: www.linkedin.com/in/tay-weebeng >> >>>>> ================================================ >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> >>>>> -- Norbert Wiener >> >>>>> >> >>>>> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Sep 5 23:09:32 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 6 Sep 2018 04:09:32 +0000 Subject: [petsc-users] Problem compiling with 64bit PETSc In-Reply-To: <4506E418-0F49-469E-8246-728553365A01@gmail.com> References: <2C69D0C3-5D83-4AD5-9B54-783A746DF0D9@anl.gov> <27433b40-13a9-0b4c-18b1-141b251c068f@gmail.com> <14BE4673-3EE5-48AC-96F1-288FCDA6517D@mcs.anl.gov> <81f3d266-59cd-4d5d-389e-039b6f1b5f72@gmail.com> <5822b9ea-05fd-3cfb-b365-07d26e81d895@gmail.com> <4506E418-0F49-469E-8246-728553365A01@gmail.com> Message-ID: <491AC580-D06F-46CF-86AB-8F80A12603C4@anl.gov> > On Sep 5, 2018, at 11:01 PM, Randall Mackie wrote: > > You can use PetscMPIInt for integers in MPI calls. > > Check petscsys.h for definitions of all of these. This is true but can be cumbersome because one may need to convert arrays of PetscInt to PetscMPIInt for MPI and then convert back after (with possible loss of precision). In C we have macros MPIU_INT that we use to indicate that the integer argument to the MPI call is 64 bit when 64 bit indices are used and 32 bit otherwise allowing users to write portable code that can just be reconfigured for 32 or 64 bit integers. I see we do not provide such a thing for Fortran; we should provide it. Unless Karl has coding time today we won't be able to get to it until tomorrow US time since it is already late in the US. Barry > > Randy > > >> On Sep 5, 2018, at 8:56 PM, TAY wee-beng wrote: >> >> Hi, >> >> My code has some problems now after converting to 64bit indices. >> >> After debugging, I realised that I'm using: >> >> call MPI_ALLGATHER(counter,1,MPI_INTEGER,counter_global,1,MPI_INTEGER,MPI_COMM_WORLD,ierr) >> >> but now counter and counter_global are both 64bit integers. So should I change all mpi routine from MPI_INTEGER to MPI_INTEGER8? >> >> But if I switch back to using the 32bit PETSc, do I have to switch back again? In that case, does it mean I need to have 2 copies of my code - one to compile with PETSc 32, another to compile with PETSc 64? >> >> Is there an easier way? >> Thank you very much. >> >> Yours sincerely, >> >> ================================================ >> TAY Wee-Beng (Zheng Weiming) ??? >> Personal research webpage: >> http://tayweebeng.wixsite.com/website >> >> Youtube research showcase: >> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >> >> linkedin: >> www.linkedin.com/in/tay-weebeng >> >> ================================================ >> >> On 5/9/2018 6:25 PM, Matthew Knepley wrote: >>> On Wed, Sep 5, 2018 at 3:27 AM TAY wee-beng wrote: >>> >>> On 31/8/2018 10:43 AM, Smith, Barry F. wrote: >>> > >>> >> On Aug 30, 2018, at 9:40 PM, TAY wee-beng wrote: >>> >> >>> >> >>> >> On 31/8/2018 10:38 AM, Smith, Barry F. wrote: >>> >>> PetscReal is by default real(8) you can leave those alone >>> >>> >>> >>> Any integer you pass to a PETSc routine needs to be declared as PetscInt (not integer) otherwise the 64 bit indices stuff won't work. >>> >>> >>> >>> Barry >>> >>> >>> >> Hi, >>> >> >>> >> ok, I got it. Btw, is it advisable to change all integer in my code to PetscInt? >>> >> >>> >> Will it cause any conflict or waste a lot of memory? >>> >> >>> >> Or should I only change those related to PETSc? >>> > That is up to you. Since you probably pass the values between PETSc and non-PETSc part of the code it is probably easier just to make all the integer PetscInt instead. No performance difference that you can measure by keeping a few integer around. >>> > >>> > Barry >>> Hi, >>> >>> For some small parts of the code, it is preferred to use integer >>> instead. Btw, to force variable as integer, I can use int(aa). However, >>> I tried to force variable as PetscInt using PetscInt(aa) but it can't work. >>> >>> Is there any way I can make it work? >>> >>> I think you just define a PetscInt variable and use assignment. >>> >>> Matt >>> >>> Thanks. >>> >> Thanks! >>> >>>> On Aug 30, 2018, at 9:35 PM, TAY wee-beng wrote: >>> >>>> >>> >>>> >>> >>>> On 31/8/2018 10:21 AM, Matthew Knepley wrote: >>> >>>>> On Thu, Aug 30, 2018 at 10:17 PM TAY wee-beng wrote: >>> >>>>> Hi, >>> >>>>> >>> >>>>> Due to my increase grid size, I have to go 64bit. I compiled the 64bit >>> >>>>> PETSc w/o error. However, when I tried to compile my code using the >>> >>>>> 64bit PETSc, I got the error below. May I know why is this so? >>> >>>>> >>> >>>>> What changes should I make? >>> >>>>> >>> >>>>> Is it possible that you did not declare some inputs as PetscInt, so the interface check is failing? >>> >>>>> >>> >>>>> Matt >>> >>>> Hi, >>> >>>> >>> >>>> I'm using the standard >>> >>>> >>> >>>> integer :: >>> >>>> >>> >>>> real(8) :: >>> >>>> >>> >>>> for some variables. For some others relating to PETSc, I use PetscInt. >>> >>>> >>> >>>> Should I change all to PetscInt and PetscReal? >>> >>>> >>> >>>> Currently, I use real(8) for all real values. If I change all to PetscReal, will PetscReal be real or real(8) by default? >>> >>>> >>> >>>> Thanks! >>> >>>>> >>> >>>>> [tsltaywb at nus02 ibm3d_IIB_mpi]$ make -f makefile_2018 >>> >>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>> >>>>> -g -ip -ipo -O3 -c -fPIC -save kinefunc.F90 >>> >>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>> >>>>> -g -ip -ipo -O3 -c -fPIC -save -w >>> >>>>> -I/home/users/nus/tsltaywb/propeller/lib/petsc-3.9.3_intel_2018_64bit_rel/include >>> >>>>> -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include >>> >>>>> global.F90 >>> >>>>> global.F90(979): error #6285: There is no matching specific subroutine >>> >>>>> for this generic subroutine call. [DMDACREATE3D] >>> >>>>> call >>> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>> >>>>> -----^ >>> >>>>> global.F90(989): error #6285: There is no matching specific subroutine >>> >>>>> for this generic subroutine call. [DMDACREATE3D] >>> >>>>> call >>> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>> >>>>> ---------^ >>> >>>>> global.F90(997): error #6285: There is no matching specific subroutine >>> >>>>> for this generic subroutine call. [DMDACREATE3D] >>> >>>>> call >>> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>> >>>>> ---------^ >>> >>>>> global.F90(1005): error #6285: There is no matching specific subroutine >>> >>>>> for this generic subroutine call. [DMDACREATE3D] >>> >>>>> call >>> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>> >>>>> ---------^ >>> >>>>> global.F90(1013): error #6285: There is no matching specific subroutine >>> >>>>> for this generic subroutine call. [DMDACREATE3D] >>> >>>>> call >>> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>> >>>>> ---------^ >>> >>>>> global.F90(1021): error #6285: There is no matching specific subroutine >>> >>>>> for this generic subroutine call. [DMDACREATE3D] >>> >>>>> call >>> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>> >>>>> ---------^ >>> >>>>> global.F90(1029): error #6285: There is no matching specific subroutine >>> >>>>> for this generic subroutine call. [DMDACREATE3D] >>> >>>>> call >>> >>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>> >>>>> ---------^ >>> >>>>> compilation aborted for global.F90 (code 1) >>> >>>>> >>> >>>>> -- >>> >>>>> Thank you very much. >>> >>>>> >>> >>>>> Yours sincerely, >>> >>>>> >>> >>>>> ================================================ >>> >>>>> TAY Wee-Beng (Zheng Weiming) ??? >>> >>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>> >>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>> >>>>> linkedin: www.linkedin.com/in/tay-weebeng >>> >>>>> ================================================ >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> -- >>> >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> >>>>> -- Norbert Wiener >>> >>>>> >>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >> > From zonexo at gmail.com Wed Sep 5 23:58:41 2018 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 6 Sep 2018 12:58:41 +0800 Subject: [petsc-users] Problem compiling with 64bit PETSc In-Reply-To: <491AC580-D06F-46CF-86AB-8F80A12603C4@anl.gov> References: <2C69D0C3-5D83-4AD5-9B54-783A746DF0D9@anl.gov> <27433b40-13a9-0b4c-18b1-141b251c068f@gmail.com> <14BE4673-3EE5-48AC-96F1-288FCDA6517D@mcs.anl.gov> <81f3d266-59cd-4d5d-389e-039b6f1b5f72@gmail.com> <5822b9ea-05fd-3cfb-b365-07d26e81d895@gmail.com> <4506E418-0F49-469E-8246-728553365A01@gmail.com> <491AC580-D06F-46CF-86AB-8F80A12603C4@anl.gov> Message-ID: <20a4fe0f-03bb-e6c4-a813-734eac73dba8@gmail.com> On 6/9/2018 12:09 PM, Smith, Barry F. wrote: > >> On Sep 5, 2018, at 11:01 PM, Randall Mackie wrote: >> >> You can use PetscMPIInt for integers in MPI calls. >> >> Check petscsys.h for definitions of all of these. > This is true but can be cumbersome because one may need to convert arrays of PetscInt to PetscMPIInt for MPI and then convert back after (with possible loss of precision). > > In C we have macros MPIU_INT that we use to indicate that the integer argument to the MPI call is 64 bit when 64 bit indices are used and 32 bit otherwise allowing users to write portable code that can just be reconfigured for 32 or 64 bit integers. I see we do not provide such a thing for Fortran; we should provide it. Unless Karl has coding time today we won't be able to get to it until tomorrow US time since it is already late in the US. > > Barry Hi, That would be great! I've no problem waiting a few days. Does it mean that I can then use? call MPI_ALLGATHER(counter,1,MPIU_INT,counter_global,1,MPIU_INT,MPI_COMM_WORLD,ierr) Thanks! >> Randy >> >> >>> On Sep 5, 2018, at 8:56 PM, TAY wee-beng wrote: >>> >>> Hi, >>> >>> My code has some problems now after converting to 64bit indices. >>> >>> After debugging, I realised that I'm using: >>> >>> call MPI_ALLGATHER(counter,1,MPI_INTEGER,counter_global,1,MPI_INTEGER,MPI_COMM_WORLD,ierr) >>> >>> but now counter and counter_global are both 64bit integers. So should I change all mpi routine from MPI_INTEGER to MPI_INTEGER8? >>> >>> But if I switch back to using the 32bit PETSc, do I have to switch back again? In that case, does it mean I need to have 2 copies of my code - one to compile with PETSc 32, another to compile with PETSc 64? >>> >>> Is there an easier way? >>> Thank you very much. >>> >>> Yours sincerely, >>> >>> ================================================ >>> TAY Wee-Beng (Zheng Weiming) ??? >>> Personal research webpage: >>> http://tayweebeng.wixsite.com/website >>> >>> Youtube research showcase: >>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>> >>> linkedin: >>> www.linkedin.com/in/tay-weebeng >>> >>> ================================================ >>> >>> On 5/9/2018 6:25 PM, Matthew Knepley wrote: >>>> On Wed, Sep 5, 2018 at 3:27 AM TAY wee-beng wrote: >>>> >>>> On 31/8/2018 10:43 AM, Smith, Barry F. wrote: >>>>>> On Aug 30, 2018, at 9:40 PM, TAY wee-beng wrote: >>>>>> >>>>>> >>>>>> On 31/8/2018 10:38 AM, Smith, Barry F. wrote: >>>>>>> PetscReal is by default real(8) you can leave those alone >>>>>>> >>>>>>> Any integer you pass to a PETSc routine needs to be declared as PetscInt (not integer) otherwise the 64 bit indices stuff won't work. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>> Hi, >>>>>> >>>>>> ok, I got it. Btw, is it advisable to change all integer in my code to PetscInt? >>>>>> >>>>>> Will it cause any conflict or waste a lot of memory? >>>>>> >>>>>> Or should I only change those related to PETSc? >>>>> That is up to you. Since you probably pass the values between PETSc and non-PETSc part of the code it is probably easier just to make all the integer PetscInt instead. No performance difference that you can measure by keeping a few integer around. >>>>> >>>>> Barry >>>> Hi, >>>> >>>> For some small parts of the code, it is preferred to use integer >>>> instead. Btw, to force variable as integer, I can use int(aa). However, >>>> I tried to force variable as PetscInt using PetscInt(aa) but it can't work. >>>> >>>> Is there any way I can make it work? >>>> >>>> I think you just define a PetscInt variable and use assignment. >>>> >>>> Matt >>>> >>>> Thanks. >>>>>> Thanks! >>>>>>>> On Aug 30, 2018, at 9:35 PM, TAY wee-beng wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 31/8/2018 10:21 AM, Matthew Knepley wrote: >>>>>>>>> On Thu, Aug 30, 2018 at 10:17 PM TAY wee-beng wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Due to my increase grid size, I have to go 64bit. I compiled the 64bit >>>>>>>>> PETSc w/o error. However, when I tried to compile my code using the >>>>>>>>> 64bit PETSc, I got the error below. May I know why is this so? >>>>>>>>> >>>>>>>>> What changes should I make? >>>>>>>>> >>>>>>>>> Is it possible that you did not declare some inputs as PetscInt, so the interface check is failing? >>>>>>>>> >>>>>>>>> Matt >>>>>>>> Hi, >>>>>>>> >>>>>>>> I'm using the standard >>>>>>>> >>>>>>>> integer :: >>>>>>>> >>>>>>>> real(8) :: >>>>>>>> >>>>>>>> for some variables. For some others relating to PETSc, I use PetscInt. >>>>>>>> >>>>>>>> Should I change all to PetscInt and PetscReal? >>>>>>>> >>>>>>>> Currently, I use real(8) for all real values. If I change all to PetscReal, will PetscReal be real or real(8) by default? >>>>>>>> >>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> [tsltaywb at nus02 ibm3d_IIB_mpi]$ make -f makefile_2018 >>>>>>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>>>>>>>> -g -ip -ipo -O3 -c -fPIC -save kinefunc.F90 >>>>>>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>>>>>>>> -g -ip -ipo -O3 -c -fPIC -save -w >>>>>>>>> -I/home/users/nus/tsltaywb/propeller/lib/petsc-3.9.3_intel_2018_64bit_rel/include >>>>>>>>> -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include >>>>>>>>> global.F90 >>>>>>>>> global.F90(979): error #6285: There is no matching specific subroutine >>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>> call >>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>> -----^ >>>>>>>>> global.F90(989): error #6285: There is no matching specific subroutine >>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>> call >>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>> ---------^ >>>>>>>>> global.F90(997): error #6285: There is no matching specific subroutine >>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>> call >>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>> ---------^ >>>>>>>>> global.F90(1005): error #6285: There is no matching specific subroutine >>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>> call >>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>> ---------^ >>>>>>>>> global.F90(1013): error #6285: There is no matching specific subroutine >>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>> call >>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>> ---------^ >>>>>>>>> global.F90(1021): error #6285: There is no matching specific subroutine >>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>> call >>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>> ---------^ >>>>>>>>> global.F90(1029): error #6285: There is no matching specific subroutine >>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>> call >>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>> ---------^ >>>>>>>>> compilation aborted for global.F90 (code 1) >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thank you very much. >>>>>>>>> >>>>>>>>> Yours sincerely, >>>>>>>>> >>>>>>>>> ================================================ >>>>>>>>> TAY Wee-Beng (Zheng Weiming) ??? >>>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>>>>> ================================================ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ From bsmith at mcs.anl.gov Thu Sep 6 00:16:11 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 6 Sep 2018 05:16:11 +0000 Subject: [petsc-users] Problem compiling with 64bit PETSc In-Reply-To: <20a4fe0f-03bb-e6c4-a813-734eac73dba8@gmail.com> References: <2C69D0C3-5D83-4AD5-9B54-783A746DF0D9@anl.gov> <27433b40-13a9-0b4c-18b1-141b251c068f@gmail.com> <14BE4673-3EE5-48AC-96F1-288FCDA6517D@mcs.anl.gov> <81f3d266-59cd-4d5d-389e-039b6f1b5f72@gmail.com> <5822b9ea-05fd-3cfb-b365-07d26e81d895@gmail.com> <4506E418-0F49-469E-8246-728553365A01@gmail.com> <491AC580-D06F-46CF-86AB-8F80A12603C4@anl.gov> <20a4fe0f-03bb-e6c4-a813-734eac73dba8@gmail.com> Message-ID: <69556586-62C5-4559-9209-CBA30E0E8302@mcs.anl.gov> > On Sep 5, 2018, at 11:58 PM, TAY wee-beng wrote: > > > On 6/9/2018 12:09 PM, Smith, Barry F. wrote: >> >>> On Sep 5, 2018, at 11:01 PM, Randall Mackie wrote: >>> >>> You can use PetscMPIInt for integers in MPI calls. >>> >>> Check petscsys.h for definitions of all of these. >> This is true but can be cumbersome because one may need to convert arrays of PetscInt to PetscMPIInt for MPI and then convert back after (with possible loss of precision). >> >> In C we have macros MPIU_INT that we use to indicate that the integer argument to the MPI call is 64 bit when 64 bit indices are used and 32 bit otherwise allowing users to write portable code that can just be reconfigured for 32 or 64 bit integers. I see we do not provide such a thing for Fortran; we should provide it. Unless Karl has coding time today we won't be able to get to it until tomorrow US time since it is already late in the US. >> >> Barry > Hi, > > That would be great! I've no problem waiting a few days. Does it mean that I can then use? > > call MPI_ALLGATHER(counter,1,MPIU_INT,counter_global,1,MPIU_INT,MPI_COMM_WORLD,ierr) Likely to be consistent with Fortran MPI standards we would use MPI_INTEGER but otherwise yes. > > Thanks! >>> Randy >>> >>> >>>> On Sep 5, 2018, at 8:56 PM, TAY wee-beng wrote: >>>> >>>> Hi, >>>> >>>> My code has some problems now after converting to 64bit indices. >>>> >>>> After debugging, I realised that I'm using: >>>> >>>> call MPI_ALLGATHER(counter,1,MPI_INTEGER,counter_global,1,MPI_INTEGER,MPI_COMM_WORLD,ierr) >>>> >>>> but now counter and counter_global are both 64bit integers. So should I change all mpi routine from MPI_INTEGER to MPI_INTEGER8? >>>> >>>> But if I switch back to using the 32bit PETSc, do I have to switch back again? In that case, does it mean I need to have 2 copies of my code - one to compile with PETSc 32, another to compile with PETSc 64? >>>> >>>> Is there an easier way? >>>> Thank you very much. >>>> >>>> Yours sincerely, >>>> >>>> ================================================ >>>> TAY Wee-Beng (Zheng Weiming) ??? >>>> Personal research webpage: >>>> http://tayweebeng.wixsite.com/website >>>> >>>> Youtube research showcase: >>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>> >>>> linkedin: >>>> www.linkedin.com/in/tay-weebeng >>>> >>>> ================================================ >>>> >>>> On 5/9/2018 6:25 PM, Matthew Knepley wrote: >>>>> On Wed, Sep 5, 2018 at 3:27 AM TAY wee-beng wrote: >>>>> >>>>> On 31/8/2018 10:43 AM, Smith, Barry F. wrote: >>>>>>> On Aug 30, 2018, at 9:40 PM, TAY wee-beng wrote: >>>>>>> >>>>>>> >>>>>>> On 31/8/2018 10:38 AM, Smith, Barry F. wrote: >>>>>>>> PetscReal is by default real(8) you can leave those alone >>>>>>>> >>>>>>>> Any integer you pass to a PETSc routine needs to be declared as PetscInt (not integer) otherwise the 64 bit indices stuff won't work. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> ok, I got it. Btw, is it advisable to change all integer in my code to PetscInt? >>>>>>> >>>>>>> Will it cause any conflict or waste a lot of memory? >>>>>>> >>>>>>> Or should I only change those related to PETSc? >>>>>> That is up to you. Since you probably pass the values between PETSc and non-PETSc part of the code it is probably easier just to make all the integer PetscInt instead. No performance difference that you can measure by keeping a few integer around. >>>>>> >>>>>> Barry >>>>> Hi, >>>>> >>>>> For some small parts of the code, it is preferred to use integer >>>>> instead. Btw, to force variable as integer, I can use int(aa). However, >>>>> I tried to force variable as PetscInt using PetscInt(aa) but it can't work. >>>>> >>>>> Is there any way I can make it work? >>>>> >>>>> I think you just define a PetscInt variable and use assignment. >>>>> >>>>> Matt >>>>> Thanks. >>>>>>> Thanks! >>>>>>>>> On Aug 30, 2018, at 9:35 PM, TAY wee-beng wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 31/8/2018 10:21 AM, Matthew Knepley wrote: >>>>>>>>>> On Thu, Aug 30, 2018 at 10:17 PM TAY wee-beng wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Due to my increase grid size, I have to go 64bit. I compiled the 64bit >>>>>>>>>> PETSc w/o error. However, when I tried to compile my code using the >>>>>>>>>> 64bit PETSc, I got the error below. May I know why is this so? >>>>>>>>>> >>>>>>>>>> What changes should I make? >>>>>>>>>> >>>>>>>>>> Is it possible that you did not declare some inputs as PetscInt, so the interface check is failing? >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I'm using the standard >>>>>>>>> >>>>>>>>> integer :: >>>>>>>>> >>>>>>>>> real(8) :: >>>>>>>>> >>>>>>>>> for some variables. For some others relating to PETSc, I use PetscInt. >>>>>>>>> >>>>>>>>> Should I change all to PetscInt and PetscReal? >>>>>>>>> >>>>>>>>> Currently, I use real(8) for all real values. If I change all to PetscReal, will PetscReal be real or real(8) by default? >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>>> [tsltaywb at nus02 ibm3d_IIB_mpi]$ make -f makefile_2018 >>>>>>>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>>>>>>>>> -g -ip -ipo -O3 -c -fPIC -save kinefunc.F90 >>>>>>>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>>>>>>>>> -g -ip -ipo -O3 -c -fPIC -save -w >>>>>>>>>> -I/home/users/nus/tsltaywb/propeller/lib/petsc-3.9.3_intel_2018_64bit_rel/include >>>>>>>>>> -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include >>>>>>>>>> global.F90 >>>>>>>>>> global.F90(979): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> -----^ >>>>>>>>>> global.F90(989): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> ---------^ >>>>>>>>>> global.F90(997): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> ---------^ >>>>>>>>>> global.F90(1005): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> ---------^ >>>>>>>>>> global.F90(1013): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> ---------^ >>>>>>>>>> global.F90(1021): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> ---------^ >>>>>>>>>> global.F90(1029): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> ---------^ >>>>>>>>>> compilation aborted for global.F90 (code 1) >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Thank you very much. >>>>>>>>>> >>>>>>>>>> Yours sincerely, >>>>>>>>>> >>>>>>>>>> ================================================ >>>>>>>>>> TAY Wee-Beng (Zheng Weiming) ??? >>>>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>>>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>>>>>> ================================================ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ > From yjwu16 at gmail.com Thu Sep 6 03:47:38 2018 From: yjwu16 at gmail.com (Yingjie Wu) Date: Thu, 6 Sep 2018 16:47:38 +0800 Subject: [petsc-users] Questions about residual function can't be reduced Message-ID: Dear Petsc developer: Hi, Thank you for your previous help. I recently modeled on PETSc's SNES example and wrote a computer program myself. This program is mainly for solving nonlinear equations of thermal hydraulics. ?????T? - ?_y??*Cp*u? - T_source = 0 w*?*u = ?g - ?_y?P? ??? 1/w * ?P ? = - ?? ?g / w ? Where P, T and u are variables, the distribution represents pressure, temperature and velocity. The rest are nonlinear physical parameters and constants. Because the program is very preliminary, so I use - snes_mf so that I can save the part of writing to calculate the Jacobian matrix. After compiling and passing, I found that the residual function had not dropped to a small enough level, but the program stopped, as follows: Setting Up: -snes_mf -snes_monitor -draw_pause 10 -snes_view 0 SNES Function norm 3.724996516631e+09 1 SNES Function norm 2.194322909557e+09 2 SNES Function norm 1.352051559826e+09 3 SNES Function norm 1.522311916217e+08 SNES Object: 1 MPI processes type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=1298 total number of function evaluations=11679 norm schedule ALWAYS SNESLineSearch Object: 1 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 KSP Object: 1 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Mat Object: 1 MPI processes type: mffd rows=300, cols=300 Matrix-free approximation: err=1.49012e-08 (relative error in function evaluation) Using wp compute h routine Does not compute normU I would like to know why the residual function can not continue to decline, and why the program will stop before convergence. I do not know much about the convergence criteria and convergence rules of PETSc for solving nonlinear equations. I hope I can get your help. I'm looking forward to your reply~ Thanks, Yingjie -------------- next part -------------- An HTML attachment was scrubbed... URL: From rupp at iue.tuwien.ac.at Thu Sep 6 07:04:58 2018 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Thu, 6 Sep 2018 14:04:58 +0200 Subject: [petsc-users] Problem compiling with 64bit PETSc In-Reply-To: <491AC580-D06F-46CF-86AB-8F80A12603C4@anl.gov> References: <2C69D0C3-5D83-4AD5-9B54-783A746DF0D9@anl.gov> <27433b40-13a9-0b4c-18b1-141b251c068f@gmail.com> <14BE4673-3EE5-48AC-96F1-288FCDA6517D@mcs.anl.gov> <81f3d266-59cd-4d5d-389e-039b6f1b5f72@gmail.com> <5822b9ea-05fd-3cfb-b365-07d26e81d895@gmail.com> <4506E418-0F49-469E-8246-728553365A01@gmail.com> <491AC580-D06F-46CF-86AB-8F80A12603C4@anl.gov> Message-ID: Hey, > In C we have macros MPIU_INT that we use to indicate that the integer argument to the MPI call is 64 bit when 64 bit indices are used and 32 bit otherwise allowing users to write portable code that can just be reconfigured for 32 or 64 bit integers. I see we do not provide such a thing for Fortran; we should provide it. Unless Karl has coding time today we won't be able to get to it until tomorrow US time since it is already late in the US. well, I'm not sufficiently experienced with the Fortran bindings (yet) to make this happen today... Best regards, Karli From knepley at gmail.com Thu Sep 6 09:34:25 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 6 Sep 2018 10:34:25 -0400 Subject: [petsc-users] Questions about residual function can't be reduced In-Reply-To: References: Message-ID: On Thu, Sep 6, 2018 at 4:47 AM Yingjie Wu wrote: > Dear Petsc developer: > Hi, > Thank you for your previous help. > I recently modeled on PETSc's SNES example and wrote a computer program > myself. This program is mainly for solving nonlinear equations of thermal > hydraulics. > > ?????T? - ?_y??*Cp*u? - T_source = 0 > > w*?*u = ?g - ?_y?P? > > ??? 1/w * ?P ? = - ?? ?g / w ? > > Where P, T and u are variables, the distribution represents pressure, > temperature and velocity. The rest are nonlinear physical parameters and > constants. > > Because the program is very preliminary, so I use - snes_mf so that I can > save the part of writing to calculate the Jacobian matrix. > > After compiling and passing, I found that the residual function had not > dropped to a small enough level, but the program stopped, as follows: > > Setting Up: -snes_mf -snes_monitor -draw_pause 10 -snes_view > > First, do not use -snes_mf. It is not for testing, but for sophisticated use. The first option you might try is -snes_fd -pc_type lu That uses a full Jacobian and LU factorization for a direct solve. Always run the solve using -snes_view -snes_converged_reason -snes_monitor -ksp_converged_reason -ksp_monitor_true_residual When that gets too expensive, you can try -snes_fd_color -snes_fd_color_use_mat -mat_coloring_type greedy but that requires you to preallocate the Jacobian matrix correctly. Thanks, Matt > 0 SNES Function norm 3.724996516631e+09 > > 1 SNES Function norm 2.194322909557e+09 > > 2 SNES Function norm 1.352051559826e+09 > > 3 SNES Function norm 1.522311916217e+08 > > SNES Object: 1 MPI processes > > type: newtonls > > maximum iterations=50, maximum function evaluations=10000 > > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > > total number of linear solver iterations=1298 > > total number of function evaluations=11679 > > norm schedule ALWAYS > > SNESLineSearch Object: 1 MPI processes > > type: bt > > interpolation: cubic > > alpha=1.000000e-04 > > maxstep=1.000000e+08, minlambda=1.000000e-12 > > tolerances: relative=1.000000e-08, absolute=1.000000e-15, > lambda=1.000000e-08 > > maximum iterations=40 > > KSP Object: 1 MPI processes > > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: none > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: mffd > > rows=300, cols=300 > > Matrix-free approximation: > > err=1.49012e-08 (relative error in function evaluation) > > Using wp compute h routine > > Does not compute normU > > I would like to know why the residual function can not continue to > decline, and why the program will stop before convergence. > I do not know much about the convergence criteria and convergence rules of > PETSc for solving nonlinear equations. I hope I can get your help. > I'm looking forward to your reply~ > > Thanks, > Yingjie > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srossi at email.unc.edu Thu Sep 6 11:16:13 2018 From: srossi at email.unc.edu (Rossi, Simone) Date: Thu, 6 Sep 2018 16:16:13 +0000 Subject: [petsc-users] FIELDSPLIT fields In-Reply-To: References: <7B57F78B-1888-411F-B510-F2083367463F@email.unc.edu> <444902CB-087B-40EE-8D1E-ADFEA598A388@ad.unc.edu> <02D0641F-B9D8-4E48-8EBC-209AFA50CA77@mcs.anl.gov> Message-ID: <4CB746B1-1913-40DF-8F7B-48E530E3871C@ad.unc.edu> Thanks. Then, I?ll worry about that when it will be running. Best, Simone On Sep 5, 2018, at 22:54, Smith, Barry F. wrote: > > >> On Sep 5, 2018, at 9:22 PM, Rossi, Simone wrote: >> >> Thanks for clarifying that. >> >> So if I have my subdomain set as >> >> 0 ?? 1 ?? 2 ?? 3 ?? 4 ?? 5 ?? 6 >> | subdomain 1 | subdomain 2 | subdomain 1 | >> >> Should I actually make sure that I define 3 separate subdomains? > > You don't have to. The degrees of freedom that define a subdomain do not have to be connected. So your first subdomain could be 0,1,2,5,6 and second subdomain 3,4. For best convergence it is better if the subdomains are connected but they need not be for correctness. If it is a struggle to make them connected then don't worry about it. > > Barry > > > >> Currently, my subdomains are defined depending on a parameter of the system of PDEs I?m solving. >> >> Thanks, >> >> Simone >> >>> On Sep 5, 2018, at 12:37 PM, Fande Kong wrote: >>> >>> >>> >>> On Wed, Sep 5, 2018 at 9:54 AM Smith, Barry F. wrote: >>> >>> 2 should belong to one of the subdomains, either one is fine. >>> >>> Barry >>> >>> >>>> On Sep 5, 2018, at 10:46 AM, Rossi, Simone wrote: >>>> >>>> I?m trying to setup GASM, but I?m probably misunderstanding something. >>>> >>>> If I have this mesh >>>> >>>> 0 ?? 1 ?? 2 ?? 3 ?? 4 >>>> subdomain 1 | subdomain 2 >>>> >>> >>> You may need to make a decision which subdomain ``2" belongs to. Most people just let the shared node go to the lower MPI rank. If so, in this example, ``2" belongs to the subdomain one. >>> >>> iis1 = {0, 1, 2} >>> ois = {0, 1, 2, 3} >>> >>> iis2 = {3, 4} >>> ois2 = {2, 3, 4} >>> >>> You consider seriously to use GASM, I would suggest to partition your problem (using ``hierach") in such a way that multi-rank subdomain is actually connected, otherwise you may end up having a deficient performance. >>> >>> >>> Thanks, >>> >>> Fande, >>> >>> >>> >>>> I create an interior (no overlap) and an outer (with overlap) IS for both subdomains. >>>> >>>> In my naive understanding >>>> >>>> iis1 = {0, 1} >>>> ois1 = {0, 1, 2} >>>> >>>> and >>>> >>>> iis2 = {3, 4} >>>> ois2 = {2, 3, 4} >>>> >>>> but then the node at the interface (node 2) does not belong to any interior IS. Should node 2 belong to both interior IS? Or should it belong only to one of the domains? >>>> >>>> Thanks, >>>> Simone >>>> >>>> >>>> On Aug 15, 2018, at 22:11, Griffith, Boyce Eugene wrote: >>>> >>>>> >>>>> >>>>>> On Aug 15, 2018, at 10:07 PM, Smith, Barry F. wrote: >>>>>> >>>>>> >>>>>> Yes you can have "overlapping fields" with FIELDSPLIT but I don't think you can use FIELDSPLIT for your case. You seem to have a geometric decomposition into regions. ASM and GASM are intended for such decompositions. Fieldsplit is for multiple fields that each live across the entire domain. >>>>> >>>>> Basically there is one field the lives on the entire domain, and another field that lives only on a subdomain. >>>>> >>>>> Perhaps we could do GASM for the geometric split and FIELDSPLIT within the subdomain with the two fields. >>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>>> On Aug 15, 2018, at 7:42 PM, Griffith, Boyce Eugene wrote: >>>>>>> >>>>>>> Is it permissible to have overlapping fields in FIELDSPLIT? We are specifically thinking about how to handle DOFs living on the interface between two regions. >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> ? Boyce >>>>>> >>>>> >>>> >> > From arkhazali at cc.iut.ac.ir Thu Sep 6 16:12:07 2018 From: arkhazali at cc.iut.ac.ir (Ali Reza Khaz'ali) Date: Fri, 7 Sep 2018 01:42:07 +0430 Subject: [petsc-users] Use block Jacobi preconditioner with SNES In-Reply-To: <8028B90E-AD96-4D6D-B789-3B3FFEDDF132@mcs.anl.gov> References: <82560715-ecaa-5ca6-4153-6d3f9cb7fbc8@cc.iut.ac.ir> <13850427-A944-4A1B-93C6-8493EC48D89C@anl.gov> <871saj1pxq.fsf@jedbrown.org> <67274A15-B6AE-4EFD-B52D-D50272791BE9@mcs.anl.gov> <17e76678-3da8-d5eb-9ee8-389ad307e2bf@cc.iut.ac.ir> <7b093784-8da8-1b00-9f31-a327d86882ab@cc.iut.ac.ir> <87in3t5a0j.fsf@jedbrown.org> <82f16e58-ab8e-2ab8-eb6b-26b9bd8e820e@cc.iut.ac.ir> <87a7p558ce.fsf@jedbrown.org> <8028B90E-AD96-4D6D-B789-3B3FFEDDF132@mcs.anl.gov> Message-ID: <292cfe21-dc4e-f70c-872c-cff94aaea6d7@cc.iut.ac.ir> I have to apologize for making this topic so lengthy. Is vpbjacobi supposed to work in parallel mode (with MPI)? On 8/30/2018 1:32 AM, Smith, Barry F. wrote: > >> On Aug 29, 2018, at 3:48 PM, Ali Reza Khaz'ali wrote: >> >> >>> Can you confirm if your code ran successfully with vpbjacobi and if the convergence history was very similar to that achieved using >>> bjacobi ? >> They have a very very very small difference (which is probably due to round-off errors), they generally behave the same. My code run successfully with both of them with the same convergence history. However, vpbjacobi seems a little faster. > Great, this is exactly what to expect. I will put the branch into next for testing and then it will be in the master branch and in the next release. > > Barry > >> Many thanks, >> Ali -- Ali Reza Khaz?ali Assistant Professor of Petroleum Engineering, Department of Chemical Engineering Isfahan University of Technology Isfahan, Iran From bsmith at mcs.anl.gov Thu Sep 6 16:15:43 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 6 Sep 2018 21:15:43 +0000 Subject: [petsc-users] Use block Jacobi preconditioner with SNES In-Reply-To: <292cfe21-dc4e-f70c-872c-cff94aaea6d7@cc.iut.ac.ir> References: <82560715-ecaa-5ca6-4153-6d3f9cb7fbc8@cc.iut.ac.ir> <13850427-A944-4A1B-93C6-8493EC48D89C@anl.gov> <871saj1pxq.fsf@jedbrown.org> <67274A15-B6AE-4EFD-B52D-D50272791BE9@mcs.anl.gov> <17e76678-3da8-d5eb-9ee8-389ad307e2bf@cc.iut.ac.ir> <7b093784-8da8-1b00-9f31-a327d86882ab@cc.iut.ac.ir> <87in3t5a0j.fsf@jedbrown.org> <82f16e58-ab8e-2ab8-eb6b-26b9bd8e820e@cc.iut.ac.ir> <87a7p558ce.fsf@jedbrown.org> <8028B90E-AD96-4D6D-B789-3B3FFEDDF132@mcs.anl.gov> <292cfe21-dc4e-f70c-872c-cff94aaea6d7@cc.iut.ac.ir> Message-ID: <85698F75-B2EE-42CE-8E91-EB74E2D6A0CA@mcs.anl.gov> I have not yet added this support, do you need it now? Barry > On Sep 6, 2018, at 4:12 PM, Ali Reza Khaz'ali wrote: > > I have to apologize for making this topic so lengthy. Is vpbjacobi supposed to work in parallel mode (with MPI)? > > > On 8/30/2018 1:32 AM, Smith, Barry F. wrote: >> >>> On Aug 29, 2018, at 3:48 PM, Ali Reza Khaz'ali wrote: >>> >>> >>>> Can you confirm if your code ran successfully with vpbjacobi and if the convergence history was very similar to that achieved using >>>> bjacobi ? >>> They have a very very very small difference (which is probably due to round-off errors), they generally behave the same. My code run successfully with both of them with the same convergence history. However, vpbjacobi seems a little faster. >> Great, this is exactly what to expect. I will put the branch into next for testing and then it will be in the master branch and in the next release. >> >> Barry >> >>> Many thanks, >>> Ali > > -- > Ali Reza Khaz?ali > Assistant Professor of Petroleum Engineering, > Department of Chemical Engineering > Isfahan University of Technology > Isfahan, Iran > From arkhazali at cc.iut.ac.ir Thu Sep 6 16:48:51 2018 From: arkhazali at cc.iut.ac.ir (arkhazali at cc.iut.ac.ir) Date: Fri, 07 Sep 2018 02:18:51 +0430 Subject: [petsc-users] Use block Jacobi preconditioner with SNES Message-ID: <29tqhus78u94vls7v04q04fd.1536269950599@email.lge.com> Yes, I'd be grateful if MPI support could be added, please. Many thanks,Ali ------ Original message------From: Smith, Barry F.Date: Fri, Sep 7, 2018 01:45To: Ali Reza Khaz'ali;Cc: PETSc;Subject:Re: [petsc-users] Use block Jacobi preconditioner with SNES I have not yet added this support, do you need it now? Barry > On Sep 6, 2018, at 4:12 PM, Ali Reza Khaz'ali wrote: > > I have to apologize for making this topic so lengthy. Is vpbjacobi supposed to work in parallel mode (with MPI)? > > > On 8/30/2018 1:32 AM, Smith, Barry F. wrote: >> >>> On Aug 29, 2018, at 3:48 PM, Ali Reza Khaz'ali wrote: >>> >>> >>>> Can you confirm if your code ran successfully with vpbjacobi and if the convergence history was very similar to that achieved using >>>> bjacobi ? >>> They have a very very very small difference (which is probably due to round-off errors), they generally behave the same. My code run successfully with both of them with the same convergence history. However, vpbjacobi seems a little faster. >> Great, this is exactly what to expect. I will put the branch into next for testing and then it will be in the master branch and in the next release. >> >> Barry >> >>> Many thanks, >>> Ali > > -- > Ali Reza Khaz?ali > Assistant Professor of Petroleum Engineering, > Department of Chemical Engineering > Isfahan University of Technology > Isfahan, Iran > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Thu Sep 6 17:04:41 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Thu, 6 Sep 2018 22:04:41 +0000 Subject: [petsc-users] DMForest Message-ID: Hello, I have a Petsc code solving PDE using FD method, and I generated the structures grid with DMDA. I was planning to implement adaptive mesh refinement to the same structures mesh but the documentation on the DMForest is not easy to find. I checked the petsc/src/dm/impls/forest/examples/tutorials but I found one tutorial which did not have so much information. I also found src/ts/examples/tutorials/ex11.c but it was for DMPlex, and finite volume method. I wonder if there is another example for DMForest? Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Sep 6 17:36:01 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 6 Sep 2018 22:36:01 +0000 Subject: [petsc-users] Use block Jacobi preconditioner with SNES In-Reply-To: <29tqhus78u94vls7v04q04fd.1536269950599@email.lge.com> References: <29tqhus78u94vls7v04q04fd.1536269950599@email.lge.com> Message-ID: <695DA8D6-DA75-45F7-B21A-382D0592E306@mcs.anl.gov> It is ready in the branch barry/add-mpivpbjacobi; please let me know if you have any difficulties Barry > On Sep 6, 2018, at 4:48 PM, arkhazali at cc.iut.ac.ir wrote: > > Yes, I'd be grateful if MPI support could be added, please. > > Many thanks, > Ali > > ------ Original message------ > From: Smith, Barry F. > Date: Fri, Sep 7, 2018 01:45 > To: Ali Reza Khaz'ali; > Cc: PETSc; > Subject:Re: [petsc-users] Use block Jacobi preconditioner with SNES > > I have not yet added this support, do you need it now? > > Barry > > > > On Sep 6, 2018, at 4:12 PM, Ali Reza Khaz'ali > wrote: > > > > I have to apologize for making this topic so lengthy. Is vpbjacobi supposed to work in parallel mode (with MPI)? > > > > > > On 8/30/2018 1:32 AM, Smith, Barry F. wrote: > >> > >>> On Aug 29, 2018, at 3:48 PM, Ali Reza Khaz'ali > wrote: > >>> > >>> > >>>> Can you confirm if your code ran successfully with vpbjacobi and if the convergence history was very similar to that achieved using > >>>> bjacobi ? > >>> They have a very very very small difference (which is probably due to round-off errors), they generally behave the same. My code run successfully with both of them with the same convergence history. However, vpbjacobi seems a little faster. > >> Great, this is exactly what to expect. I will put the branch into next for testing and then it will be in the master branch and in the next release. > >> > >> Barry > >> > >>> Many thanks, > >>> Ali > > > > -- > > Ali Reza Khaz?ali > > Assistant Professor of Petroleum Engineering, > > Department of Chemical Engineering > > Isfahan University of Technology > > Isfahan, Iran > > > > From bsmith at mcs.anl.gov Thu Sep 6 17:46:08 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 6 Sep 2018 22:46:08 +0000 Subject: [petsc-users] Problem compiling with 64bit PETSc In-Reply-To: <20a4fe0f-03bb-e6c4-a813-734eac73dba8@gmail.com> References: <2C69D0C3-5D83-4AD5-9B54-783A746DF0D9@anl.gov> <27433b40-13a9-0b4c-18b1-141b251c068f@gmail.com> <14BE4673-3EE5-48AC-96F1-288FCDA6517D@mcs.anl.gov> <81f3d266-59cd-4d5d-389e-039b6f1b5f72@gmail.com> <5822b9ea-05fd-3cfb-b365-07d26e81d895@gmail.com> <4506E418-0F49-469E-8246-728553365A01@gmail.com> <491AC580-D06F-46CF-86AB-8F80A12603C4@anl.gov> <20a4fe0f-03bb-e6c4-a813-734eac73dba8@gmail.com> Message-ID: <9CFD717C-19AC-49C4-B78B-97C7731856C6@mcs.anl.gov> I have added MPIU_INTEGER to the branch barry/add-mpiu_integer so you can make your MPI calls portable between 32 and 64 bit indices in Fortran. Please let me know if you have any difficulties. Barry > On Sep 5, 2018, at 11:58 PM, TAY wee-beng wrote: > > > On 6/9/2018 12:09 PM, Smith, Barry F. wrote: >> >>> On Sep 5, 2018, at 11:01 PM, Randall Mackie wrote: >>> >>> You can use PetscMPIInt for integers in MPI calls. >>> >>> Check petscsys.h for definitions of all of these. >> This is true but can be cumbersome because one may need to convert arrays of PetscInt to PetscMPIInt for MPI and then convert back after (with possible loss of precision). >> >> In C we have macros MPIU_INT that we use to indicate that the integer argument to the MPI call is 64 bit when 64 bit indices are used and 32 bit otherwise allowing users to write portable code that can just be reconfigured for 32 or 64 bit integers. I see we do not provide such a thing for Fortran; we should provide it. Unless Karl has coding time today we won't be able to get to it until tomorrow US time since it is already late in the US. >> >> Barry > Hi, > > That would be great! I've no problem waiting a few days. Does it mean that I can then use? > > call MPI_ALLGATHER(counter,1,MPIU_INT,counter_global,1,MPIU_INT,MPI_COMM_WORLD,ierr) > > Thanks! >>> Randy >>> >>> >>>> On Sep 5, 2018, at 8:56 PM, TAY wee-beng wrote: >>>> >>>> Hi, >>>> >>>> My code has some problems now after converting to 64bit indices. >>>> >>>> After debugging, I realised that I'm using: >>>> >>>> call MPI_ALLGATHER(counter,1,MPI_INTEGER,counter_global,1,MPI_INTEGER,MPI_COMM_WORLD,ierr) >>>> >>>> but now counter and counter_global are both 64bit integers. So should I change all mpi routine from MPI_INTEGER to MPI_INTEGER8? >>>> >>>> But if I switch back to using the 32bit PETSc, do I have to switch back again? In that case, does it mean I need to have 2 copies of my code - one to compile with PETSc 32, another to compile with PETSc 64? >>>> >>>> Is there an easier way? >>>> Thank you very much. >>>> >>>> Yours sincerely, >>>> >>>> ================================================ >>>> TAY Wee-Beng (Zheng Weiming) ??? >>>> Personal research webpage: >>>> http://tayweebeng.wixsite.com/website >>>> >>>> Youtube research showcase: >>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>> >>>> linkedin: >>>> www.linkedin.com/in/tay-weebeng >>>> >>>> ================================================ >>>> >>>> On 5/9/2018 6:25 PM, Matthew Knepley wrote: >>>>> On Wed, Sep 5, 2018 at 3:27 AM TAY wee-beng wrote: >>>>> >>>>> On 31/8/2018 10:43 AM, Smith, Barry F. wrote: >>>>>>> On Aug 30, 2018, at 9:40 PM, TAY wee-beng wrote: >>>>>>> >>>>>>> >>>>>>> On 31/8/2018 10:38 AM, Smith, Barry F. wrote: >>>>>>>> PetscReal is by default real(8) you can leave those alone >>>>>>>> >>>>>>>> Any integer you pass to a PETSc routine needs to be declared as PetscInt (not integer) otherwise the 64 bit indices stuff won't work. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> ok, I got it. Btw, is it advisable to change all integer in my code to PetscInt? >>>>>>> >>>>>>> Will it cause any conflict or waste a lot of memory? >>>>>>> >>>>>>> Or should I only change those related to PETSc? >>>>>> That is up to you. Since you probably pass the values between PETSc and non-PETSc part of the code it is probably easier just to make all the integer PetscInt instead. No performance difference that you can measure by keeping a few integer around. >>>>>> >>>>>> Barry >>>>> Hi, >>>>> >>>>> For some small parts of the code, it is preferred to use integer >>>>> instead. Btw, to force variable as integer, I can use int(aa). However, >>>>> I tried to force variable as PetscInt using PetscInt(aa) but it can't work. >>>>> >>>>> Is there any way I can make it work? >>>>> >>>>> I think you just define a PetscInt variable and use assignment. >>>>> >>>>> Matt >>>>> Thanks. >>>>>>> Thanks! >>>>>>>>> On Aug 30, 2018, at 9:35 PM, TAY wee-beng wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 31/8/2018 10:21 AM, Matthew Knepley wrote: >>>>>>>>>> On Thu, Aug 30, 2018 at 10:17 PM TAY wee-beng wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Due to my increase grid size, I have to go 64bit. I compiled the 64bit >>>>>>>>>> PETSc w/o error. However, when I tried to compile my code using the >>>>>>>>>> 64bit PETSc, I got the error below. May I know why is this so? >>>>>>>>>> >>>>>>>>>> What changes should I make? >>>>>>>>>> >>>>>>>>>> Is it possible that you did not declare some inputs as PetscInt, so the interface check is failing? >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I'm using the standard >>>>>>>>> >>>>>>>>> integer :: >>>>>>>>> >>>>>>>>> real(8) :: >>>>>>>>> >>>>>>>>> for some variables. For some others relating to PETSc, I use PetscInt. >>>>>>>>> >>>>>>>>> Should I change all to PetscInt and PetscReal? >>>>>>>>> >>>>>>>>> Currently, I use real(8) for all real values. If I change all to PetscReal, will PetscReal be real or real(8) by default? >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>>> [tsltaywb at nus02 ibm3d_IIB_mpi]$ make -f makefile_2018 >>>>>>>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>>>>>>>>> -g -ip -ipo -O3 -c -fPIC -save kinefunc.F90 >>>>>>>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>>>>>>>>> -g -ip -ipo -O3 -c -fPIC -save -w >>>>>>>>>> -I/home/users/nus/tsltaywb/propeller/lib/petsc-3.9.3_intel_2018_64bit_rel/include >>>>>>>>>> -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include >>>>>>>>>> global.F90 >>>>>>>>>> global.F90(979): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> -----^ >>>>>>>>>> global.F90(989): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> ---------^ >>>>>>>>>> global.F90(997): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> ---------^ >>>>>>>>>> global.F90(1005): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> ---------^ >>>>>>>>>> global.F90(1013): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> ---------^ >>>>>>>>>> global.F90(1021): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> ---------^ >>>>>>>>>> global.F90(1029): error #6285: There is no matching specific subroutine >>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>> call >>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>> ---------^ >>>>>>>>>> compilation aborted for global.F90 (code 1) >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Thank you very much. >>>>>>>>>> >>>>>>>>>> Yours sincerely, >>>>>>>>>> >>>>>>>>>> ================================================ >>>>>>>>>> TAY Wee-Beng (Zheng Weiming) ??? >>>>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>>>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>>>>>> ================================================ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ > From knepley at gmail.com Thu Sep 6 18:46:55 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 6 Sep 2018 19:46:55 -0400 Subject: [petsc-users] DMForest In-Reply-To: References: Message-ID: On Thu, Sep 6, 2018 at 6:04 PM Sepideh Kavousi wrote: > Hello, > > I have a Petsc code solving PDE using FD method, and I generated the > structures grid with DMDA. I was planning to implement adaptive mesh > refinement to the same structures mesh but the documentation on the > DMForest is not easy to find. I checked the > petsc/src/dm/impls/forest/examples/tutorials but I found one tutorial > which did not have so much information. I also found > src/ts/examples/tutorials/ex11.c but it was for DMPlex, and finite volume > method. > > I wonder if there is another example for DMForest? > > TS ex11 works for Plex or Forest. Thanks, Matt > Thanks, > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Sep 6 19:17:12 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 7 Sep 2018 00:17:12 +0000 Subject: [petsc-users] Problem compiling with 64bit PETSc In-Reply-To: <9CFD717C-19AC-49C4-B78B-97C7731856C6@mcs.anl.gov> References: <2C69D0C3-5D83-4AD5-9B54-783A746DF0D9@anl.gov> <27433b40-13a9-0b4c-18b1-141b251c068f@gmail.com> <14BE4673-3EE5-48AC-96F1-288FCDA6517D@mcs.anl.gov> <81f3d266-59cd-4d5d-389e-039b6f1b5f72@gmail.com> <5822b9ea-05fd-3cfb-b365-07d26e81d895@gmail.com> <4506E418-0F49-469E-8246-728553365A01@gmail.com> <491AC580-D06F-46CF-86AB-8F80A12603C4@anl.gov> <20a4fe0f-03bb-e6c4-a813-734eac73dba8@gmail.com> <9CFD717C-19AC-49C4-B78B-97C7731856C6@mcs.anl.gov> Message-ID: <9D2805E6-6D5A-42B6-AF5B-2B28C90E1155@mcs.anl.gov> Ignore this message; MPIU_INTEGER already exists you can just use it to replace MPI_INTEGER for places where you are passing PetscInt through MPI calls. Barry > On Sep 6, 2018, at 5:46 PM, Smith, Barry F. wrote: > > > I have added MPIU_INTEGER to the branch barry/add-mpiu_integer so you can make your MPI calls portable between 32 and 64 bit indices in Fortran. > > Please let me know if you have any difficulties. > > Barry > > >> On Sep 5, 2018, at 11:58 PM, TAY wee-beng wrote: >> >> >> On 6/9/2018 12:09 PM, Smith, Barry F. wrote: >>> >>>> On Sep 5, 2018, at 11:01 PM, Randall Mackie wrote: >>>> >>>> You can use PetscMPIInt for integers in MPI calls. >>>> >>>> Check petscsys.h for definitions of all of these. >>> This is true but can be cumbersome because one may need to convert arrays of PetscInt to PetscMPIInt for MPI and then convert back after (with possible loss of precision). >>> >>> In C we have macros MPIU_INT that we use to indicate that the integer argument to the MPI call is 64 bit when 64 bit indices are used and 32 bit otherwise allowing users to write portable code that can just be reconfigured for 32 or 64 bit integers. I see we do not provide such a thing for Fortran; we should provide it. Unless Karl has coding time today we won't be able to get to it until tomorrow US time since it is already late in the US. >>> >>> Barry >> Hi, >> >> That would be great! I've no problem waiting a few days. Does it mean that I can then use? >> >> call MPI_ALLGATHER(counter,1,MPIU_INT,counter_global,1,MPIU_INT,MPI_COMM_WORLD,ierr) >> >> Thanks! >>>> Randy >>>> >>>> >>>>> On Sep 5, 2018, at 8:56 PM, TAY wee-beng wrote: >>>>> >>>>> Hi, >>>>> >>>>> My code has some problems now after converting to 64bit indices. >>>>> >>>>> After debugging, I realised that I'm using: >>>>> >>>>> call MPI_ALLGATHER(counter,1,MPI_INTEGER,counter_global,1,MPI_INTEGER,MPI_COMM_WORLD,ierr) >>>>> >>>>> but now counter and counter_global are both 64bit integers. So should I change all mpi routine from MPI_INTEGER to MPI_INTEGER8? >>>>> >>>>> But if I switch back to using the 32bit PETSc, do I have to switch back again? In that case, does it mean I need to have 2 copies of my code - one to compile with PETSc 32, another to compile with PETSc 64? >>>>> >>>>> Is there an easier way? >>>>> Thank you very much. >>>>> >>>>> Yours sincerely, >>>>> >>>>> ================================================ >>>>> TAY Wee-Beng (Zheng Weiming) ??? >>>>> Personal research webpage: >>>>> http://tayweebeng.wixsite.com/website >>>>> >>>>> Youtube research showcase: >>>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>> >>>>> linkedin: >>>>> www.linkedin.com/in/tay-weebeng >>>>> >>>>> ================================================ >>>>> >>>>> On 5/9/2018 6:25 PM, Matthew Knepley wrote: >>>>>> On Wed, Sep 5, 2018 at 3:27 AM TAY wee-beng wrote: >>>>>> >>>>>> On 31/8/2018 10:43 AM, Smith, Barry F. wrote: >>>>>>>> On Aug 30, 2018, at 9:40 PM, TAY wee-beng wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 31/8/2018 10:38 AM, Smith, Barry F. wrote: >>>>>>>>> PetscReal is by default real(8) you can leave those alone >>>>>>>>> >>>>>>>>> Any integer you pass to a PETSc routine needs to be declared as PetscInt (not integer) otherwise the 64 bit indices stuff won't work. >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> ok, I got it. Btw, is it advisable to change all integer in my code to PetscInt? >>>>>>>> >>>>>>>> Will it cause any conflict or waste a lot of memory? >>>>>>>> >>>>>>>> Or should I only change those related to PETSc? >>>>>>> That is up to you. Since you probably pass the values between PETSc and non-PETSc part of the code it is probably easier just to make all the integer PetscInt instead. No performance difference that you can measure by keeping a few integer around. >>>>>>> >>>>>>> Barry >>>>>> Hi, >>>>>> >>>>>> For some small parts of the code, it is preferred to use integer >>>>>> instead. Btw, to force variable as integer, I can use int(aa). However, >>>>>> I tried to force variable as PetscInt using PetscInt(aa) but it can't work. >>>>>> >>>>>> Is there any way I can make it work? >>>>>> >>>>>> I think you just define a PetscInt variable and use assignment. >>>>>> >>>>>> Matt >>>>>> Thanks. >>>>>>>> Thanks! >>>>>>>>>> On Aug 30, 2018, at 9:35 PM, TAY wee-beng wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 31/8/2018 10:21 AM, Matthew Knepley wrote: >>>>>>>>>>> On Thu, Aug 30, 2018 at 10:17 PM TAY wee-beng wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Due to my increase grid size, I have to go 64bit. I compiled the 64bit >>>>>>>>>>> PETSc w/o error. However, when I tried to compile my code using the >>>>>>>>>>> 64bit PETSc, I got the error below. May I know why is this so? >>>>>>>>>>> >>>>>>>>>>> What changes should I make? >>>>>>>>>>> >>>>>>>>>>> Is it possible that you did not declare some inputs as PetscInt, so the interface check is failing? >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I'm using the standard >>>>>>>>>> >>>>>>>>>> integer :: >>>>>>>>>> >>>>>>>>>> real(8) :: >>>>>>>>>> >>>>>>>>>> for some variables. For some others relating to PETSc, I use PetscInt. >>>>>>>>>> >>>>>>>>>> Should I change all to PetscInt and PetscReal? >>>>>>>>>> >>>>>>>>>> Currently, I use real(8) for all real values. If I change all to PetscReal, will PetscReal be real or real(8) by default? >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>>> [tsltaywb at nus02 ibm3d_IIB_mpi]$ make -f makefile_2018 >>>>>>>>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>>>>>>>>>> -g -ip -ipo -O3 -c -fPIC -save kinefunc.F90 >>>>>>>>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>>>>>>>>>> -g -ip -ipo -O3 -c -fPIC -save -w >>>>>>>>>>> -I/home/users/nus/tsltaywb/propeller/lib/petsc-3.9.3_intel_2018_64bit_rel/include >>>>>>>>>>> -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include >>>>>>>>>>> global.F90 >>>>>>>>>>> global.F90(979): error #6285: There is no matching specific subroutine >>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>> call >>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>> -----^ >>>>>>>>>>> global.F90(989): error #6285: There is no matching specific subroutine >>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>> call >>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>> ---------^ >>>>>>>>>>> global.F90(997): error #6285: There is no matching specific subroutine >>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>> call >>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>> ---------^ >>>>>>>>>>> global.F90(1005): error #6285: There is no matching specific subroutine >>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>> call >>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>> ---------^ >>>>>>>>>>> global.F90(1013): error #6285: There is no matching specific subroutine >>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>> call >>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>> ---------^ >>>>>>>>>>> global.F90(1021): error #6285: There is no matching specific subroutine >>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>> call >>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>> ---------^ >>>>>>>>>>> global.F90(1029): error #6285: There is no matching specific subroutine >>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>> call >>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>> ---------^ >>>>>>>>>>> compilation aborted for global.F90 (code 1) >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Thank you very much. >>>>>>>>>>> >>>>>>>>>>> Yours sincerely, >>>>>>>>>>> >>>>>>>>>>> ================================================ >>>>>>>>>>> TAY Wee-Beng (Zheng Weiming) ??? >>>>>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>>>>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>>>>>>> ================================================ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >> > From zonexo at gmail.com Fri Sep 7 01:36:59 2018 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 7 Sep 2018 14:36:59 +0800 Subject: [petsc-users] Problem compiling with 64bit PETSc In-Reply-To: <9D2805E6-6D5A-42B6-AF5B-2B28C90E1155@mcs.anl.gov> References: <2C69D0C3-5D83-4AD5-9B54-783A746DF0D9@anl.gov> <27433b40-13a9-0b4c-18b1-141b251c068f@gmail.com> <14BE4673-3EE5-48AC-96F1-288FCDA6517D@mcs.anl.gov> <81f3d266-59cd-4d5d-389e-039b6f1b5f72@gmail.com> <5822b9ea-05fd-3cfb-b365-07d26e81d895@gmail.com> <4506E418-0F49-469E-8246-728553365A01@gmail.com> <491AC580-D06F-46CF-86AB-8F80A12603C4@anl.gov> <20a4fe0f-03bb-e6c4-a813-734eac73dba8@gmail.com> <9CFD717C-19AC-49C4-B78B-97C7731856C6@mcs.anl.gov> <9D2805E6-6D5A-42B6-AF5B-2B28C90E1155@mcs.anl.gov> Message-ID: <861949a1-3bc0-d23e-e585-474220d6a223@gmail.com> Hi, Thank you very much. I got it working now. Yours sincerely, ================================================ TAY Wee-Beng (Zheng Weiming) ??? Personal research webpage: http://tayweebeng.wixsite.com/website Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA linkedin: www.linkedin.com/in/tay-weebeng ================================================ On 7/9/2018 8:17 AM, Smith, Barry F. wrote: > Ignore this message; MPIU_INTEGER already exists you can just use it to replace MPI_INTEGER for places where you are passing PetscInt through MPI calls. > > Barry > > >> On Sep 6, 2018, at 5:46 PM, Smith, Barry F. wrote: >> >> >> I have added MPIU_INTEGER to the branch barry/add-mpiu_integer so you can make your MPI calls portable between 32 and 64 bit indices in Fortran. >> >> Please let me know if you have any difficulties. >> >> Barry >> >> >>> On Sep 5, 2018, at 11:58 PM, TAY wee-beng wrote: >>> >>> >>> On 6/9/2018 12:09 PM, Smith, Barry F. wrote: >>>>> On Sep 5, 2018, at 11:01 PM, Randall Mackie wrote: >>>>> >>>>> You can use PetscMPIInt for integers in MPI calls. >>>>> >>>>> Check petscsys.h for definitions of all of these. >>>> This is true but can be cumbersome because one may need to convert arrays of PetscInt to PetscMPIInt for MPI and then convert back after (with possible loss of precision). >>>> >>>> In C we have macros MPIU_INT that we use to indicate that the integer argument to the MPI call is 64 bit when 64 bit indices are used and 32 bit otherwise allowing users to write portable code that can just be reconfigured for 32 or 64 bit integers. I see we do not provide such a thing for Fortran; we should provide it. Unless Karl has coding time today we won't be able to get to it until tomorrow US time since it is already late in the US. >>>> >>>> Barry >>> Hi, >>> >>> That would be great! I've no problem waiting a few days. Does it mean that I can then use? >>> >>> call MPI_ALLGATHER(counter,1,MPIU_INT,counter_global,1,MPIU_INT,MPI_COMM_WORLD,ierr) >>> >>> Thanks! >>>>> Randy >>>>> >>>>> >>>>>> On Sep 5, 2018, at 8:56 PM, TAY wee-beng wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> My code has some problems now after converting to 64bit indices. >>>>>> >>>>>> After debugging, I realised that I'm using: >>>>>> >>>>>> call MPI_ALLGATHER(counter,1,MPI_INTEGER,counter_global,1,MPI_INTEGER,MPI_COMM_WORLD,ierr) >>>>>> >>>>>> but now counter and counter_global are both 64bit integers. So should I change all mpi routine from MPI_INTEGER to MPI_INTEGER8? >>>>>> >>>>>> But if I switch back to using the 32bit PETSc, do I have to switch back again? In that case, does it mean I need to have 2 copies of my code - one to compile with PETSc 32, another to compile with PETSc 64? >>>>>> >>>>>> Is there an easier way? >>>>>> Thank you very much. >>>>>> >>>>>> Yours sincerely, >>>>>> >>>>>> ================================================ >>>>>> TAY Wee-Beng (Zheng Weiming) ??? >>>>>> Personal research webpage: >>>>>> http://tayweebeng.wixsite.com/website >>>>>> >>>>>> Youtube research showcase: >>>>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>> >>>>>> linkedin: >>>>>> www.linkedin.com/in/tay-weebeng >>>>>> >>>>>> ================================================ >>>>>> >>>>>> On 5/9/2018 6:25 PM, Matthew Knepley wrote: >>>>>>> On Wed, Sep 5, 2018 at 3:27 AM TAY wee-beng wrote: >>>>>>> >>>>>>> On 31/8/2018 10:43 AM, Smith, Barry F. wrote: >>>>>>>>> On Aug 30, 2018, at 9:40 PM, TAY wee-beng wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 31/8/2018 10:38 AM, Smith, Barry F. wrote: >>>>>>>>>> PetscReal is by default real(8) you can leave those alone >>>>>>>>>> >>>>>>>>>> Any integer you pass to a PETSc routine needs to be declared as PetscInt (not integer) otherwise the 64 bit indices stuff won't work. >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> ok, I got it. Btw, is it advisable to change all integer in my code to PetscInt? >>>>>>>>> >>>>>>>>> Will it cause any conflict or waste a lot of memory? >>>>>>>>> >>>>>>>>> Or should I only change those related to PETSc? >>>>>>>> That is up to you. Since you probably pass the values between PETSc and non-PETSc part of the code it is probably easier just to make all the integer PetscInt instead. No performance difference that you can measure by keeping a few integer around. >>>>>>>> >>>>>>>> Barry >>>>>>> Hi, >>>>>>> >>>>>>> For some small parts of the code, it is preferred to use integer >>>>>>> instead. Btw, to force variable as integer, I can use int(aa). However, >>>>>>> I tried to force variable as PetscInt using PetscInt(aa) but it can't work. >>>>>>> >>>>>>> Is there any way I can make it work? >>>>>>> >>>>>>> I think you just define a PetscInt variable and use assignment. >>>>>>> >>>>>>> Matt >>>>>>> Thanks. >>>>>>>>> Thanks! >>>>>>>>>>> On Aug 30, 2018, at 9:35 PM, TAY wee-beng wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 31/8/2018 10:21 AM, Matthew Knepley wrote: >>>>>>>>>>>> On Thu, Aug 30, 2018 at 10:17 PM TAY wee-beng wrote: >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> Due to my increase grid size, I have to go 64bit. I compiled the 64bit >>>>>>>>>>>> PETSc w/o error. However, when I tried to compile my code using the >>>>>>>>>>>> 64bit PETSc, I got the error below. May I know why is this so? >>>>>>>>>>>> >>>>>>>>>>>> What changes should I make? >>>>>>>>>>>> >>>>>>>>>>>> Is it possible that you did not declare some inputs as PetscInt, so the interface check is failing? >>>>>>>>>>>> >>>>>>>>>>>> Matt >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I'm using the standard >>>>>>>>>>> >>>>>>>>>>> integer :: >>>>>>>>>>> >>>>>>>>>>> real(8) :: >>>>>>>>>>> >>>>>>>>>>> for some variables. For some others relating to PETSc, I use PetscInt. >>>>>>>>>>> >>>>>>>>>>> Should I change all to PetscInt and PetscReal? >>>>>>>>>>> >>>>>>>>>>> Currently, I use real(8) for all real values. If I change all to PetscReal, will PetscReal be real or real(8) by default? >>>>>>>>>>> >>>>>>>>>>> Thanks! >>>>>>>>>>>> [tsltaywb at nus02 ibm3d_IIB_mpi]$ make -f makefile_2018 >>>>>>>>>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>>>>>>>>>>> -g -ip -ipo -O3 -c -fPIC -save kinefunc.F90 >>>>>>>>>>>> /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 >>>>>>>>>>>> -g -ip -ipo -O3 -c -fPIC -save -w >>>>>>>>>>>> -I/home/users/nus/tsltaywb/propeller/lib/petsc-3.9.3_intel_2018_64bit_rel/include >>>>>>>>>>>> -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include >>>>>>>>>>>> global.F90 >>>>>>>>>>>> global.F90(979): error #6285: There is no matching specific subroutine >>>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>>> call >>>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>>> -----^ >>>>>>>>>>>> global.F90(989): error #6285: There is no matching specific subroutine >>>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>>> call >>>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>>> ---------^ >>>>>>>>>>>> global.F90(997): error #6285: There is no matching specific subroutine >>>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>>> call >>>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>>> ---------^ >>>>>>>>>>>> global.F90(1005): error #6285: There is no matching specific subroutine >>>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>>> call >>>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>>> ---------^ >>>>>>>>>>>> global.F90(1013): error #6285: There is no matching specific subroutine >>>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>>> call >>>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>>> ---------^ >>>>>>>>>>>> global.F90(1021): error #6285: There is no matching specific subroutine >>>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>>> call >>>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>>> ---------^ >>>>>>>>>>>> global.F90(1029): error #6285: There is no matching specific subroutine >>>>>>>>>>>> for this generic subroutine call. [DMDACREATE3D] >>>>>>>>>>>> call >>>>>>>>>>>> DMDACreate3d(MPI_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& >>>>>>>>>>>> ---------^ >>>>>>>>>>>> compilation aborted for global.F90 (code 1) >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Thank you very much. >>>>>>>>>>>> >>>>>>>>>>>> Yours sincerely, >>>>>>>>>>>> >>>>>>>>>>>> ================================================ >>>>>>>>>>>> TAY Wee-Beng (Zheng Weiming) ??? >>>>>>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>>>>>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>>>>>>>> ================================================ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>> >>>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ From zonexo at gmail.com Fri Sep 7 01:44:00 2018 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 7 Sep 2018 14:44:00 +0800 Subject: [petsc-users] Unable to read in values thru namelist in Fortran after using PETSc 64bit in linux Message-ID: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com> Hi, I found that I am unable to read in values thru namelist in Fortran after using PETSc 64bit in linux. I have a parameter txt file which is read in using namelist in Fortran? namelist /body_input/ no_body, convex_body, motion_type, hover, wing_config? ... open (unit = 44 , FILE = "ibm3d_input.txt" , status = "old", iostat = openstatus(4)) ??? ??? if (openstatus(4) > 0) then ??? ??? ??? print *, "ibm3d_input file not present or wrong filename." ??? ??? ??? stop ??? ??? end if ??? ??? read (44,nml = solver_input) ??? ??? read (44,nml = grid_input) ??? ??? read (44,nml = body_input)... After using PETSc 64bit, my code aborts and I realise that it is because the values have became NaN. Strangely, it does not occur in windows with VS2008. I wonder if it's a bug with the Intel Fortran compiler 2018. Anyone has similar experience? -- Thank you very much. Yours sincerely, ================================================ TAY Wee-Beng (Zheng Weiming) ??? Personal research webpage: http://tayweebeng.wixsite.com/website Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA linkedin: www.linkedin.com/in/tay-weebeng ================================================ From bhatiamanav at gmail.com Fri Sep 7 01:44:19 2018 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Fri, 7 Sep 2018 01:44:19 -0500 Subject: [petsc-users] Slepc solver not converging for generalized eigenproblem Message-ID: Hi, I am attempting to compute the eigenvalues of the generalized nonhermitian eigenproblem M1 x = lambda M0 x M1 and M0 are in the attached text files. The solver is unable to converge with 300 iterations. I have played around with the max it (as high as 1500) and it still is unable to converge. What would be the best way to work around this issue? I would appreciate your help. Thanks, Manav EPS Object: 1 MPI processes type: krylovschur 50% of basis vectors kept after restart using the locking variant problem type: generalized non-symmetric eigenvalue problem selected portion of the spectrum: smallest eigenvalues in magnitude number of eigenvalues (nev): 10 number of column vectors (ncv): 50 maximum dimension of projected problem (mpd): 50 maximum number of iterations: 300 tolerance: 1e-08 convergence test: relative to the eigenvalue BV Object: 1 MPI processes type: svec 51 columns of global length 178 vector orthogonalization method: classical Gram-Schmidt orthogonalization refinement: if needed (eta: 0.7071) block orthogonalization method: GS doing matmult as a single matrix-matrix product DS Object: 1 MPI processes type: nhep ST Object: 1 MPI processes type: shift shift: 0. number of matrices: 2 all matrices have different nonzero pattern KSP Object: (st_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-08, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (st_) 1 MPI processes type: lu out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 0., needed 0. Factored matrix follows: Mat Object: 1 MPI processes type: seqdense rows=178, cols=178 package used to perform factorization: petsc total: nonzeros=31684, allocated nonzeros=31684 total number of mallocs used during MatSetValues calls =0 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqdense rows=178, cols=178 total: nonzeros=31684, allocated nonzeros=31684 total number of mallocs used during MatSetValues calls =0 Linear eigensolve did not converge due to DIVERGED_ITS; iterations 300 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: M0.txt URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: M1.txt URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Fri Sep 7 02:04:00 2018 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 7 Sep 2018 09:04:00 +0200 Subject: [petsc-users] Slepc solver not converging for generalized eigenproblem In-Reply-To: References: Message-ID: The matrices are not in a format that I can easily load. Anyway, I see that you are using "smallest magnitude", which will give bad convergence in most cases. It is specially not recommended in generalized eigenproblems: instead of smallest eigenvalues of (M0,M1) you should compute largest eigenvalues of (M1,M0), and evaluate the reciprocals of the eigenvalues. Alternatively, use shift-and-invert to compute eigenvalues of (M0,M1) closest to the target sigma=0. If M0 is singular, then use a small nonzero value for sigma. Jose > El 7 sept 2018, a las 8:44, Manav Bhatia escribi?: > > Hi, > > I am attempting to compute the eigenvalues of the generalized nonhermitian eigenproblem > > M1 x = lambda M0 x > > M1 and M0 are in the attached text files. The solver is unable to converge with 300 iterations. I have played around with the max it (as high as 1500) and it still is unable to converge. > > What would be the best way to work around this issue? > > I would appreciate your help. > > Thanks, > Manav > > > > > > EPS Object: 1 MPI processes > type: krylovschur > 50% of basis vectors kept after restart > using the locking variant > problem type: generalized non-symmetric eigenvalue problem > selected portion of the spectrum: smallest eigenvalues in magnitude > number of eigenvalues (nev): 10 > number of column vectors (ncv): 50 > maximum dimension of projected problem (mpd): 50 > maximum number of iterations: 300 > tolerance: 1e-08 > convergence test: relative to the eigenvalue > BV Object: 1 MPI processes > type: svec > 51 columns of global length 178 > vector orthogonalization method: classical Gram-Schmidt > orthogonalization refinement: if needed (eta: 0.7071) > block orthogonalization method: GS > doing matmult as a single matrix-matrix product > DS Object: 1 MPI processes > type: nhep > ST Object: 1 MPI processes > type: shift > shift: 0. > number of matrices: 2 > all matrices have different nonzero pattern > KSP Object: (st_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-08, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (st_) 1 MPI processes > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 0., needed 0. > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqdense > rows=178, cols=178 > package used to perform factorization: petsc > total: nonzeros=31684, allocated nonzeros=31684 > total number of mallocs used during MatSetValues calls =0 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqdense > rows=178, cols=178 > total: nonzeros=31684, allocated nonzeros=31684 > total number of mallocs used during MatSetValues calls =0 > Linear eigensolve did not converge due to DIVERGED_ITS; iterations 300 From bhatiamanav at gmail.com Fri Sep 7 02:06:20 2018 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Fri, 7 Sep 2018 02:06:20 -0500 Subject: [petsc-users] Slepc solver not converging for generalized eigenproblem In-Reply-To: References: Message-ID: Thanks, Jose. I was able to get this to work with the largest eigenvalues option. Thanks! -Manav > On Sep 7, 2018, at 2:04 AM, Jose E. Roman wrote: > > The matrices are not in a format that I can easily load. Anyway, I see that you are using "smallest magnitude", which will give bad convergence in most cases. It is specially not recommended in generalized eigenproblems: instead of smallest eigenvalues of (M0,M1) you should compute largest eigenvalues of (M1,M0), and evaluate the reciprocals of the eigenvalues. Alternatively, use shift-and-invert to compute eigenvalues of (M0,M1) closest to the target sigma=0. If M0 is singular, then use a small nonzero value for sigma. > > Jose > > >> El 7 sept 2018, a las 8:44, Manav Bhatia escribi?: >> >> Hi, >> >> I am attempting to compute the eigenvalues of the generalized nonhermitian eigenproblem >> >> M1 x = lambda M0 x >> >> M1 and M0 are in the attached text files. The solver is unable to converge with 300 iterations. I have played around with the max it (as high as 1500) and it still is unable to converge. >> >> What would be the best way to work around this issue? >> >> I would appreciate your help. >> >> Thanks, >> Manav >> >> >> >> >> >> EPS Object: 1 MPI processes >> type: krylovschur >> 50% of basis vectors kept after restart >> using the locking variant >> problem type: generalized non-symmetric eigenvalue problem >> selected portion of the spectrum: smallest eigenvalues in magnitude >> number of eigenvalues (nev): 10 >> number of column vectors (ncv): 50 >> maximum dimension of projected problem (mpd): 50 >> maximum number of iterations: 300 >> tolerance: 1e-08 >> convergence test: relative to the eigenvalue >> BV Object: 1 MPI processes >> type: svec >> 51 columns of global length 178 >> vector orthogonalization method: classical Gram-Schmidt >> orthogonalization refinement: if needed (eta: 0.7071) >> block orthogonalization method: GS >> doing matmult as a single matrix-matrix product >> DS Object: 1 MPI processes >> type: nhep >> ST Object: 1 MPI processes >> type: shift >> shift: 0. >> number of matrices: 2 >> all matrices have different nonzero pattern >> KSP Object: (st_) 1 MPI processes >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-08, absolute=1e-50, divergence=10000. >> left preconditioning >> using NONE norm type for convergence test >> PC Object: (st_) 1 MPI processes >> type: lu >> out-of-place factorization >> tolerance for zero pivot 2.22045e-14 >> matrix ordering: nd >> factor fill ratio given 0., needed 0. >> Factored matrix follows: >> Mat Object: 1 MPI processes >> type: seqdense >> rows=178, cols=178 >> package used to perform factorization: petsc >> total: nonzeros=31684, allocated nonzeros=31684 >> total number of mallocs used during MatSetValues calls =0 >> linear system matrix = precond matrix: >> Mat Object: 1 MPI processes >> type: seqdense >> rows=178, cols=178 >> total: nonzeros=31684, allocated nonzeros=31684 >> total number of mallocs used during MatSetValues calls =0 >> Linear eigensolve did not converge due to DIVERGED_ITS; iterations 300 > From yjwu16 at gmail.com Fri Sep 7 05:39:26 2018 From: yjwu16 at gmail.com (Yingjie Wu) Date: Fri, 7 Sep 2018 18:39:26 +0800 Subject: [petsc-users] Approximation of matrix vector product by finite difference method In-Reply-To: References: Message-ID: Thank you very much for your reply. I'm a bit confused if I use what you recommend: - snes_fd -pc_type lu means that I explicitly construct the Jacobian matrix using the finite difference method, construct the precondition matrix using the completely LU decomposition, and solve the step size \ delta x with the GMRES method (default). In fact, what I want to use is to approximate the vector product of a matrix with finite difference, so that the explicit construction of Jacobian matrices can be avoided. If so, should I use MatrixFreeMethod? How should I set it up? If I want to set up precondition, what do I need to add? In addition, I want to output variables in each nolinear step. What should I add code to make SNES step by step? There may be many problems, but they bother me very much. I am looking forward to your reply. Thanks, Yingjie Matthew Knepley ?2018?9?6??? ??10:34??? > On Thu, Sep 6, 2018 at 4:47 AM Yingjie Wu wrote: > >> Dear Petsc developer: >> Hi, >> Thank you for your previous help. >> I recently modeled on PETSc's SNES example and wrote a computer program >> myself. This program is mainly for solving nonlinear equations of thermal >> hydraulics. >> >> ?????T? - ?_y??*Cp*u? - T_source = 0 >> >> w*?*u = ?g - ?_y?P? >> >> ??? 1/w * ?P ? = - ?? ?g / w ? >> >> Where P, T and u are variables, the distribution represents pressure, >> temperature and velocity. The rest are nonlinear physical parameters and >> constants. >> >> Because the program is very preliminary, so I use - snes_mf so that I can >> save the part of writing to calculate the Jacobian matrix. >> >> After compiling and passing, I found that the residual function had not >> dropped to a small enough level, but the program stopped, as follows: >> >> Setting Up: -snes_mf -snes_monitor -draw_pause 10 -snes_view >> >> > First, do not use -snes_mf. It is not for testing, but for sophisticated > use. The first option you > might try is > > -snes_fd -pc_type lu > > That uses a full Jacobian and LU factorization for a direct solve. Always > run the solve using > > -snes_view -snes_converged_reason -snes_monitor -ksp_converged_reason > -ksp_monitor_true_residual > > When that gets too expensive, you can try > > -snes_fd_color -snes_fd_color_use_mat -mat_coloring_type greedy > > but that requires you to preallocate the Jacobian matrix correctly. > > Thanks, > > Matt > >> 0 SNES Function norm 3.724996516631e+09 >> >> 1 SNES Function norm 2.194322909557e+09 >> >> 2 SNES Function norm 1.352051559826e+09 >> >> 3 SNES Function norm 1.522311916217e+08 >> >> SNES Object: 1 MPI processes >> >> type: newtonls >> >> maximum iterations=50, maximum function evaluations=10000 >> >> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >> >> total number of linear solver iterations=1298 >> >> total number of function evaluations=11679 >> >> norm schedule ALWAYS >> >> SNESLineSearch Object: 1 MPI processes >> >> type: bt >> >> interpolation: cubic >> >> alpha=1.000000e-04 >> >> maxstep=1.000000e+08, minlambda=1.000000e-12 >> >> tolerances: relative=1.000000e-08, absolute=1.000000e-15, >> lambda=1.000000e-08 >> >> maximum iterations=40 >> >> KSP Object: 1 MPI processes >> >> type: gmres >> >> restart=30, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> >> happy breakdown tolerance 1e-30 >> >> maximum iterations=10000, initial guess is zero >> >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> >> left preconditioning >> >> using PRECONDITIONED norm type for convergence test >> >> PC Object: 1 MPI processes >> >> type: none >> >> linear system matrix = precond matrix: >> >> Mat Object: 1 MPI processes >> >> type: mffd >> >> rows=300, cols=300 >> >> Matrix-free approximation: >> >> err=1.49012e-08 (relative error in function evaluation) >> >> Using wp compute h routine >> >> Does not compute normU >> >> I would like to know why the residual function can not continue to >> decline, and why the program will stop before convergence. >> I do not know much about the convergence criteria and convergence rules >> of PETSc for solving nonlinear equations. I hope I can get your help. >> I'm looking forward to your reply~ >> >> Thanks, >> Yingjie >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Sep 7 08:16:45 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 7 Sep 2018 09:16:45 -0400 Subject: [petsc-users] Approximation of matrix vector product by finite difference method In-Reply-To: References: Message-ID: On Fri, Sep 7, 2018 at 6:39 AM Yingjie Wu wrote: > Thank you very much for your reply. > > I'm a bit confused if I use what you recommend: - snes_fd -pc_type lu > means that I explicitly construct the Jacobian matrix using the finite > difference method, construct the precondition matrix using the completely > LU decomposition, and solve the step size \ delta x with the GMRES method > (default). > You should do this first in order to check that your code reproduces the right physics. Then you should do -snes_fd_color -snes_fd_color_use_mat -mat_coloring_type greedy for larger problems. Finally, if you want to go even larger, then you use -snes_mf and a user preconditioner. However, there is not much benefit of MF with a low order method because you will have to form some preconditioner. Thanks, Matt > In fact, what I want to use is to approximate the vector product of a > matrix with finite difference, so that the explicit construction of > Jacobian matrices can be avoided. If so, should I use MatrixFreeMethod? How > should I set it up? If I want to set up precondition, what do I need to > add? > > In addition, I want to output variables in each nolinear step. What should > I add code to make SNES step by step? > There may be many problems, but they bother me very much. I am looking > forward to your reply. > > Thanks, > Yingjie > > > Matthew Knepley ?2018?9?6??? ??10:34??? > >> On Thu, Sep 6, 2018 at 4:47 AM Yingjie Wu wrote: >> >>> Dear Petsc developer: >>> Hi, >>> Thank you for your previous help. >>> I recently modeled on PETSc's SNES example and wrote a computer program >>> myself. This program is mainly for solving nonlinear equations of thermal >>> hydraulics. >>> >>> ?????T? - ?_y??*Cp*u? - T_source = 0 >>> >>> w*?*u = ?g - ?_y?P? >>> >>> ??? 1/w * ?P ? = - ?? ?g / w ? >>> >>> Where P, T and u are variables, the distribution represents pressure, >>> temperature and velocity. The rest are nonlinear physical parameters and >>> constants. >>> >>> Because the program is very preliminary, so I use - snes_mf so that I >>> can save the part of writing to calculate the Jacobian matrix. >>> >>> After compiling and passing, I found that the residual function had not >>> dropped to a small enough level, but the program stopped, as follows: >>> >>> Setting Up: -snes_mf -snes_monitor -draw_pause 10 -snes_view >>> >>> >> First, do not use -snes_mf. It is not for testing, but for sophisticated >> use. The first option you >> might try is >> >> -snes_fd -pc_type lu >> >> That uses a full Jacobian and LU factorization for a direct solve. Always >> run the solve using >> >> -snes_view -snes_converged_reason -snes_monitor -ksp_converged_reason >> -ksp_monitor_true_residual >> >> When that gets too expensive, you can try >> >> -snes_fd_color -snes_fd_color_use_mat -mat_coloring_type greedy >> >> but that requires you to preallocate the Jacobian matrix correctly. >> >> Thanks, >> >> Matt >> >>> 0 SNES Function norm 3.724996516631e+09 >>> >>> 1 SNES Function norm 2.194322909557e+09 >>> >>> 2 SNES Function norm 1.352051559826e+09 >>> >>> 3 SNES Function norm 1.522311916217e+08 >>> >>> SNES Object: 1 MPI processes >>> >>> type: newtonls >>> >>> maximum iterations=50, maximum function evaluations=10000 >>> >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>> >>> total number of linear solver iterations=1298 >>> >>> total number of function evaluations=11679 >>> >>> norm schedule ALWAYS >>> >>> SNESLineSearch Object: 1 MPI processes >>> >>> type: bt >>> >>> interpolation: cubic >>> >>> alpha=1.000000e-04 >>> >>> maxstep=1.000000e+08, minlambda=1.000000e-12 >>> >>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, >>> lambda=1.000000e-08 >>> >>> maximum iterations=40 >>> >>> KSP Object: 1 MPI processes >>> >>> type: gmres >>> >>> restart=30, using Classical (unmodified) Gram-Schmidt >>> Orthogonalization with no iterative refinement >>> >>> happy breakdown tolerance 1e-30 >>> >>> maximum iterations=10000, initial guess is zero >>> >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >>> >>> left preconditioning >>> >>> using PRECONDITIONED norm type for convergence test >>> >>> PC Object: 1 MPI processes >>> >>> type: none >>> >>> linear system matrix = precond matrix: >>> >>> Mat Object: 1 MPI processes >>> >>> type: mffd >>> >>> rows=300, cols=300 >>> >>> Matrix-free approximation: >>> >>> err=1.49012e-08 (relative error in function evaluation) >>> >>> Using wp compute h routine >>> >>> Does not compute normU >>> >>> I would like to know why the residual function can not continue to >>> decline, and why the program will stop before convergence. >>> I do not know much about the convergence criteria and convergence rules >>> of PETSc for solving nonlinear equations. I hope I can get your help. >>> I'm looking forward to your reply~ >>> >>> Thanks, >>> Yingjie >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Sep 7 12:14:20 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 7 Sep 2018 17:14:20 +0000 Subject: [petsc-users] Unable to read in values thru namelist in Fortran after using PETSc 64bit in linux In-Reply-To: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com> References: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com> Message-ID: <0AC1BA91-75F8-4AAB-A803-832B089A2C44@anl.gov> You can try valgrind http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Barry > On Sep 7, 2018, at 1:44 AM, TAY wee-beng wrote: > > Hi, > > I found that I am unable to read in values thru namelist in Fortran after using PETSc 64bit in linux. > > I have a parameter txt file which is read in using namelist in Fortran? > > namelist /body_input/ no_body, convex_body, motion_type, hover, wing_config ... > > open (unit = 44 , FILE = "ibm3d_input.txt" , status = "old", iostat = openstatus(4)) > > if (openstatus(4) > 0) then > > print *, "ibm3d_input file not present or wrong filename." > > stop > > end if > > read (44,nml = solver_input) > > read (44,nml = grid_input) > > read (44,nml = body_input)... > > > After using PETSc 64bit, my code aborts and I realise that it is because the values have became NaN. Strangely, it does not occur in windows with VS2008. > > I wonder if it's a bug with the Intel Fortran compiler 2018. > > Anyone has similar experience? > > -- > Thank you very much. > > Yours sincerely, > > ================================================ > TAY Wee-Beng (Zheng Weiming) ??? > Personal research webpage: http://tayweebeng.wixsite.com/website > Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > linkedin: www.linkedin.com/in/tay-weebeng > ================================================ > From bsmith at mcs.anl.gov Fri Sep 7 12:25:25 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 7 Sep 2018 17:25:25 +0000 Subject: [petsc-users] Approximation of matrix vector product by finite difference method In-Reply-To: References: Message-ID: <978702C1-63DD-4471-A479-8D2E880554AA@anl.gov> > On Sep 7, 2018, at 5:39 AM, Yingjie Wu wrote: > > Thank you very much for your reply. > > I'm a bit confused if I use what you recommend: - snes_fd -pc_type lu means that I explicitly construct the Jacobian matrix using the finite difference method, construct the precondition matrix using the completely LU decomposition, and solve the step size \ delta x with the GMRES method (default). > > In fact, what I want to use is to approximate the vector product of a matrix with finite difference, so that the explicit construction of Jacobian matrices can be avoided. If so, should I use MatrixFreeMethod? How should I set it up? If I want to set up precondition, what do I need to add? > > In addition, I want to output variables in each nolinear step. What should I add code to make SNES step by step? SNESMonitorSet() > There may be many problems, but they bother me very much. I am looking forward to your reply. > > Thanks, > Yingjie > > > Matthew Knepley ?2018?9?6??? ??10:34??? > On Thu, Sep 6, 2018 at 4:47 AM Yingjie Wu wrote: > Dear Petsc developer: > Hi, > Thank you for your previous help. > I recently modeled on PETSc's SNES example and wrote a computer program myself. This program is mainly for solving nonlinear equations of thermal hydraulics. > ?????T? - ?_y??*Cp*u? - T_source = 0 > > w*?*u = ?g - ?_y?P? > > ??? 1/w * ?P ? = - ?? ?g / w ? > Where P, T and u are variables, the distribution represents pressure, temperature and velocity. The rest are nonlinear physical parameters and constants. > Because the program is very preliminary, so I use - snes_mf so that I can save the part of writing to calculate the Jacobian matrix. > After compiling and passing, I found that the residual function had not dropped to a small enough level, but the program stopped, as follows: > Setting Up: -snes_mf -snes_monitor -draw_pause 10 -snes_view > > First, do not use -snes_mf. It is not for testing, but for sophisticated use. The first option you > might try is > > -snes_fd -pc_type lu > > That uses a full Jacobian and LU factorization for a direct solve. Always run the solve using > > -snes_view -snes_converged_reason -snes_monitor -ksp_converged_reason -ksp_monitor_true_residual > > When that gets too expensive, you can try > > -snes_fd_color -snes_fd_color_use_mat -mat_coloring_type greedy > > but that requires you to preallocate the Jacobian matrix correctly. > > Thanks, > > Matt > 0 SNES Function norm 3.724996516631e+09 > > 1 SNES Function norm 2.194322909557e+09 > > 2 SNES Function norm 1.352051559826e+09 > > 3 SNES Function norm 1.522311916217e+08 > > SNES Object: 1 MPI processes > > type: newtonls > > maximum iterations=50, maximum function evaluations=10000 > > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > > total number of linear solver iterations=1298 > > total number of function evaluations=11679 > > norm schedule ALWAYS > > SNESLineSearch Object: 1 MPI processes > > type: bt > > interpolation: cubic > > alpha=1.000000e-04 > > maxstep=1.000000e+08, minlambda=1.000000e-12 > > tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 > > maximum iterations=40 > > KSP Object: 1 MPI processes > > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: none > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: mffd > > rows=300, cols=300 > > Matrix-free approximation: > > err=1.49012e-08 (relative error in function evaluation) > > Using wp compute h routine > > Does not compute normU > > I would like to know why the residual function can not continue to decline, and why the program will stop before convergence. > I do not know much about the convergence criteria and convergence rules of PETSc for solving nonlinear equations. I hope I can get your help. > I'm looking forward to your reply~ > > Thanks, > Yingjie > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From arkhazali at cc.iut.ac.ir Fri Sep 7 15:22:50 2018 From: arkhazali at cc.iut.ac.ir (Ali Reza Khaz'ali) Date: Sat, 8 Sep 2018 00:52:50 +0430 Subject: [petsc-users] Use block Jacobi preconditioner with SNES In-Reply-To: <695DA8D6-DA75-45F7-B21A-382D0592E306@mcs.anl.gov> References: <29tqhus78u94vls7v04q04fd.1536269950599@email.lge.com> <695DA8D6-DA75-45F7-B21A-382D0592E306@mcs.anl.gov> Message-ID: <5ef51e40-6a33-3f56-ebbe-d6cb3a19060c@cc.iut.ac.ir> Thanks a lot. It is working as expected. However, splitting a block matrix over MPI processes, sometimes results in splitting one block over 2 (or possibly more) processors. It causes the block Jacobi preconditioner (+Linear solver) to fail. Is there any way in PETSc to enforce a pre-specified array/matrix distribution over MPI processes? I want to distribute the array/matrix entries, so that no block is divided between two MPI processes. Many Thanks, Ali On 9/7/2018 3:06 AM, Smith, Barry F. wrote: > It is ready in the branch barry/add-mpivpbjacobi; please let me know if you have any difficulties > > Barry > > >> On Sep 6, 2018, at 4:48 PM, arkhazali at cc.iut.ac.ir wrote: >> >> Yes, I'd be grateful if MPI support could be added, please. >> >> Many thanks, >> Ali >> >> ------ Original message------ >> From: Smith, Barry F. >> Date: Fri, Sep 7, 2018 01:45 >> To: Ali Reza Khaz'ali; >> Cc: PETSc; >> Subject:Re: [petsc-users] Use block Jacobi preconditioner with SNES >> >> I have not yet added this support, do you need it now? >> >> Barry >> >> >>> On Sep 6, 2018, at 4:12 PM, Ali Reza Khaz'ali >> wrote: >>> I have to apologize for making this topic so lengthy. Is vpbjacobi supposed to work in parallel mode (with MPI)? >>> >>> >>> On 8/30/2018 1:32 AM, Smith, Barry F. wrote: >>>>> On Aug 29, 2018, at 3:48 PM, Ali Reza Khaz'ali >> wrote: >>>>> >>>>>> Can you confirm if your code ran successfully with vpbjacobi and if the convergence history was very similar to that achieved using >>>>>> bjacobi ? >>>>> They have a very very very small difference (which is probably due to round-off errors), they generally behave the same. My code run successfully with both of them with the same convergence history. However, vpbjacobi seems a little faster. >>>> Great, this is exactly what to expect. I will put the branch into next for testing and then it will be in the master branch and in the next release. >>>> >>>> Barry >>>> >>>>> Many thanks, >>>>> Ali >>> -- >>> Ali Reza Khaz?ali >>> Assistant Professor of Petroleum Engineering, >>> Department of Chemical Engineering >>> Isfahan University of Technology >>> Isfahan, Iran >>> >> -- Ali Reza Khaz?ali Assistant Professor of Petroleum Engineering, Department of Chemical Engineering Isfahan University of Technology Isfahan, Iran From bsmith at mcs.anl.gov Fri Sep 7 15:41:55 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 7 Sep 2018 20:41:55 +0000 Subject: [petsc-users] Use block Jacobi preconditioner with SNES In-Reply-To: <5ef51e40-6a33-3f56-ebbe-d6cb3a19060c@cc.iut.ac.ir> References: <29tqhus78u94vls7v04q04fd.1536269950599@email.lge.com> <695DA8D6-DA75-45F7-B21A-382D0592E306@mcs.anl.gov> <5ef51e40-6a33-3f56-ebbe-d6cb3a19060c@cc.iut.ac.ir> Message-ID: > On Sep 7, 2018, at 3:22 PM, Ali Reza Khaz'ali wrote: > > Thanks a lot. It is working as expected. However, splitting a block matrix over MPI processes, sometimes results in splitting one block over 2 (or possibly more) processors. It causes the block Jacobi preconditioner (+Linear solver) to fail. Is there any way in PETSc to enforce a pre-specified array/matrix distribution over MPI processes? I want to distribute the array/matrix entries, so that no block is divided between two MPI processes. How are you creating the matrix? If you use MatSetSizes() or MatCreateMPIAIJ() then just pass in the local size, not the global size and so long as you set an appropriate local size you can insure each block is associated with a single process. Of course you need to use the same local size when creating the vectors. Barry > > > Many Thanks, > > Ali > > > On 9/7/2018 3:06 AM, Smith, Barry F. wrote: >> It is ready in the branch barry/add-mpivpbjacobi; please let me know if you have any difficulties >> >> Barry >> >> >>> On Sep 6, 2018, at 4:48 PM, arkhazali at cc.iut.ac.ir wrote: >>> >>> Yes, I'd be grateful if MPI support could be added, please. >>> >>> Many thanks, >>> Ali >>> >>> ------ Original message------ >>> From: Smith, Barry F. >>> Date: Fri, Sep 7, 2018 01:45 >>> To: Ali Reza Khaz'ali; >>> Cc: PETSc; >>> Subject:Re: [petsc-users] Use block Jacobi preconditioner with SNES >>> >>> I have not yet added this support, do you need it now? >>> >>> Barry >>> >>> >>>> On Sep 6, 2018, at 4:12 PM, Ali Reza Khaz'ali >>> wrote: >>>> I have to apologize for making this topic so lengthy. Is vpbjacobi supposed to work in parallel mode (with MPI)? >>>> >>>> >>>> On 8/30/2018 1:32 AM, Smith, Barry F. wrote: >>>>>> On Aug 29, 2018, at 3:48 PM, Ali Reza Khaz'ali >>> wrote: >>>>>> >>>>>>> Can you confirm if your code ran successfully with vpbjacobi and if the convergence history was very similar to that achieved using >>>>>>> bjacobi ? >>>>>> They have a very very very small difference (which is probably due to round-off errors), they generally behave the same. My code run successfully with both of them with the same convergence history. However, vpbjacobi seems a little faster. >>>>> Great, this is exactly what to expect. I will put the branch into next for testing and then it will be in the master branch and in the next release. >>>>> >>>>> Barry >>>>> >>>>>> Many thanks, >>>>>> Ali >>>> -- >>>> Ali Reza Khaz?ali >>>> Assistant Professor of Petroleum Engineering, >>>> Department of Chemical Engineering >>>> Isfahan University of Technology >>>> Isfahan, Iran >>>> >>> > > -- > Ali Reza Khaz?ali > Assistant Professor of Petroleum Engineering, > Department of Chemical Engineering > Isfahan University of Technology > Isfahan, Iran > From arkhazali at cc.iut.ac.ir Sat Sep 8 12:23:21 2018 From: arkhazali at cc.iut.ac.ir (Ali Reza Khaz'ali) Date: Sat, 8 Sep 2018 21:53:21 +0430 Subject: [petsc-users] Use block Jacobi preconditioner with SNES In-Reply-To: References: <29tqhus78u94vls7v04q04fd.1536269950599@email.lge.com> <695DA8D6-DA75-45F7-B21A-382D0592E306@mcs.anl.gov> <5ef51e40-6a33-3f56-ebbe-d6cb3a19060c@cc.iut.ac.ir> Message-ID: <73461a02-0028-6e8d-afd4-76a7a3d2f971@cc.iut.ac.ir> I am using MatCreateAIJ. After applying you advice, my code is working without problems. Thanks again! Best Wishes, Ali On 9/8/2018 1:11 AM, Smith, Barry F. wrote: > >> On Sep 7, 2018, at 3:22 PM, Ali Reza Khaz'ali wrote: >> >> Thanks a lot. It is working as expected. However, splitting a block matrix over MPI processes, sometimes results in splitting one block over 2 (or possibly more) processors. It causes the block Jacobi preconditioner (+Linear solver) to fail. Is there any way in PETSc to enforce a pre-specified array/matrix distribution over MPI processes? I want to distribute the array/matrix entries, so that no block is divided between two MPI processes. > How are you creating the matrix? If you use MatSetSizes() or MatCreateMPIAIJ() then just pass in the local size, not the global size and so long as you set an appropriate local size you can insure each block is associated with a single process. Of course you need to use the same local size when creating the vectors. > > > Barry > >> >> Many Thanks, >> >> Ali >> >> >> On 9/7/2018 3:06 AM, Smith, Barry F. wrote: >>> It is ready in the branch barry/add-mpivpbjacobi; please let me know if you have any difficulties >>> >>> Barry >>> >>> >>>> On Sep 6, 2018, at 4:48 PM, arkhazali at cc.iut.ac.ir wrote: >>>> >>>> Yes, I'd be grateful if MPI support could be added, please. >>>> >>>> Many thanks, >>>> Ali >>>> >>>> ------ Original message------ >>>> From: Smith, Barry F. >>>> Date: Fri, Sep 7, 2018 01:45 >>>> To: Ali Reza Khaz'ali; >>>> Cc: PETSc; >>>> Subject:Re: [petsc-users] Use block Jacobi preconditioner with SNES >>>> >>>> I have not yet added this support, do you need it now? >>>> >>>> Barry >>>> >>>> >>>>> On Sep 6, 2018, at 4:12 PM, Ali Reza Khaz'ali >>>> wrote: >>>>> I have to apologize for making this topic so lengthy. Is vpbjacobi supposed to work in parallel mode (with MPI)? >>>>> >>>>> >>>>> On 8/30/2018 1:32 AM, Smith, Barry F. wrote: >>>>>>> On Aug 29, 2018, at 3:48 PM, Ali Reza Khaz'ali >>>> wrote: >>>>>>>> Can you confirm if your code ran successfully with vpbjacobi and if the convergence history was very similar to that achieved using >>>>>>>> bjacobi ? >>>>>>> They have a very very very small difference (which is probably due to round-off errors), they generally behave the same. My code run successfully with both of them with the same convergence history. However, vpbjacobi seems a little faster. >>>>>> Great, this is exactly what to expect. I will put the branch into next for testing and then it will be in the master branch and in the next release. >>>>>> >>>>>> Barry >>>>>> >>>>>>> Many thanks, >>>>>>> Ali >>>>> -- >>>>> Ali Reza Khaz?ali >>>>> Assistant Professor of Petroleum Engineering, >>>>> Department of Chemical Engineering >>>>> Isfahan University of Technology >>>>> Isfahan, Iran >>>>> >> -- >> Ali Reza Khaz?ali >> Assistant Professor of Petroleum Engineering, >> Department of Chemical Engineering >> Isfahan University of Technology >> Isfahan, Iran >> -- Ali Reza Khaz?ali Assistant Professor of Petroleum Engineering, Department of Chemical Engineering Isfahan University of Technology Isfahan, Iran From griesser.jan at googlemail.com Mon Sep 10 08:56:26 2018 From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=) Date: Mon, 10 Sep 2018 15:56:26 +0200 Subject: [petsc-users] How to compile PETSC4py correctly with MPI ? Message-ID: Hey, i started again trying to solve an eigenvalue problem with petsc4py and slepc4py. I followed the instruction on the website and installed petsc and petsc4py via pip3. MPI4PY and the corresponding MPI version is already compiled on our cluster. Unfortunately when i start the following script by using: mpirun -n 2 python3 test3.py import time import numpy as np import sys, petsc4py petsc4py.init(sys.argv) from petsc4py import PETSc from mpi4py import MPI print(MPI.COMM_WORLD.Get_size()) # For print Print = PETSc.Sys.Print def construct_operator(m, n): """ Standard symmetric eigenproblem corresponding to the Laplacian operator in 2 dimensions. """ # Create matrix for 2D Laplacian operator A = PETSc.Mat().create() A.setSizes([m*n, m*n]) A.setFromOptions( ) A.setUp() # Fill matrix hx = 1.0/(m-1) # x grid spacing hy = 1.0/(n-1) # y grid spacing diagv = 2.0*hy/hx + 2.0*hx/hy offdx = -1.0*hy/hx offdy = -1.0*hx/hy Istart, Iend = A.getOwnershipRange() print("Istart: ", Istart) print("Iend: ", Iend) for I in range(Istart, Iend) : A[I,I] = diagv i = I//n # map row number to j = I - i*n # grid coordinates if i> 0 : J = I-n; A[I,J] = offdx if i< m-1: J = I+n; A[I,J] = offdx if j> 0 : J = I-1; A[I,J] = offdy if j< n-1: J = I+1; A[I,J] = offdy A.assemble() return A def main(): opts = PETSc.Options() N = opts.getInt('N', 96) m = opts.getInt('m', N) n = opts.getInt('n', m) Print("Symmetric Eigenproblem (sparse matrix), " "N=%d (%dx%d grid)" % (m*n, m, n)) A = construct_operator(m,n) Symmetric Eigenproblem (sparse matrix), N=9216 (96x96 grid) Symmetric Eigenproblem (sparse matrix), N=9216 (96x96 grid) 2 Istart: 0 Iend: 9216 2 Istart: 0 Iend: 9216 if __name__ == '__main__': main() it returns me the following output: instead of the one i assumed to get from a local execution of my file: 2 2 Symmetric Eigenproblem (sparse matrix), N=9216 (96x96 grid) Istart: 4608 Iend: 9216 Istart: 0 Iend: 4608 Therefore i assume somehow my MPI version is not linked correctly to my file and i checked with "ldd" my petsc file: linux-vdso.so.1 => (0x00007ffd8db6a000) liblapack.so.3 => /lib64/liblapack.so.3 (0x00007f1bd95e8000) libblas.so.3 => /lib64/libblas.so.3 (0x00007f1bd938e000) libm.so.6 => /lib64/libm.so.6 (0x00007f1bd908c000) libX11.so.6 => /lib64/libX11.so.6 (0x00007f1bd8d4e000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1bd8b31000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f1bd8829000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f1bd8625000) libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f1bd8302000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f1bd80ec000) libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f1bd7eb0000) libc.so.6 => /lib64/libc.so.6 (0x00007f1bd7aec000) /lib64/ld-linux-x86-64.so.2 (0x0000557325576000) libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f1bd78c4000) libXau.so.6 => /lib64/libXau.so.6 (0x00007f1bd76bf000) I assume that pip3 was for some reasons not able to link my mpi version to it. Can i force pip to link mpi or is there another mistake somewhere? -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Sep 10 09:12:03 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 10 Sep 2018 09:12:03 -0500 Subject: [petsc-users] How to compile PETSC4py correctly with MPI ? In-Reply-To: References: Message-ID: 1. you might want to locate and check configure.log for this build of PETSc to see exactly how its built [and which MPI got used] 2. And if its not built as you desired - use env variable PETSC_CONFIGURE_OPTIONS to specify additional necessary options to petsc configure - to built it as desired. 3. I'm not sure how pip chooses MPI used by MPI4PY - Lisandro might have to help with this. There is a non-python way to build this [which Lisandro doesn't recommend]: use PETSc configure to build mpi4py and petsc4py. i.e: ./configure --download-mpi4py --download-petsc4py [but then - you would install slepc/slepc4py manually after that] Satish On Mon, 10 Sep 2018, Jan Grie?er wrote: > Hey, > i started again trying to solve an eigenvalue problem with petsc4py and > slepc4py. I followed the instruction on the website and installed petsc and > petsc4py via pip3. MPI4PY and the corresponding MPI version is already > compiled on our cluster. > Unfortunately when i start the following script by using: > mpirun -n 2 python3 test3.py > > import time > import numpy as np > import sys, petsc4py > petsc4py.init(sys.argv) > from petsc4py import PETSc > > from mpi4py import MPI > print(MPI.COMM_WORLD.Get_size()) > > > # For print > Print = PETSc.Sys.Print > > def construct_operator(m, n): > """ > Standard symmetric eigenproblem corresponding to the > Laplacian operator in 2 dimensions. > """ > # Create matrix for 2D Laplacian operator > A = PETSc.Mat().create() > A.setSizes([m*n, m*n]) > A.setFromOptions( ) > A.setUp() > # Fill matrix > hx = 1.0/(m-1) # x grid spacing > hy = 1.0/(n-1) # y grid spacing > diagv = 2.0*hy/hx + 2.0*hx/hy > offdx = -1.0*hy/hx > offdy = -1.0*hx/hy > Istart, Iend = A.getOwnershipRange() > print("Istart: ", Istart) > print("Iend: ", Iend) > for I in range(Istart, Iend) : > A[I,I] = diagv > i = I//n # map row number to > j = I - i*n # grid coordinates > if i> 0 : J = I-n; A[I,J] = offdx > if i< m-1: J = I+n; A[I,J] = offdx > if j> 0 : J = I-1; A[I,J] = offdy > if j< n-1: J = I+1; A[I,J] = offdy > A.assemble() > return A > > def main(): > opts = PETSc.Options() > N = opts.getInt('N', 96) > m = opts.getInt('m', N) > n = opts.getInt('n', m) > Print("Symmetric Eigenproblem (sparse matrix), " > "N=%d (%dx%d grid)" % (m*n, m, n)) > A = construct_operator(m,n) > Symmetric Eigenproblem (sparse matrix), N=9216 (96x96 grid) > Symmetric Eigenproblem (sparse matrix), N=9216 (96x96 grid) > 2 > Istart: 0 > Iend: 9216 > 2 > Istart: 0 > Iend: 9216 > > if __name__ == '__main__': > main() > > it returns me the following output: > > instead of the one i assumed to get from a local execution of my file: > 2 > 2 > Symmetric Eigenproblem (sparse matrix), N=9216 (96x96 grid) > Istart: 4608 > Iend: 9216 > Istart: 0 > Iend: 4608 > > Therefore i assume somehow my MPI version is not linked correctly to my > file and i checked with "ldd" my petsc file: > linux-vdso.so.1 => (0x00007ffd8db6a000) > liblapack.so.3 => /lib64/liblapack.so.3 (0x00007f1bd95e8000) > libblas.so.3 => /lib64/libblas.so.3 (0x00007f1bd938e000) > libm.so.6 => /lib64/libm.so.6 (0x00007f1bd908c000) > libX11.so.6 => /lib64/libX11.so.6 (0x00007f1bd8d4e000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1bd8b31000) > libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f1bd8829000) > libdl.so.2 => /lib64/libdl.so.2 (0x00007f1bd8625000) > libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f1bd8302000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f1bd80ec000) > libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f1bd7eb0000) > libc.so.6 => /lib64/libc.so.6 (0x00007f1bd7aec000) > /lib64/ld-linux-x86-64.so.2 (0x0000557325576000) > libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f1bd78c4000) > libXau.so.6 => /lib64/libXau.so.6 (0x00007f1bd76bf000) > I assume that pip3 was for some reasons not able to link my mpi version to > it. Can i force pip to link mpi or is there another mistake somewhere? > From knepley at gmail.com Mon Sep 10 09:51:27 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 10 Sep 2018 10:51:27 -0400 Subject: [petsc-users] How to compile PETSC4py correctly with MPI ? In-Reply-To: References: Message-ID: On Mon, Sep 10, 2018 at 9:56 AM Jan Grie?er wrote: > Hey, > i started again trying to solve an eigenvalue problem with petsc4py and > slepc4py. I followed the instruction on the website and installed petsc and > petsc4py via pip3. MPI4PY and the corresponding MPI version is already > compiled on our cluster. > Unfortunately when i start the following script by using: > mpirun -n 2 python3 test3.py > Its likely that the 'mpirun' in your path does not match the MPI that MPI4PY was installed with. Matt > import time > import numpy as np > import sys, petsc4py > petsc4py.init(sys.argv) > from petsc4py import PETSc > > from mpi4py import MPI > print(MPI.COMM_WORLD.Get_size()) > > > # For print > Print = PETSc.Sys.Print > > def construct_operator(m, n): > """ > Standard symmetric eigenproblem corresponding to the > Laplacian operator in 2 dimensions. > """ > # Create matrix for 2D Laplacian operator > A = PETSc.Mat().create() > A.setSizes([m*n, m*n]) > A.setFromOptions( ) > A.setUp() > # Fill matrix > hx = 1.0/(m-1) # x grid spacing > hy = 1.0/(n-1) # y grid spacing > diagv = 2.0*hy/hx + 2.0*hx/hy > offdx = -1.0*hy/hx > offdy = -1.0*hx/hy > Istart, Iend = A.getOwnershipRange() > print("Istart: ", Istart) > print("Iend: ", Iend) > for I in range(Istart, Iend) : > A[I,I] = diagv > i = I//n # map row number to > j = I - i*n # grid coordinates > if i> 0 : J = I-n; A[I,J] = offdx > if i< m-1: J = I+n; A[I,J] = offdx > if j> 0 : J = I-1; A[I,J] = offdy > if j< n-1: J = I+1; A[I,J] = offdy > A.assemble() > return A > > def main(): > opts = PETSc.Options() > N = opts.getInt('N', 96) > m = opts.getInt('m', N) > n = opts.getInt('n', m) > Print("Symmetric Eigenproblem (sparse matrix), " > "N=%d (%dx%d grid)" % (m*n, m, n)) > A = construct_operator(m,n) > Symmetric Eigenproblem (sparse matrix), N=9216 (96x96 grid) > Symmetric Eigenproblem (sparse matrix), N=9216 (96x96 grid) > 2 > Istart: 0 > Iend: 9216 > 2 > Istart: 0 > Iend: 9216 > > if __name__ == '__main__': > main() > > it returns me the following output: > > instead of the one i assumed to get from a local execution of my file: > 2 > 2 > Symmetric Eigenproblem (sparse matrix), N=9216 (96x96 grid) > Istart: 4608 > Iend: 9216 > Istart: 0 > Iend: 4608 > > Therefore i assume somehow my MPI version is not linked correctly to my > file and i checked with "ldd" my petsc file: > linux-vdso.so.1 => (0x00007ffd8db6a000) > liblapack.so.3 => /lib64/liblapack.so.3 (0x00007f1bd95e8000) > libblas.so.3 => /lib64/libblas.so.3 (0x00007f1bd938e000) > libm.so.6 => /lib64/libm.so.6 (0x00007f1bd908c000) > libX11.so.6 => /lib64/libX11.so.6 (0x00007f1bd8d4e000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1bd8b31000) > libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f1bd8829000) > libdl.so.2 => /lib64/libdl.so.2 (0x00007f1bd8625000) > libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f1bd8302000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f1bd80ec000) > libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f1bd7eb0000) > libc.so.6 => /lib64/libc.so.6 (0x00007f1bd7aec000) > /lib64/ld-linux-x86-64.so.2 (0x0000557325576000) > libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f1bd78c4000) > libXau.so.6 => /lib64/libXau.so.6 (0x00007f1bd76bf000) > I assume that pip3 was for some reasons not able to link my mpi version to > it. Can i force pip to link mpi or is there another mistake somewhere? > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Mon Sep 10 11:10:21 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Mon, 10 Sep 2018 16:10:21 +0000 Subject: [petsc-users] DMForest In-Reply-To: References: , Message-ID: Matthew, It must be definitely because of my limited knowledge but, as an engineer, the documentation for DMForest is not clear to me. Does DMforest change the mesh of domain dynamically as the system changes? I am studying solidification problem by solving cahn-hillard equation. The solid phase starts from a point and it grows in the liquid phase by time. Does DMForest dynamically refine mesh at regions close to interface and coarse at domain far from the interface ( interface is moving)? Thanks, Sepideh ________________________________ From: Matthew Knepley Sent: Thursday, September 6, 2018 6:46:55 PM To: Sepideh Kavousi Cc: PETSc Subject: Re: [petsc-users] DMForest On Thu, Sep 6, 2018 at 6:04 PM Sepideh Kavousi > wrote: Hello, I have a Petsc code solving PDE using FD method, and I generated the structures grid with DMDA. I was planning to implement adaptive mesh refinement to the same structures mesh but the documentation on the DMForest is not easy to find. I checked the petsc/src/dm/impls/forest/examples/tutorials but I found one tutorial which did not have so much information. I also found src/ts/examples/tutorials/ex11.c but it was for DMPlex, and finite volume method. I wonder if there is another example for DMForest? TS ex11 works for Plex or Forest. Thanks, Matt Thanks, -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 10 11:30:46 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 10 Sep 2018 12:30:46 -0400 Subject: [petsc-users] DMForest In-Reply-To: References: Message-ID: On Mon, Sep 10, 2018 at 12:10 PM Sepideh Kavousi wrote: > Matthew, > > It must be definitely because of my limited knowledge but, as an engineer, > the documentation for DMForest is not clear to me. Does DMforest change the > mesh of domain dynamically as the system changes? > > > I am studying solidification problem by solving cahn-hillard equation. > The solid phase starts from a point and it grows in the liquid phase by > time. Does DMForest dynamically refine mesh at regions close to interface > and coarse at domain far from the interface ( interface is moving)? > DMForest is only able to do what you tell it to do. In the most basic usage, you can tell it which cells to refine and which to coarsen. We have a more sophisticated interface which takes an error approximator and some limits for coarsening and refining, but that is not fully release yet. You can see it working in TS ex11. The documentation for adaptivity is definitely missing. This is because we are still working out what the best way to do these things is. Thanks, Matt > Thanks, > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Thursday, September 6, 2018 6:46:55 PM > *To:* Sepideh Kavousi > *Cc:* PETSc > *Subject:* Re: [petsc-users] DMForest > > On Thu, Sep 6, 2018 at 6:04 PM Sepideh Kavousi wrote: > > Hello, > > I have a Petsc code solving PDE using FD method, and I generated the > structures grid with DMDA. I was planning to implement adaptive mesh > refinement to the same structures mesh but the documentation on the > DMForest is not easy to find. I checked the > petsc/src/dm/impls/forest/examples/tutorials but I found one tutorial > which did not have so much information. I also found > src/ts/examples/tutorials/ex11.c but it was for DMPlex, and finite volume > method. > > I wonder if there is another example for DMForest? > > > TS ex11 works for Plex or Forest. > > Thanks, > > Matt > > > Thanks, > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fabian.Jakub at physik.uni-muenchen.de Mon Sep 10 16:02:31 2018 From: Fabian.Jakub at physik.uni-muenchen.de (Fabian.Jakub) Date: Mon, 10 Sep 2018 23:02:31 +0200 Subject: [petsc-users] Checking for NULL Objects in Fortran Message-ID: <7b43037c-1109-6233-ea7f-4a98b4f3ced0@physik.uni-muenchen.de> Greetings my dear PETSc list. In my Fortran Code, I usually checked against PETSC_NULL_DM or PETSC_NULL_IS after calling methods such as DMPlexDistribute or DMGetStratumIS. Recently this fails me. I saw the DMPlexDistribute example in dm/label/examples/tutorials/ex1f90.F90 where the value of dm_distributed%v is compared against -1 Now, after the recent change of PETSC_NULL_DM (tDM(1) -> tDM(0) in 3d1df95b1) I am not 100% percent sure how to deal with it. Could you please clarify if testing against ``-1`` is the intended way to do it or may this change again soonish? Many Thanks, Fabian Here some code snippets to clarify what lead me to put up this question... This used to validate: call DMGetStratumIS(dm, 'DomainBoundary', SIDEFACE, bc_idx, ierr) if (bc_ids.eq.PETSC_NULL_IS) then ! dont have domain boundary points ... endif or call DMPlexDistribute(dm, i0, PETSC_NULL_SF, dmdist, ierr) if (dmdist.eq.PETSC_NULL_DM) then ! DM was not distributed endif Now I could do something like: if(bc_ids%v.eq.-1) of (dmdist%v.eq.-1) ... -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From jed at jedbrown.org Mon Sep 10 16:24:08 2018 From: jed at jedbrown.org (Jed Brown) Date: Mon, 10 Sep 2018 15:24:08 -0600 Subject: [petsc-users] Checking for NULL Objects in Fortran In-Reply-To: <7b43037c-1109-6233-ea7f-4a98b4f3ced0@physik.uni-muenchen.de> References: <7b43037c-1109-6233-ea7f-4a98b4f3ced0@physik.uni-muenchen.de> Message-ID: <87efe1f3lz.fsf@jedbrown.org> "Fabian.Jakub" writes: > Greetings my dear PETSc list. > > In my Fortran Code, I usually checked against PETSC_NULL_DM or > PETSC_NULL_IS after calling methods such as DMPlexDistribute or > DMGetStratumIS. > > Recently this fails me. > > I saw the DMPlexDistribute example in > dm/label/examples/tutorials/ex1f90.F90 > > where the value of dm_distributed%v is compared against -1 This code if (dmDist%v /= -1) then call DMDestroy(dm,ierr);CHKERRA(ierr) dm%v = dmDist%v end if should have used PETSC_NULL_DM instead of -1, but now it can just call DMDestroy(dm,ierr);CHKERRA(ierr) without needing to check. > Now, after the recent change of PETSC_NULL_DM (tDM(1) -> tDM(0) in > 3d1df95b1) I am not 100% percent sure how to deal with it. > > Could you please clarify if testing against ``-1`` is the intended way > to do it or may this change again soonish? > > Many Thanks, > > Fabian > > > Here some code snippets to clarify what lead me to put up this question... > > This used to validate: > > call DMGetStratumIS(dm, 'DomainBoundary', SIDEFACE, bc_idx, ierr) > if (bc_ids.eq.PETSC_NULL_IS) then ! dont have domain boundary points > ... > endif > > or > > call DMPlexDistribute(dm, i0, PETSC_NULL_SF, dmdist, ierr) > if (dmdist.eq.PETSC_NULL_DM) then ! DM was not distributed > endif > > > Now I could do something like: > > if(bc_ids%v.eq.-1) > > of (dmdist%v.eq.-1) > > ... From bsmith at mcs.anl.gov Mon Sep 10 16:30:31 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 10 Sep 2018 21:30:31 +0000 Subject: [petsc-users] Checking for NULL Objects in Fortran In-Reply-To: <7b43037c-1109-6233-ea7f-4a98b4f3ced0@physik.uni-muenchen.de> References: <7b43037c-1109-6233-ea7f-4a98b4f3ced0@physik.uni-muenchen.de> Message-ID: Sorry for your difficulties The recent branch barry/change-fortran-null was intended to simplify the handling of NULL slightly in Fortran and make it more consistent with the handling in C. It essentially changed %v to be equal to 0 for NULL objects rather than -1. This allows easier passing objects directly to C code also. We did not intend to change the user API; we intended that comparisons with PETSC_NULL_DM would behave exactly as before. Unfortunately it looks like some places in the code had -1 hardwired, for example, /* Definitions of Fortran Wrapper routines */ PETSC_EXTERN void PETSC_STDCALL dmplexdistribute_(DM *dm, PetscInt *overlap, PetscSF *sf, DM *dmParallel, int *ierr) { CHKFORTRANNULLOBJECT(sf); CHKFORTRANNULLOBJECT(dmParallel); *ierr = DMPlexDistribute(*dm, *overlap, sf, dmParallel); if (dmParallel && !*dmParallel) *dmParallel = (DM)-1; } I did not know about this hardwiring and thus did not update them while making the new branch. The branch barry/change-fortran-null-fixup fixes the two bad uses of -1 that you pointed out. Please try it and let us know other problems that come up since we are planning a release very shortly. > On Sep 10, 2018, at 4:02 PM, Fabian.Jakub wrote: > > Greetings my dear PETSc list. > > In my Fortran Code, I usually checked against PETSC_NULL_DM or > PETSC_NULL_IS after calling methods such as DMPlexDistribute or > DMGetStratumIS. > > Recently this fails me. > > I saw the DMPlexDistribute example in > dm/label/examples/tutorials/ex1f90.F90 > > where the value of dm_distributed%v is compared against -1 We do not recommend the usage of dm_distributed%v /= 0 and instead recommend comparing with the appropriate PETSC_NULL_XXX as you have done. (the %v business will work, we just don't recommend it). > > Now, after the recent change of PETSC_NULL_DM (tDM(1) -> tDM(0) in > 3d1df95b1) I am not 100% percent sure how to deal with it. > > Could you please clarify if testing against ``-1`` is the intended way > to do it or may this change again soonish? Definitely not. Your code should remain the same and we simply need to fix the stray -1 to change them to 0. Barry > > Many Thanks, > > Fabian > > > Here some code snippets to clarify what lead me to put up this question... > > This used to validate: > > call DMGetStratumIS(dm, 'DomainBoundary', SIDEFACE, bc_idx, ierr) > if (bc_ids.eq.PETSC_NULL_IS) then ! dont have domain boundary points > ... > endif > > or > > call DMPlexDistribute(dm, i0, PETSC_NULL_SF, dmdist, ierr) > if (dmdist.eq.PETSC_NULL_DM) then ! DM was not distributed > endif > > > Now I could do something like: > > if(bc_ids%v.eq.-1) > > of (dmdist%v.eq.-1) > > ... > From Fabian.Jakub at physik.uni-muenchen.de Mon Sep 10 16:40:44 2018 From: Fabian.Jakub at physik.uni-muenchen.de (Fabian Jakub) Date: Mon, 10 Sep 2018 23:40:44 +0200 Subject: [petsc-users] Checking for NULL Objects in Fortran In-Reply-To: References: <7b43037c-1109-6233-ea7f-4a98b4f3ced0@physik.uni-muenchen.de> Message-ID: <53a25d0d-edec-6797-66de-40cb95474c17@physik.uni-muenchen.de> Allright, Many thanks for the super prompt response! On 10.09.2018 23:30, Smith, Barry F. wrote: > Sorry for your difficulties > > The recent branch barry/change-fortran-null was intended to simplify the handling of NULL slightly in Fortran and make it more consistent with the handling in C. It essentially changed %v to be equal to 0 for NULL objects rather than -1. This allows easier passing objects directly to C code also. > > We did not intend to change the user API; we intended that comparisons with PETSC_NULL_DM would behave exactly as before. > > Unfortunately it looks like some places in the code had -1 hardwired, for example, > > /* Definitions of Fortran Wrapper routines */ > PETSC_EXTERN void PETSC_STDCALL dmplexdistribute_(DM *dm, PetscInt *overlap, PetscSF *sf, DM *dmParallel, int *ierr) > { > CHKFORTRANNULLOBJECT(sf); > CHKFORTRANNULLOBJECT(dmParallel); > *ierr = DMPlexDistribute(*dm, *overlap, sf, dmParallel); > if (dmParallel && !*dmParallel) *dmParallel = (DM)-1; > } > > I did not know about this hardwiring and thus did not update them while making the new branch. > > The branch barry/change-fortran-null-fixup fixes the two bad uses of -1 that you pointed out. Please try it and let us know other problems that come up since we are planning a release very shortly. > > > > >> On Sep 10, 2018, at 4:02 PM, Fabian.Jakub wrote: >> >> Greetings my dear PETSc list. >> >> In my Fortran Code, I usually checked against PETSC_NULL_DM or >> PETSC_NULL_IS after calling methods such as DMPlexDistribute or >> DMGetStratumIS. >> >> Recently this fails me. >> >> I saw the DMPlexDistribute example in >> dm/label/examples/tutorials/ex1f90.F90 >> >> where the value of dm_distributed%v is compared against -1 > We do not recommend the usage of dm_distributed%v /= 0 and instead recommend comparing with the appropriate PETSC_NULL_XXX as you have done. (the %v business will work, we just don't recommend it). > >> Now, after the recent change of PETSC_NULL_DM (tDM(1) -> tDM(0) in >> 3d1df95b1) I am not 100% percent sure how to deal with it. >> >> Could you please clarify if testing against ``-1`` is the intended way >> to do it or may this change again soonish? > Definitely not. Your code should remain the same and we simply need to fix the stray -1 to change them to 0. > > Barry > >> Many Thanks, >> >> Fabian >> >> >> Here some code snippets to clarify what lead me to put up this question... >> >> This used to validate: >> >> call DMGetStratumIS(dm, 'DomainBoundary', SIDEFACE, bc_idx, ierr) >> if (bc_ids.eq.PETSC_NULL_IS) then ! dont have domain boundary points >> ... >> endif >> >> or >> >> call DMPlexDistribute(dm, i0, PETSC_NULL_SF, dmdist, ierr) >> if (dmdist.eq.PETSC_NULL_DM) then ! DM was not distributed >> endif >> >> >> Now I could do something like: >> >> if(bc_ids%v.eq.-1) >> >> of (dmdist%v.eq.-1) >> >> ... >> From ysjosh.lo at gmail.com Tue Sep 11 15:51:46 2018 From: ysjosh.lo at gmail.com (Josh L) Date: Tue, 11 Sep 2018 15:51:46 -0500 Subject: [petsc-users] TAO with time dependent problem Message-ID: Hi, I am using PETSc TAO to solve the following equation obtained from minimizing certain energy and constitutive model: u_xx - u - u_t =0 (1) For simplicity, the coefficients are neglected. In the routine to form function value and gradient, I use (1) to form my gradient vector, and my energy definition to calculate function value.Is it correct? I only found one tao example that solves time dependent problem, but it is using TS which I am not using. Thanks, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysjosh.lo at gmail.com Tue Sep 11 23:33:45 2018 From: ysjosh.lo at gmail.com (Josh L) Date: Tue, 11 Sep 2018 23:33:45 -0500 Subject: [petsc-users] SNES with constraint diverges Message-ID: Hi, I am using SNES to solve an nonlinear equation f(u), and I know all the u_i should be 0 and 1. First, I use SNES without constraint, and it converges. But, If I set SNESSetType(snes,SNESVINEWTONRSLS,ierr) SNESVISetVariableBounds(snes,xl,xu,ierr) where xl and xu is vector, and xl_i=0 and xu_i=1 then SNES fails to converge, because linesearch fails(snes reason = -6), and the norm of residual is not reducing(the norm of incremental solution is reducing) The reason to add constraint is that I want to implement some irreversibility. Thanks, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Sep 11 23:37:57 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 12 Sep 2018 04:37:57 +0000 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: References: Message-ID: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> Do you start with initial conditions of 0 <= u_i <= 1 ? Run with -snes_monitor -snes_converged_reason -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the output Barry > On Sep 11, 2018, at 11:33 PM, Josh L wrote: > > Hi, > > I am using SNES to solve an nonlinear equation f(u), and I know all the u_i should be 0 and 1. > > First, I use SNES without constraint, and it converges. > > But, If I set > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > SNESVISetVariableBounds(snes,xl,xu,ierr) > > where xl and xu is vector, and xl_i=0 and xu_i=1 > > then SNES fails to converge, because linesearch fails(snes reason = -6), and the norm of residual is not reducing(the norm of incremental solution is reducing) > > The reason to add constraint is that I want to implement some irreversibility. > > > Thanks, > Josh > From ysjosh.lo at gmail.com Tue Sep 11 23:53:14 2018 From: ysjosh.lo at gmail.com (Josh L) Date: Tue, 11 Sep 2018 23:53:14 -0500 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> Message-ID: Yes, I initialize all u_i to 1.0 2018-09-11 23:37 GMT-05:00 Smith, Barry F. : > > Do you start with initial conditions of 0 <= u_i <= 1 ? > > Run with -snes_monitor -snes_converged_reason > -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the > output > > Barry > > > > On Sep 11, 2018, at 11:33 PM, Josh L wrote: > > > > Hi, > > > > I am using SNES to solve an nonlinear equation f(u), and I know all the > u_i should be 0 and 1. > > > > First, I use SNES without constraint, and it converges. > > > > But, If I set > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > > SNESVISetVariableBounds(snes,xl,xu,ierr) > > > > where xl and xu is vector, and xl_i=0 and xu_i=1 > > > > then SNES fails to converge, because linesearch fails(snes reason = -6), > and the norm of residual is not reducing(the norm of incremental solution > is reducing) > > > > The reason to add constraint is that I want to implement some > irreversibility. > > > > > > Thanks, > > Josh > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysjosh.lo at gmail.com Wed Sep 12 00:25:42 2018 From: ysjosh.lo at gmail.com (Josh L) Date: Wed, 12 Sep 2018 00:25:42 -0500 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> Message-ID: The following is the output. Thanks. 0 SNES Function norm 3.775139483936e-07 Line search: Using full step: fnorm 3.775139483936e-07 gnorm 1.043499007769e-11 1 SNES Function norm 1.043499007769e-11 Line search: Using full step: fnorm 1.043499007769e-11 gnorm 5.302082026305e-12 2 SNES Function norm 5.302082026305e-12 Line search: Using full step: fnorm 5.302082026305e-12 gnorm 3.136358665117e-12 3 SNES Function norm 3.136358665117e-12 Line search: Using full step: fnorm 3.136358665117e-12 gnorm 2.170819065257e-12 4 SNES Function norm 2.170819065257e-12 Line search: Using full step: fnorm 2.170819065257e-12 gnorm 1.475760790713e-12 5 SNES Function norm 1.475760790713e-12 Line search: Using full step: fnorm 1.475760790713e-12 gnorm 1.130642522675e-12 6 SNES Function norm 1.130642522675e-12 Line search: Using full step: fnorm 1.130642522675e-12 gnorm 8.647827205825e-13 7 SNES Function norm 8.647827205825e-13 Line search: Using full step: fnorm 8.647827205825e-13 gnorm 7.403983354386e-13 8 SNES Function norm 7.403983354386e-13 Line search: Using full step: fnorm 7.403983354386e-13 gnorm 6.291615062788e-13 9 SNES Function norm 6.291615062788e-13 Line search: Using full step: fnorm 6.291615062788e-13 gnorm 5.682820624094e-13 10 SNES Function norm 5.682820624094e-13 Line search: Using full step: fnorm 5.682820624094e-13 gnorm 5.300267435005e-13 11 SNES Function norm 5.300267435005e-13 Line search: Using full step: fnorm 5.300267435005e-13 gnorm 5.096339188439e-13 12 SNES Function norm 5.096339188439e-13 Line search: Using full step: fnorm 5.096339188439e-13 gnorm 4.970340343183e-13 13 SNES Function norm 4.970340343183e-13 Line search: Using full step: fnorm 4.970340343183e-13 gnorm 4.916509482510e-13 14 SNES Function norm 4.916509482510e-13 Line search: Using full step: fnorm 4.916509482510e-13 gnorm 4.875840608086e-13 15 SNES Function norm 4.875840608086e-13 Line search: Using full step: fnorm 4.875840608086e-13 gnorm 4.859609948396e-13 16 SNES Function norm 4.859609948396e-13 Line search: Using full step: fnorm 4.859609948396e-13 gnorm 4.846457390135e-13 17 SNES Function norm 4.846457390135e-13 Line search: Using full step: fnorm 4.846457390135e-13 gnorm 4.840667643605e-13 18 SNES Function norm 4.840667643605e-13 Line search: Using full step: fnorm 4.840667643605e-13 gnorm 4.837878467904e-13 19 SNES Function norm 4.837878467904e-13 Line search: Using full step: fnorm 4.837878467904e-13 gnorm 4.835385298533e-13 20 SNES Function norm 4.835385298533e-13 Line search: Using full step: fnorm 4.835385298533e-13 gnorm 4.833767981145e-13 21 SNES Function norm 4.833767981145e-13 Line search: Aborted due to ynorm < stol*xnorm (6.489173391862e-15 < 2.387840730261e-06) and inadequate full step. Nonlinear smu_ solve did not converge due to DIVERGED_LINE_SEARCH iterations 21 2018-09-11 23:37 GMT-05:00 Smith, Barry F. : > > Do you start with initial conditions of 0 <= u_i <= 1 ? > > Run with -snes_monitor -snes_converged_reason > -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the > output > > Barry > > > > On Sep 11, 2018, at 11:33 PM, Josh L wrote: > > > > Hi, > > > > I am using SNES to solve an nonlinear equation f(u), and I know all the > u_i should be 0 and 1. > > > > First, I use SNES without constraint, and it converges. > > > > But, If I set > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > > SNESVISetVariableBounds(snes,xl,xu,ierr) > > > > where xl and xu is vector, and xl_i=0 and xu_i=1 > > > > then SNES fails to converge, because linesearch fails(snes reason = -6), > and the norm of residual is not reducing(the norm of incremental solution > is reducing) > > > > The reason to add constraint is that I want to implement some > irreversibility. > > > > > > Thanks, > > Josh > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysjosh.lo at gmail.com Wed Sep 12 00:31:15 2018 From: ysjosh.lo at gmail.com (Josh L) Date: Wed, 12 Sep 2018 00:31:15 -0500 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> Message-ID: The following is the converging one(without setting SNESVI and variable bounds) 0 SNES Function norm 3.775139279770e-07 Line search: Using full step: fnorm 3.775139279770e-07 gnorm 1.911103519935e-12 1 SNES Function norm 1.911103519935e-12 Nonlinear smu_ solve converged due to CONVERGED_SNORM_RELATIVE iterations 1 2018-09-12 0:25 GMT-05:00 Josh L : > > The following is the output. Thanks. > > 0 SNES Function norm 3.775139483936e-07 > Line search: Using full step: fnorm 3.775139483936e-07 gnorm > 1.043499007769e-11 > 1 SNES Function norm 1.043499007769e-11 > Line search: Using full step: fnorm 1.043499007769e-11 gnorm > 5.302082026305e-12 > 2 SNES Function norm 5.302082026305e-12 > Line search: Using full step: fnorm 5.302082026305e-12 gnorm > 3.136358665117e-12 > 3 SNES Function norm 3.136358665117e-12 > Line search: Using full step: fnorm 3.136358665117e-12 gnorm > 2.170819065257e-12 > 4 SNES Function norm 2.170819065257e-12 > Line search: Using full step: fnorm 2.170819065257e-12 gnorm > 1.475760790713e-12 > 5 SNES Function norm 1.475760790713e-12 > Line search: Using full step: fnorm 1.475760790713e-12 gnorm > 1.130642522675e-12 > 6 SNES Function norm 1.130642522675e-12 > Line search: Using full step: fnorm 1.130642522675e-12 gnorm > 8.647827205825e-13 > 7 SNES Function norm 8.647827205825e-13 > Line search: Using full step: fnorm 8.647827205825e-13 gnorm > 7.403983354386e-13 > 8 SNES Function norm 7.403983354386e-13 > Line search: Using full step: fnorm 7.403983354386e-13 gnorm > 6.291615062788e-13 > 9 SNES Function norm 6.291615062788e-13 > Line search: Using full step: fnorm 6.291615062788e-13 gnorm > 5.682820624094e-13 > 10 SNES Function norm 5.682820624094e-13 > Line search: Using full step: fnorm 5.682820624094e-13 gnorm > 5.300267435005e-13 > 11 SNES Function norm 5.300267435005e-13 > Line search: Using full step: fnorm 5.300267435005e-13 gnorm > 5.096339188439e-13 > 12 SNES Function norm 5.096339188439e-13 > Line search: Using full step: fnorm 5.096339188439e-13 gnorm > 4.970340343183e-13 > 13 SNES Function norm 4.970340343183e-13 > Line search: Using full step: fnorm 4.970340343183e-13 gnorm > 4.916509482510e-13 > 14 SNES Function norm 4.916509482510e-13 > Line search: Using full step: fnorm 4.916509482510e-13 gnorm > 4.875840608086e-13 > 15 SNES Function norm 4.875840608086e-13 > Line search: Using full step: fnorm 4.875840608086e-13 gnorm > 4.859609948396e-13 > 16 SNES Function norm 4.859609948396e-13 > Line search: Using full step: fnorm 4.859609948396e-13 gnorm > 4.846457390135e-13 > 17 SNES Function norm 4.846457390135e-13 > Line search: Using full step: fnorm 4.846457390135e-13 gnorm > 4.840667643605e-13 > 18 SNES Function norm 4.840667643605e-13 > Line search: Using full step: fnorm 4.840667643605e-13 gnorm > 4.837878467904e-13 > 19 SNES Function norm 4.837878467904e-13 > Line search: Using full step: fnorm 4.837878467904e-13 gnorm > 4.835385298533e-13 > 20 SNES Function norm 4.835385298533e-13 > Line search: Using full step: fnorm 4.835385298533e-13 gnorm > 4.833767981145e-13 > 21 SNES Function norm 4.833767981145e-13 > Line search: Aborted due to ynorm < stol*xnorm (6.489173391862e-15 < > 2.387840730261e-06) and inadequate full step. > Nonlinear smu_ solve did not converge due to DIVERGED_LINE_SEARCH > iterations 21 > > > > > 2018-09-11 23:37 GMT-05:00 Smith, Barry F. : > >> >> Do you start with initial conditions of 0 <= u_i <= 1 ? >> >> Run with -snes_monitor -snes_converged_reason >> -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the >> output >> >> Barry >> >> >> > On Sep 11, 2018, at 11:33 PM, Josh L wrote: >> > >> > Hi, >> > >> > I am using SNES to solve an nonlinear equation f(u), and I know all the >> u_i should be 0 and 1. >> > >> > First, I use SNES without constraint, and it converges. >> > >> > But, If I set >> > SNESSetType(snes,SNESVINEWTONRSLS,ierr) >> > SNESVISetVariableBounds(snes,xl,xu,ierr) >> > >> > where xl and xu is vector, and xl_i=0 and xu_i=1 >> > >> > then SNES fails to converge, because linesearch fails(snes reason = >> -6), and the norm of residual is not reducing(the norm of incremental >> solution is reducing) >> > >> > The reason to add constraint is that I want to implement some >> irreversibility. >> > >> > >> > Thanks, >> > Josh >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 12 04:53:46 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Sep 2018 05:53:46 -0400 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> Message-ID: On Wed, Sep 12, 2018 at 1:25 AM Josh L wrote: > > The following is the output. Thanks. > I do not know what you residual looks like, but it entirely possible that your Jacobian is not accurate enough to drive the residual below 5e-13. You could just try -snes_atol 1e-12 unless you really think you need to be more accurate. Also have you non-dimensionalized your problem? Thanks, Matt > 0 SNES Function norm 3.775139483936e-07 > Line search: Using full step: fnorm 3.775139483936e-07 gnorm > 1.043499007769e-11 > 1 SNES Function norm 1.043499007769e-11 > Line search: Using full step: fnorm 1.043499007769e-11 gnorm > 5.302082026305e-12 > 2 SNES Function norm 5.302082026305e-12 > Line search: Using full step: fnorm 5.302082026305e-12 gnorm > 3.136358665117e-12 > 3 SNES Function norm 3.136358665117e-12 > Line search: Using full step: fnorm 3.136358665117e-12 gnorm > 2.170819065257e-12 > 4 SNES Function norm 2.170819065257e-12 > Line search: Using full step: fnorm 2.170819065257e-12 gnorm > 1.475760790713e-12 > 5 SNES Function norm 1.475760790713e-12 > Line search: Using full step: fnorm 1.475760790713e-12 gnorm > 1.130642522675e-12 > 6 SNES Function norm 1.130642522675e-12 > Line search: Using full step: fnorm 1.130642522675e-12 gnorm > 8.647827205825e-13 > 7 SNES Function norm 8.647827205825e-13 > Line search: Using full step: fnorm 8.647827205825e-13 gnorm > 7.403983354386e-13 > 8 SNES Function norm 7.403983354386e-13 > Line search: Using full step: fnorm 7.403983354386e-13 gnorm > 6.291615062788e-13 > 9 SNES Function norm 6.291615062788e-13 > Line search: Using full step: fnorm 6.291615062788e-13 gnorm > 5.682820624094e-13 > 10 SNES Function norm 5.682820624094e-13 > Line search: Using full step: fnorm 5.682820624094e-13 gnorm > 5.300267435005e-13 > 11 SNES Function norm 5.300267435005e-13 > Line search: Using full step: fnorm 5.300267435005e-13 gnorm > 5.096339188439e-13 > 12 SNES Function norm 5.096339188439e-13 > Line search: Using full step: fnorm 5.096339188439e-13 gnorm > 4.970340343183e-13 > 13 SNES Function norm 4.970340343183e-13 > Line search: Using full step: fnorm 4.970340343183e-13 gnorm > 4.916509482510e-13 > 14 SNES Function norm 4.916509482510e-13 > Line search: Using full step: fnorm 4.916509482510e-13 gnorm > 4.875840608086e-13 > 15 SNES Function norm 4.875840608086e-13 > Line search: Using full step: fnorm 4.875840608086e-13 gnorm > 4.859609948396e-13 > 16 SNES Function norm 4.859609948396e-13 > Line search: Using full step: fnorm 4.859609948396e-13 gnorm > 4.846457390135e-13 > 17 SNES Function norm 4.846457390135e-13 > Line search: Using full step: fnorm 4.846457390135e-13 gnorm > 4.840667643605e-13 > 18 SNES Function norm 4.840667643605e-13 > Line search: Using full step: fnorm 4.840667643605e-13 gnorm > 4.837878467904e-13 > 19 SNES Function norm 4.837878467904e-13 > Line search: Using full step: fnorm 4.837878467904e-13 gnorm > 4.835385298533e-13 > 20 SNES Function norm 4.835385298533e-13 > Line search: Using full step: fnorm 4.835385298533e-13 gnorm > 4.833767981145e-13 > 21 SNES Function norm 4.833767981145e-13 > Line search: Aborted due to ynorm < stol*xnorm (6.489173391862e-15 < > 2.387840730261e-06) and inadequate full step. > Nonlinear smu_ solve did not converge due to DIVERGED_LINE_SEARCH > iterations 21 > > > > > 2018-09-11 23:37 GMT-05:00 Smith, Barry F. : > >> >> Do you start with initial conditions of 0 <= u_i <= 1 ? >> >> Run with -snes_monitor -snes_converged_reason >> -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the >> output >> >> Barry >> >> >> > On Sep 11, 2018, at 11:33 PM, Josh L wrote: >> > >> > Hi, >> > >> > I am using SNES to solve an nonlinear equation f(u), and I know all the >> u_i should be 0 and 1. >> > >> > First, I use SNES without constraint, and it converges. >> > >> > But, If I set >> > SNESSetType(snes,SNESVINEWTONRSLS,ierr) >> > SNESVISetVariableBounds(snes,xl,xu,ierr) >> > >> > where xl and xu is vector, and xl_i=0 and xu_i=1 >> > >> > then SNES fails to converge, because linesearch fails(snes reason = >> -6), and the norm of residual is not reducing(the norm of incremental >> solution is reducing) >> > >> > The reason to add constraint is that I want to implement some >> irreversibility. >> > >> > >> > Thanks, >> > Josh >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From phil.tooley at sheffield.ac.uk Wed Sep 12 08:27:25 2018 From: phil.tooley at sheffield.ac.uk (Phil Tooley) Date: Wed, 12 Sep 2018 14:27:25 +0100 Subject: [petsc-users] Indexing/using a 3D DMDA as a 1D vector Message-ID: <5866baee-9b50-089d-f19c-60035e23dea7@sheffield.ac.uk> I will preface this by saying I am new to PETSc and am still trying to get my head around all of the layout mapping that is done.? That means I may well have fundamentally misunderstood something, but hopefully someone will be able to to put me right. In my application I have some 3D pixel data which I want to manipulate using finite difference methods and then transform by viewing as a 1-D vector and multiplying by a large sparse matrix operator. I would assume that the correct way to do this is by creating a DMDA to hold the image data and ghosting appropriately to apply my finite difference operations.? Then I had hoped that I could use some form of application ordering to allow viewing the data as a vector that can be multiplied with my operator matrix.? This is where I have come unstuck, I may just be missing something obivous but I can't figure out how to do this.? Can anyone point me in the correct direction please? Many Thanks -- Phil Tooley Research Software Engineering University of Sheffield From adener at anl.gov Wed Sep 12 09:26:52 2018 From: adener at anl.gov (Dener, Alp) Date: Wed, 12 Sep 2018 14:26:52 +0000 Subject: [petsc-users] TAO with time dependent problem In-Reply-To: References: Message-ID: <5F7B684F-528C-466F-9504-62FCD4768EE0@anl.gov> Hi Josh, In the FormFunctionAndGradient() function, the gradient you compute and provide TAO must be the gradient of the objective function with respect to the optimization variables. If your optimization variables are u, and your objective function is J(u), then FormFunctionGradient would need to compute J(u) and dJ/du. I?m not entirely sure if I understand your problem correctly, but it seems that your minimal energy problem reduces to the solution of equation (1). Is this correct? TAO is not the right package to solve time-dependent PDEs like (1) directly. You would want to use TS for that. However, TAO can be used to solve the original energy minimization problem. A good example of this is the canonical obstacle problem that aims to minimize the Dirichlet energy function subject to a constraint that represents the obstacle. That problem reduces to the solution of the Laplace equation with the appropriate boundary conditions matching the obstacle, which would be discretized and solved as a stand-alone PDE. However, the same problem can also be solved as an optimization problem where the objective function is the Dirichlet energy function, and the associated gradient is related to the stiffness matrix associated with the Laplace equation. The two formulations are mathematically equivalent. If I?m interpreting your problem correctly, you should be able to take a similar approach, but you would need to derive the gradient of your energy function and see how you can construct it in relation to equation (1). ?? Alp Dener Argonne National Laboratory http://www.mcs.anl.gov/person/alp-dener On Sep 11, 2018, at 3:51 PM, Josh L > wrote: Hi, I am using PETSc TAO to solve the following equation obtained from minimizing certain energy and constitutive model: u_xx - u - u_t =0 (1) For simplicity, the coefficients are neglected. In the routine to form function value and gradient, I use (1) to form my gradient vector, and my energy definition to calculate function value.Is it correct? I only found one tao example that solves time dependent problem, but it is using TS which I am not using. Thanks, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From phil.tooley at sheffield.ac.uk Wed Sep 12 09:43:41 2018 From: phil.tooley at sheffield.ac.uk (Phil Tooley) Date: Wed, 12 Sep 2018 15:43:41 +0100 Subject: [petsc-users] Indexing/using a 3D DMDA as a 1D vector In-Reply-To: <5866baee-9b50-089d-f19c-60035e23dea7@sheffield.ac.uk> References: <5866baee-9b50-089d-f19c-60035e23dea7@sheffield.ac.uk> Message-ID: <86046374-0c06-7e78-18b6-0cee8989a525@sheffield.ac.uk> Ok, So I have figured this out a bit better now, facepalm moment as I finally realised that this is exactly what the DMDA exists to do...? I need to create a vector using the DMCreate*Vector functions and can then insert values using the assembly functions.? Then, if I understand correctly, in order to have the state I need to multiply with my operator I should make use of the DMDAGlobalToNaturalBegin() and End() functions to ensure the correct ordering.? Do I then need to ensure I have reverted it to global ordering before I try to apply finite difference methods again? Thanks P On 12/09/18 14:27, Phil Tooley wrote: > I will preface this by saying I am new to PETSc and am still trying to > get my head around all of the layout mapping that is done.? That means I > may well have fundamentally misunderstood something, but hopefully > someone will be able to to put me right. > > In my application I have some 3D pixel data which I want to manipulate > using finite difference methods and then transform by viewing as a 1-D > vector and multiplying by a large sparse matrix operator. > > I would assume that the correct way to do this is by creating a DMDA to > hold the image data and ghosting appropriately to apply my finite > difference operations.? Then I had hoped that I could use some form of > application ordering to allow viewing the data as a vector that can be > multiplied with my operator matrix.? This is where I have come unstuck, > I may just be missing something obivous but I can't figure out how to do > this.? Can anyone point me in the correct direction please? > > Many Thanks > -- Phil Tooley Research Software Engineering University of Sheffield From knepley at gmail.com Wed Sep 12 09:56:29 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Sep 2018 10:56:29 -0400 Subject: [petsc-users] Indexing/using a 3D DMDA as a 1D vector In-Reply-To: <5866baee-9b50-089d-f19c-60035e23dea7@sheffield.ac.uk> References: <5866baee-9b50-089d-f19c-60035e23dea7@sheffield.ac.uk> Message-ID: On Wed, Sep 12, 2018 at 9:31 AM Phil Tooley wrote: > I will preface this by saying I am new to PETSc and am still trying to > get my head around all of the layout mapping that is done. That means I > may well have fundamentally misunderstood something, but hopefully > someone will be able to to put me right. > > In my application I have some 3D pixel data which I want to manipulate > using finite difference methods and then transform by viewing as a 1-D > vector and multiplying by a large sparse matrix operator. > > I would assume that the correct way to do this is by creating a DMDA to > hold the image data and ghosting appropriately to apply my finite > difference operations. Then I had hoped that I could use some form of > application ordering to allow viewing the data as a vector that can be > multiplied with my operator matrix. This is where I have come unstuck, > I may just be missing something obivous but I can't figure out how to do > this. Can anyone point me in the correct direction please? > Okay, the initial intuition is correct. You can use a DMDA to organize your pixel data. Note that DMDA only holds metadata about the Cartesian mesh. It never holds data. All data is held in Vec objects. So when you load data, it is loaded into a Vec. Thus you automatically have a 1D representation. It might not be the 1D representation you are looking for. In a running program, we organize the data so it is lexicographic _per subdomain_. However, lots of users want it organized lexicographically for the entire domain. Thus we have a convenience feature, that when you output or input a vector, we convert to the "natural" ordering. You can also get the natural ordering yourself using a GlobalToNatural scatter. Feel free to mail with more questions. Thanks, Matt > Many Thanks > > -- > Phil Tooley > Research Software Engineering > University of Sheffield > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 12 09:57:28 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Sep 2018 10:57:28 -0400 Subject: [petsc-users] Indexing/using a 3D DMDA as a 1D vector In-Reply-To: <86046374-0c06-7e78-18b6-0cee8989a525@sheffield.ac.uk> References: <5866baee-9b50-089d-f19c-60035e23dea7@sheffield.ac.uk> <86046374-0c06-7e78-18b6-0cee8989a525@sheffield.ac.uk> Message-ID: On Wed, Sep 12, 2018 at 10:43 AM Phil Tooley wrote: > Ok, So I have figured this out a bit better now, facepalm moment as I > finally realised that this is exactly what the DMDA exists to do... I > need to create a vector using the DMCreate*Vector functions and can then > insert values using the assembly functions. > > Then, if I understand correctly, in order to have the state I need to > multiply with my operator I should make use of the > DMDAGlobalToNaturalBegin() and End() functions to ensure the correct > ordering. Do I then need to ensure I have reverted it to global > ordering before I try to apply finite difference methods again? > Yep. Depending on what your operator is, you could also try to implement it in the FD way as well. Thanks, Matt > Thanks > > P > > > On 12/09/18 14:27, Phil Tooley wrote: > > I will preface this by saying I am new to PETSc and am still trying to > > get my head around all of the layout mapping that is done. That means I > > may well have fundamentally misunderstood something, but hopefully > > someone will be able to to put me right. > > > > In my application I have some 3D pixel data which I want to manipulate > > using finite difference methods and then transform by viewing as a 1-D > > vector and multiplying by a large sparse matrix operator. > > > > I would assume that the correct way to do this is by creating a DMDA to > > hold the image data and ghosting appropriately to apply my finite > > difference operations. Then I had hoped that I could use some form of > > application ordering to allow viewing the data as a vector that can be > > multiplied with my operator matrix. This is where I have come unstuck, > > I may just be missing something obivous but I can't figure out how to do > > this. Can anyone point me in the correct direction please? > > > > Many Thanks > > > > -- > Phil Tooley > Research Software Engineering > University of Sheffield > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysjosh.lo at gmail.com Wed Sep 12 10:36:25 2018 From: ysjosh.lo at gmail.com (Josh L) Date: Wed, 12 Sep 2018 10:36:25 -0500 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> Message-ID: Yes. The problem is normalized. Thanks, Josh 2018-09-12 4:53 GMT-05:00 Matthew Knepley : > On Wed, Sep 12, 2018 at 1:25 AM Josh L wrote: > >> >> The following is the output. Thanks. >> > > I do not know what you residual looks like, but it entirely possible that > your Jacobian is not > accurate enough to drive the residual below 5e-13. You could just try > -snes_atol 1e-12 > unless you really think you need to be more accurate. Also have you > non-dimensionalized > your problem? > > Thanks, > > Matt > > >> 0 SNES Function norm 3.775139483936e-07 >> Line search: Using full step: fnorm 3.775139483936e-07 gnorm >> 1.043499007769e-11 >> 1 SNES Function norm 1.043499007769e-11 >> Line search: Using full step: fnorm 1.043499007769e-11 gnorm >> 5.302082026305e-12 >> 2 SNES Function norm 5.302082026305e-12 >> Line search: Using full step: fnorm 5.302082026305e-12 gnorm >> 3.136358665117e-12 >> 3 SNES Function norm 3.136358665117e-12 >> Line search: Using full step: fnorm 3.136358665117e-12 gnorm >> 2.170819065257e-12 >> 4 SNES Function norm 2.170819065257e-12 >> Line search: Using full step: fnorm 2.170819065257e-12 gnorm >> 1.475760790713e-12 >> 5 SNES Function norm 1.475760790713e-12 >> Line search: Using full step: fnorm 1.475760790713e-12 gnorm >> 1.130642522675e-12 >> 6 SNES Function norm 1.130642522675e-12 >> Line search: Using full step: fnorm 1.130642522675e-12 gnorm >> 8.647827205825e-13 >> 7 SNES Function norm 8.647827205825e-13 >> Line search: Using full step: fnorm 8.647827205825e-13 gnorm >> 7.403983354386e-13 >> 8 SNES Function norm 7.403983354386e-13 >> Line search: Using full step: fnorm 7.403983354386e-13 gnorm >> 6.291615062788e-13 >> 9 SNES Function norm 6.291615062788e-13 >> Line search: Using full step: fnorm 6.291615062788e-13 gnorm >> 5.682820624094e-13 >> 10 SNES Function norm 5.682820624094e-13 >> Line search: Using full step: fnorm 5.682820624094e-13 gnorm >> 5.300267435005e-13 >> 11 SNES Function norm 5.300267435005e-13 >> Line search: Using full step: fnorm 5.300267435005e-13 gnorm >> 5.096339188439e-13 >> 12 SNES Function norm 5.096339188439e-13 >> Line search: Using full step: fnorm 5.096339188439e-13 gnorm >> 4.970340343183e-13 >> 13 SNES Function norm 4.970340343183e-13 >> Line search: Using full step: fnorm 4.970340343183e-13 gnorm >> 4.916509482510e-13 >> 14 SNES Function norm 4.916509482510e-13 >> Line search: Using full step: fnorm 4.916509482510e-13 gnorm >> 4.875840608086e-13 >> 15 SNES Function norm 4.875840608086e-13 >> Line search: Using full step: fnorm 4.875840608086e-13 gnorm >> 4.859609948396e-13 >> 16 SNES Function norm 4.859609948396e-13 >> Line search: Using full step: fnorm 4.859609948396e-13 gnorm >> 4.846457390135e-13 >> 17 SNES Function norm 4.846457390135e-13 >> Line search: Using full step: fnorm 4.846457390135e-13 gnorm >> 4.840667643605e-13 >> 18 SNES Function norm 4.840667643605e-13 >> Line search: Using full step: fnorm 4.840667643605e-13 gnorm >> 4.837878467904e-13 >> 19 SNES Function norm 4.837878467904e-13 >> Line search: Using full step: fnorm 4.837878467904e-13 gnorm >> 4.835385298533e-13 >> 20 SNES Function norm 4.835385298533e-13 >> Line search: Using full step: fnorm 4.835385298533e-13 gnorm >> 4.833767981145e-13 >> 21 SNES Function norm 4.833767981145e-13 >> Line search: Aborted due to ynorm < stol*xnorm (6.489173391862e-15 >> < 2.387840730261e-06) and inadequate full step. >> Nonlinear smu_ solve did not converge due to DIVERGED_LINE_SEARCH >> iterations 21 >> >> >> >> >> 2018-09-11 23:37 GMT-05:00 Smith, Barry F. : >> >>> >>> Do you start with initial conditions of 0 <= u_i <= 1 ? >>> >>> Run with -snes_monitor -snes_converged_reason >>> -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the >>> output >>> >>> Barry >>> >>> >>> > On Sep 11, 2018, at 11:33 PM, Josh L wrote: >>> > >>> > Hi, >>> > >>> > I am using SNES to solve an nonlinear equation f(u), and I know all >>> the u_i should be 0 and 1. >>> > >>> > First, I use SNES without constraint, and it converges. >>> > >>> > But, If I set >>> > SNESSetType(snes,SNESVINEWTONRSLS,ierr) >>> > SNESVISetVariableBounds(snes,xl,xu,ierr) >>> > >>> > where xl and xu is vector, and xl_i=0 and xu_i=1 >>> > >>> > then SNES fails to converge, because linesearch fails(snes reason = >>> -6), and the norm of residual is not reducing(the norm of incremental >>> solution is reducing) >>> > >>> > The reason to add constraint is that I want to implement some >>> irreversibility. >>> > >>> > >>> > Thanks, >>> > Josh >>> > >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysjosh.lo at gmail.com Wed Sep 12 11:01:44 2018 From: ysjosh.lo at gmail.com (Josh L) Date: Wed, 12 Sep 2018 11:01:44 -0500 Subject: [petsc-users] TAO with time dependent problem In-Reply-To: <5F7B684F-528C-466F-9504-62FCD4768EE0@anl.gov> References: <5F7B684F-528C-466F-9504-62FCD4768EE0@anl.gov> Message-ID: Hi Alp, Yes, my minimal energy problem reduces to the solution of (1). Is there any reference on the problem you mention in the last paragraph? Thanks, Josh 2018-09-12 9:26 GMT-05:00 Dener, Alp : > Hi Josh, > > In the FormFunctionAndGradient() function, the gradient you compute and > provide TAO must be the gradient of the objective function with respect to > the optimization variables. If your optimization variables are u, and your > objective function is J(u), then FormFunctionGradient would need to compute > J(u) and dJ/du. > > I?m not entirely sure if I understand your problem correctly, but it seems > that your minimal energy problem reduces to the solution of equation (1). > Is this correct? > > TAO is not the right package to solve time-dependent PDEs like (1) > directly. You would want to use TS for that. However, TAO can be used to > solve the original energy minimization problem. A good example of this is > the canonical obstacle problem that aims to minimize the Dirichlet energy > function subject to a constraint that represents the obstacle. That problem > reduces to the solution of the Laplace equation with the appropriate > boundary conditions matching the obstacle, which would be discretized and > solved as a stand-alone PDE. However, the same problem can also be solved > as an optimization problem where the objective function is the Dirichlet > energy function, and the associated gradient is related to the stiffness > matrix associated with the Laplace equation. The two formulations are > mathematically equivalent. If I?m interpreting your problem correctly, you > should be able to take a similar approach, but you would need to derive the > gradient of your energy function and see how you can construct it in > relation to equation (1). > > ?? > *Alp Dener* > Argonne National Laboratory > http://www.mcs.anl.gov/person/alp-dener > > > > > > > > > > > On Sep 11, 2018, at 3:51 PM, Josh L wrote: > > Hi, > > I am using PETSc TAO to solve the following equation obtained from > minimizing certain energy and constitutive model: > > u_xx - u - u_t =0 (1) > > For simplicity, the coefficients are neglected. > In the routine to form function value and gradient, I use (1) to form my > gradient vector, and my energy definition to calculate function value.Is it > correct? > > I only found one tao example that solves time dependent problem, but it is > using TS which I am not using. > > > Thanks, > Josh > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adener at anl.gov Wed Sep 12 11:45:17 2018 From: adener at anl.gov (Dener, Alp) Date: Wed, 12 Sep 2018 16:45:17 +0000 Subject: [petsc-users] TAO with time dependent problem In-Reply-To: References: <5F7B684F-528C-466F-9504-62FCD4768EE0@anl.gov> Message-ID: Hi Josh, Luis Caffarelli has some writings from his Fermi lecture on the Obstacle problem that may or may not be helpful. He primarily looks at it in the context of variational inequalities. http://user.math.uzh.ch/ros-oton/Caffarelli-The-obstacle-problem-SNS-1998.pdf I don?t have an academic reference for the solution of this as an optimization problem, but I have implemented it before as a TAO/MFEM test case for the ATPESC 2018 Hands-On Lessons. https://xsdk-project.github.io/ATPESC2018HandsOnLessons/lessons/obstacle_tao/ Hope that helps. ?? Alp Dener Argonne National Laboratory http://www.mcs.anl.gov/person/alp-dener On Sep 12, 2018, at 11:01 AM, Josh L > wrote: Hi Alp, Yes, my minimal energy problem reduces to the solution of (1). Is there any reference on the problem you mention in the last paragraph? Thanks, Josh 2018-09-12 9:26 GMT-05:00 Dener, Alp >: Hi Josh, In the FormFunctionAndGradient() function, the gradient you compute and provide TAO must be the gradient of the objective function with respect to the optimization variables. If your optimization variables are u, and your objective function is J(u), then FormFunctionGradient would need to compute J(u) and dJ/du. I?m not entirely sure if I understand your problem correctly, but it seems that your minimal energy problem reduces to the solution of equation (1). Is this correct? TAO is not the right package to solve time-dependent PDEs like (1) directly. You would want to use TS for that. However, TAO can be used to solve the original energy minimization problem. A good example of this is the canonical obstacle problem that aims to minimize the Dirichlet energy function subject to a constraint that represents the obstacle. That problem reduces to the solution of the Laplace equation with the appropriate boundary conditions matching the obstacle, which would be discretized and solved as a stand-alone PDE. However, the same problem can also be solved as an optimization problem where the objective function is the Dirichlet energy function, and the associated gradient is related to the stiffness matrix associated with the Laplace equation. The two formulations are mathematically equivalent. If I?m interpreting your problem correctly, you should be able to take a similar approach, but you would need to derive the gradient of your energy function and see how you can construct it in relation to equation (1). ?? Alp Dener Argonne National Laboratory http://www.mcs.anl.gov/person/alp-dener On Sep 11, 2018, at 3:51 PM, Josh L > wrote: Hi, I am using PETSc TAO to solve the following equation obtained from minimizing certain energy and constitutive model: u_xx - u - u_t =0 (1) For simplicity, the coefficients are neglected. In the routine to form function value and gradient, I use (1) to form my gradient vector, and my energy definition to calculate function value.Is it correct? I only found one tao example that solves time dependent problem, but it is using TS which I am not using. Thanks, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Sep 12 12:02:39 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 12 Sep 2018 17:02:39 +0000 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> Message-ID: <29A829ED-8055-4C68-A751-25C8F27F6A46@mcs.anl.gov> You have too tight a convergence tolerance for your problem. You can't expect to get more than 1.e-12 as the minimum residual norm or even less. How close is your solution to 1 and -1? If you really need much higher convergence you can try ./configure --with-precision=__float128 Barry > On Sep 11, 2018, at 11:53 PM, Josh L wrote: > > Yes, I initialize all u_i to 1.0 > > > > 2018-09-11 23:37 GMT-05:00 Smith, Barry F. : > > Do you start with initial conditions of 0 <= u_i <= 1 ? > > Run with -snes_monitor -snes_converged_reason -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the output > > Barry > > > > On Sep 11, 2018, at 11:33 PM, Josh L wrote: > > > > Hi, > > > > I am using SNES to solve an nonlinear equation f(u), and I know all the u_i should be 0 and 1. > > > > First, I use SNES without constraint, and it converges. > > > > But, If I set > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > > SNESVISetVariableBounds(snes,xl,xu,ierr) > > > > where xl and xu is vector, and xl_i=0 and xu_i=1 > > > > then SNES fails to converge, because linesearch fails(snes reason = -6), and the norm of residual is not reducing(the norm of incremental solution is reducing) > > > > The reason to add constraint is that I want to implement some irreversibility. > > > > > > Thanks, > > Josh > > > > From ysjosh.lo at gmail.com Wed Sep 12 15:32:06 2018 From: ysjosh.lo at gmail.com (Josh L) Date: Wed, 12 Sep 2018 15:32:06 -0500 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: <29A829ED-8055-4C68-A751-25C8F27F6A46@mcs.anl.gov> References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> <29A829ED-8055-4C68-A751-25C8F27F6A46@mcs.anl.gov> Message-ID: Hi, my solution is mostly very close to 1. only for a very small area where solution goes from 0 to 1(a smeared crack). I set -snes_atol 1e-7 and it is converging. I've noticed the following: There is a difference between the function norm. I calculate the function norm in FormFunction, so every time it is called it gives the function norm , and the result is different from the function norm given by -snes_monitor if i set SNESSetType(snes,SNESVINEWTONRSLS,ierr) SNESVISetVariableBounds(snes,xl,xu,ierr) The function norm calculated in FromFunction is NOT reducing, however, the function norm given by -snes_monitor is reducing They are the same if I just use regular SNES without setting variable bounds. Thanks, Josh 2018-09-12 12:02 GMT-05:00 Smith, Barry F. : > > You have too tight a convergence tolerance for your problem. You > can't expect to get more than 1.e-12 as the minimum residual norm or even > less. > > How close is your solution to 1 and -1? > > If you really need much higher convergence you can try ./configure > --with-precision=__float128 > > > Barry > > > On Sep 11, 2018, at 11:53 PM, Josh L wrote: > > > > Yes, I initialize all u_i to 1.0 > > > > > > > > 2018-09-11 23:37 GMT-05:00 Smith, Barry F. : > > > > Do you start with initial conditions of 0 <= u_i <= 1 ? > > > > Run with -snes_monitor -snes_converged_reason > -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the > output > > > > Barry > > > > > > > On Sep 11, 2018, at 11:33 PM, Josh L wrote: > > > > > > Hi, > > > > > > I am using SNES to solve an nonlinear equation f(u), and I know all > the u_i should be 0 and 1. > > > > > > First, I use SNES without constraint, and it converges. > > > > > > But, If I set > > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > > > SNESVISetVariableBounds(snes,xl,xu,ierr) > > > > > > where xl and xu is vector, and xl_i=0 and xu_i=1 > > > > > > then SNES fails to converge, because linesearch fails(snes reason = > -6), and the norm of residual is not reducing(the norm of incremental > solution is reducing) > > > > > > The reason to add constraint is that I want to implement some > irreversibility. > > > > > > > > > Thanks, > > > Josh > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera-w at sdsu.edu Wed Sep 12 16:13:16 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Wed, 12 Sep 2018 14:13:16 -0700 Subject: [petsc-users] Requesting multi-GPUs Message-ID: Hello guys, I am working in a multi-gpu cluster and i want to request 2 or more GPUs, how can i do that from PETSc? evidently mpirun -n # is for requesting processors, but what if i want to use one mpi processor but several GPUs instead? Also, i understand the GPU handles the linear system solver, but what about the data management? can i do DMs for other than the linear solver using the GPUs ? Thanks once more, -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Sep 12 16:16:05 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 12 Sep 2018 21:16:05 +0000 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> <29A829ED-8055-4C68-A751-25C8F27F6A46@mcs.anl.gov> Message-ID: <75D93FDF-8086-416B-A03A-46F372DA718C@mcs.anl.gov> The function norm given by the -snes_monitor is the function norm for the DVI problem (which depends on where the solution is constrained) while FormFunction() norm you compute is for the original unconstrained problem. I am still not sure why you solve this as a DVI. If you solve it without the constraints do you get the "wrong" answer? Barry > On Sep 12, 2018, at 3:32 PM, Josh L wrote: > > Hi, > > my solution is mostly very close to 1. only for a very small area where solution goes from 0 to 1(a smeared crack). > > I set -snes_atol 1e-7 and it is converging. > > I've noticed the following: > > There is a difference between the function norm. > > I calculate the function norm in FormFunction, so every time it is called it gives the function norm > , and the result is different from the function norm given by -snes_monitor if i set > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > SNESVISetVariableBounds(snes,xl,xu,ierr) > > The function norm calculated in FromFunction is NOT reducing, however, the function norm given by -snes_monitor is reducing > They are the same if I just use regular SNES without setting variable bounds. > > > Thanks, > Josh > > 2018-09-12 12:02 GMT-05:00 Smith, Barry F. : > > You have too tight a convergence tolerance for your problem. You can't expect to get more than 1.e-12 as the minimum residual norm or even less. > > How close is your solution to 1 and -1? > > If you really need much higher convergence you can try ./configure --with-precision=__float128 > > > Barry > > > On Sep 11, 2018, at 11:53 PM, Josh L wrote: > > > > Yes, I initialize all u_i to 1.0 > > > > > > > > 2018-09-11 23:37 GMT-05:00 Smith, Barry F. : > > > > Do you start with initial conditions of 0 <= u_i <= 1 ? > > > > Run with -snes_monitor -snes_converged_reason -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the output > > > > Barry > > > > > > > On Sep 11, 2018, at 11:33 PM, Josh L wrote: > > > > > > Hi, > > > > > > I am using SNES to solve an nonlinear equation f(u), and I know all the u_i should be 0 and 1. > > > > > > First, I use SNES without constraint, and it converges. > > > > > > But, If I set > > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > > > SNESVISetVariableBounds(snes,xl,xu,ierr) > > > > > > where xl and xu is vector, and xl_i=0 and xu_i=1 > > > > > > then SNES fails to converge, because linesearch fails(snes reason = -6), and the norm of residual is not reducing(the norm of incremental solution is reducing) > > > > > > The reason to add constraint is that I want to implement some irreversibility. > > > > > > > > > Thanks, > > > Josh > > > > > > > > > From knepley at gmail.com Wed Sep 12 16:20:58 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Sep 2018 17:20:58 -0400 Subject: [petsc-users] Requesting multi-GPUs In-Reply-To: References: Message-ID: On Wed, Sep 12, 2018 at 5:13 PM Manuel Valera wrote: > Hello guys, > > I am working in a multi-gpu cluster and i want to request 2 or more GPUs, > how can i do that from PETSc? evidently mpirun -n # is for requesting > processors, but what if i want to use one mpi processor but several GPUs > instead? > We do not do that. You would run the same number of MPI processes as GPUs. Note that you can have more than 1 MPI process on a processor. Matt > Also, i understand the GPU handles the linear system solver, but what > about the data management? can i do DMs for other than the linear solver > using the GPUs > ? > > Thanks once more, > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 12 16:23:05 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Sep 2018 17:23:05 -0400 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: <75D93FDF-8086-416B-A03A-46F372DA718C@mcs.anl.gov> References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> <29A829ED-8055-4C68-A751-25C8F27F6A46@mcs.anl.gov> <75D93FDF-8086-416B-A03A-46F372DA718C@mcs.anl.gov> Message-ID: On Wed, Sep 12, 2018 at 5:16 PM Smith, Barry F. wrote: > > The function norm given by the -snes_monitor is the function norm for > the DVI problem (which depends on where the solution is constrained) while > FormFunction() norm you compute is for the original unconstrained problem. > > I am still not sure why you solve this as a DVI. If you solve it > without the constraints do you get the "wrong" answer? > If this is a variational crack problem, you can solve it unconstrained if you use a quadratic penalty term. However, that induces spurious long range communication between crack tips. If you bound the phase field between 0 and 1, you can use a linear penalty which has physical behavior. Matt > Barry > > > > On Sep 12, 2018, at 3:32 PM, Josh L wrote: > > > > Hi, > > > > my solution is mostly very close to 1. only for a very small area where > solution goes from 0 to 1(a smeared crack). > > > > I set -snes_atol 1e-7 and it is converging. > > > > I've noticed the following: > > > > There is a difference between the function norm. > > > > I calculate the function norm in FormFunction, so every time it is > called it gives the function norm > > , and the result is different from the function norm given by > -snes_monitor if i set > > > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > > SNESVISetVariableBounds(snes,xl,xu,ierr) > > > > The function norm calculated in FromFunction is NOT reducing, however, > the function norm given by -snes_monitor is reducing > > They are the same if I just use regular SNES without setting variable > bounds. > > > > > > Thanks, > > Josh > > > > 2018-09-12 12:02 GMT-05:00 Smith, Barry F. : > > > > You have too tight a convergence tolerance for your problem. You > can't expect to get more than 1.e-12 as the minimum residual norm or even > less. > > > > How close is your solution to 1 and -1? > > > > If you really need much higher convergence you can try ./configure > --with-precision=__float128 > > > > > > Barry > > > > > On Sep 11, 2018, at 11:53 PM, Josh L wrote: > > > > > > Yes, I initialize all u_i to 1.0 > > > > > > > > > > > > 2018-09-11 23:37 GMT-05:00 Smith, Barry F. : > > > > > > Do you start with initial conditions of 0 <= u_i <= 1 ? > > > > > > Run with -snes_monitor -snes_converged_reason > -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the > output > > > > > > Barry > > > > > > > > > > On Sep 11, 2018, at 11:33 PM, Josh L wrote: > > > > > > > > Hi, > > > > > > > > I am using SNES to solve an nonlinear equation f(u), and I know all > the u_i should be 0 and 1. > > > > > > > > First, I use SNES without constraint, and it converges. > > > > > > > > But, If I set > > > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > > > > SNESVISetVariableBounds(snes,xl,xu,ierr) > > > > > > > > where xl and xu is vector, and xl_i=0 and xu_i=1 > > > > > > > > then SNES fails to converge, because linesearch fails(snes reason = > -6), and the norm of residual is not reducing(the norm of incremental > solution is reducing) > > > > > > > > The reason to add constraint is that I want to implement some > irreversibility. > > > > > > > > > > > > Thanks, > > > > Josh > > > > > > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera-w at sdsu.edu Wed Sep 12 16:31:15 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Wed, 12 Sep 2018 14:31:15 -0700 Subject: [petsc-users] Requesting multi-GPUs In-Reply-To: References: Message-ID: Ok then, how can i try getting more than one GPU with the same number of MPI processes? Thanks, On Wed, Sep 12, 2018 at 2:20 PM, Matthew Knepley wrote: > On Wed, Sep 12, 2018 at 5:13 PM Manuel Valera wrote: > >> Hello guys, >> >> I am working in a multi-gpu cluster and i want to request 2 or more GPUs, >> how can i do that from PETSc? evidently mpirun -n # is for requesting >> processors, but what if i want to use one mpi processor but several GPUs >> instead? >> > > We do not do that. You would run the same number of MPI processes as GPUs. > Note that > you can have more than 1 MPI process on a processor. > > Matt > > >> Also, i understand the GPU handles the linear system solver, but what >> about the data management? can i do DMs for other than the linear solver >> using the GPUs >> ? >> >> Thanks once more, >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysjosh.lo at gmail.com Wed Sep 12 16:31:16 2018 From: ysjosh.lo at gmail.com (Josh L) Date: Wed, 12 Sep 2018 16:31:16 -0500 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> <29A829ED-8055-4C68-A751-25C8F27F6A46@mcs.anl.gov> <75D93FDF-8086-416B-A03A-46F372DA718C@mcs.anl.gov> Message-ID: Barry, 0 represents fully damaged and 1 is intact. if I unload it, the crack shouldn't heal(solution goes back to 1), so I set the constraint as 0 <= u^{i+1} <= u^{i} i is either load step or time step. Matt, Yes, that is what I am going to do next. a linear penalty gives a quadratic distribution instead of the exponential distribution given by quadratic penalty term. Thanks, Josh 2018-09-12 16:23 GMT-05:00 Matthew Knepley : > On Wed, Sep 12, 2018 at 5:16 PM Smith, Barry F. > wrote: > >> >> The function norm given by the -snes_monitor is the function norm for >> the DVI problem (which depends on where the solution is constrained) while >> FormFunction() norm you compute is for the original unconstrained problem. >> >> I am still not sure why you solve this as a DVI. If you solve it >> without the constraints do you get the "wrong" answer? >> > > If this is a variational crack problem, you can solve it unconstrained if > you use a quadratic penalty term. However, > that induces spurious long range communication between crack tips. If you > bound the phase field between 0 and 1, > you can use a linear penalty which has physical behavior. > > Matt > > >> Barry >> >> >> > On Sep 12, 2018, at 3:32 PM, Josh L wrote: >> > >> > Hi, >> > >> > my solution is mostly very close to 1. only for a very small area >> where solution goes from 0 to 1(a smeared crack). >> > >> > I set -snes_atol 1e-7 and it is converging. >> > >> > I've noticed the following: >> > >> > There is a difference between the function norm. >> > >> > I calculate the function norm in FormFunction, so every time it is >> called it gives the function norm >> > , and the result is different from the function norm given by >> -snes_monitor if i set >> > >> > SNESSetType(snes,SNESVINEWTONRSLS,ierr) >> > SNESVISetVariableBounds(snes,xl,xu,ierr) >> > >> > The function norm calculated in FromFunction is NOT reducing, however, >> the function norm given by -snes_monitor is reducing >> > They are the same if I just use regular SNES without setting variable >> bounds. >> > >> > >> > Thanks, >> > Josh >> > >> > 2018-09-12 12:02 GMT-05:00 Smith, Barry F. : >> > >> > You have too tight a convergence tolerance for your problem. You >> can't expect to get more than 1.e-12 as the minimum residual norm or even >> less. >> > >> > How close is your solution to 1 and -1? >> > >> > If you really need much higher convergence you can try >> ./configure --with-precision=__float128 >> > >> > >> > Barry >> > >> > > On Sep 11, 2018, at 11:53 PM, Josh L wrote: >> > > >> > > Yes, I initialize all u_i to 1.0 >> > > >> > > >> > > >> > > 2018-09-11 23:37 GMT-05:00 Smith, Barry F. : >> > > >> > > Do you start with initial conditions of 0 <= u_i <= 1 ? >> > > >> > > Run with -snes_monitor -snes_converged_reason >> -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the >> output >> > > >> > > Barry >> > > >> > > >> > > > On Sep 11, 2018, at 11:33 PM, Josh L wrote: >> > > > >> > > > Hi, >> > > > >> > > > I am using SNES to solve an nonlinear equation f(u), and I know all >> the u_i should be 0 and 1. >> > > > >> > > > First, I use SNES without constraint, and it converges. >> > > > >> > > > But, If I set >> > > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) >> > > > SNESVISetVariableBounds(snes,xl,xu,ierr) >> > > > >> > > > where xl and xu is vector, and xl_i=0 and xu_i=1 >> > > > >> > > > then SNES fails to converge, because linesearch fails(snes reason = >> -6), and the norm of residual is not reducing(the norm of incremental >> solution is reducing) >> > > > >> > > > The reason to add constraint is that I want to implement some >> irreversibility. >> > > > >> > > > >> > > > Thanks, >> > > > Josh >> > > > >> > > >> > > >> > >> > >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 12 16:38:36 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Sep 2018 17:38:36 -0400 Subject: [petsc-users] Requesting multi-GPUs In-Reply-To: References: Message-ID: On Wed, Sep 12, 2018 at 5:31 PM Manuel Valera wrote: > Ok then, how can i try getting more than one GPU with the same number of > MPI processes? > I do not believe we handle more than one GPU per MPI process. Is that what you are asking? Thanks, Matt > Thanks, > > On Wed, Sep 12, 2018 at 2:20 PM, Matthew Knepley > wrote: > >> On Wed, Sep 12, 2018 at 5:13 PM Manuel Valera wrote: >> >>> Hello guys, >>> >>> I am working in a multi-gpu cluster and i want to request 2 or more >>> GPUs, how can i do that from PETSc? evidently mpirun -n # is for requesting >>> processors, but what if i want to use one mpi processor but several GPUs >>> instead? >>> >> >> We do not do that. You would run the same number of MPI processes as >> GPUs. Note that >> you can have more than 1 MPI process on a processor. >> >> Matt >> >> >>> Also, i understand the GPU handles the linear system solver, but what >>> about the data management? can i do DMs for other than the linear solver >>> using the GPUs >>> ? >>> >>> Thanks once more, >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Sep 12 16:41:47 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 12 Sep 2018 21:41:47 +0000 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> <29A829ED-8055-4C68-A751-25C8F27F6A46@mcs.anl.gov> <75D93FDF-8086-416B-A03A-46F372DA718C@mcs.anl.gov> Message-ID: <26155E11-F833-4DB5-AF5A-4FD7F38149D6@mcs.anl.gov> > On Sep 12, 2018, at 4:31 PM, Josh L wrote: > > Barry, > > 0 represents fully damaged and 1 is intact. if I unload it, the crack shouldn't heal(solution goes back to 1), so I set the constraint as > 0 <= u^{i+1} <= u^{i} > i is either load step or time step. Ok, then how do you know that the entire FormFunction() norm will/should even decrease as SNES converges to the DVI solution? Barry > > > > Matt, > > Yes, that is what I am going to do next. a linear penalty gives a quadratic distribution instead of the exponential distribution given by quadratic penalty term. > > > Thanks, > Josh > > > > 2018-09-12 16:23 GMT-05:00 Matthew Knepley : > On Wed, Sep 12, 2018 at 5:16 PM Smith, Barry F. wrote: > > The function norm given by the -snes_monitor is the function norm for the DVI problem (which depends on where the solution is constrained) while FormFunction() norm you compute is for the original unconstrained problem. > > I am still not sure why you solve this as a DVI. If you solve it without the constraints do you get the "wrong" answer? > > If this is a variational crack problem, you can solve it unconstrained if you use a quadratic penalty term. However, > that induces spurious long range communication between crack tips. If you bound the phase field between 0 and 1, > you can use a linear penalty which has physical behavior. > > Matt > > Barry > > > > On Sep 12, 2018, at 3:32 PM, Josh L wrote: > > > > Hi, > > > > my solution is mostly very close to 1. only for a very small area where solution goes from 0 to 1(a smeared crack). > > > > I set -snes_atol 1e-7 and it is converging. > > > > I've noticed the following: > > > > There is a difference between the function norm. > > > > I calculate the function norm in FormFunction, so every time it is called it gives the function norm > > , and the result is different from the function norm given by -snes_monitor if i set > > > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > > SNESVISetVariableBounds(snes,xl,xu,ierr) > > > > The function norm calculated in FromFunction is NOT reducing, however, the function norm given by -snes_monitor is reducing > > They are the same if I just use regular SNES without setting variable bounds. > > > > > > Thanks, > > Josh > > > > 2018-09-12 12:02 GMT-05:00 Smith, Barry F. : > > > > You have too tight a convergence tolerance for your problem. You can't expect to get more than 1.e-12 as the minimum residual norm or even less. > > > > How close is your solution to 1 and -1? > > > > If you really need much higher convergence you can try ./configure --with-precision=__float128 > > > > > > Barry > > > > > On Sep 11, 2018, at 11:53 PM, Josh L wrote: > > > > > > Yes, I initialize all u_i to 1.0 > > > > > > > > > > > > 2018-09-11 23:37 GMT-05:00 Smith, Barry F. : > > > > > > Do you start with initial conditions of 0 <= u_i <= 1 ? > > > > > > Run with -snes_monitor -snes_converged_reason -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the output > > > > > > Barry > > > > > > > > > > On Sep 11, 2018, at 11:33 PM, Josh L wrote: > > > > > > > > Hi, > > > > > > > > I am using SNES to solve an nonlinear equation f(u), and I know all the u_i should be 0 and 1. > > > > > > > > First, I use SNES without constraint, and it converges. > > > > > > > > But, If I set > > > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > > > > SNESVISetVariableBounds(snes,xl,xu,ierr) > > > > > > > > where xl and xu is vector, and xl_i=0 and xu_i=1 > > > > > > > > then SNES fails to converge, because linesearch fails(snes reason = -6), and the norm of residual is not reducing(the norm of incremental solution is reducing) > > > > > > > > The reason to add constraint is that I want to implement some irreversibility. > > > > > > > > > > > > Thanks, > > > > Josh > > > > > > > > > > > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > From ysjosh.lo at gmail.com Wed Sep 12 17:32:44 2018 From: ysjosh.lo at gmail.com (Josh L) Date: Wed, 12 Sep 2018 17:32:44 -0500 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: <26155E11-F833-4DB5-AF5A-4FD7F38149D6@mcs.anl.gov> References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> <29A829ED-8055-4C68-A751-25C8F27F6A46@mcs.anl.gov> <75D93FDF-8086-416B-A03A-46F372DA718C@mcs.anl.gov> <26155E11-F833-4DB5-AF5A-4FD7F38149D6@mcs.anl.gov> Message-ID: Barry, Yes, you are right, the entire FormFunction norm doesn't necessarily decreases as SNES converges to the DVI solution. Thanks, Josh 2018-09-12 16:41 GMT-05:00 Smith, Barry F. : > > > > On Sep 12, 2018, at 4:31 PM, Josh L wrote: > > > > Barry, > > > > 0 represents fully damaged and 1 is intact. if I unload it, the crack > shouldn't heal(solution goes back to 1), so I set the constraint as > > 0 <= u^{i+1} <= u^{i} > > i is either load step or time step. > > Ok, then how do you know that the entire FormFunction() norm > will/should even decrease as SNES converges to the DVI solution? > > Barry > > > > > > > > > Matt, > > > > Yes, that is what I am going to do next. a linear penalty gives a > quadratic distribution instead of the exponential distribution given by > quadratic penalty term. > > > > > > Thanks, > > Josh > > > > > > > > 2018-09-12 16:23 GMT-05:00 Matthew Knepley : > > On Wed, Sep 12, 2018 at 5:16 PM Smith, Barry F. > wrote: > > > > The function norm given by the -snes_monitor is the function norm for > the DVI problem (which depends on where the solution is constrained) while > FormFunction() norm you compute is for the original unconstrained problem. > > > > I am still not sure why you solve this as a DVI. If you solve it > without the constraints do you get the "wrong" answer? > > > > If this is a variational crack problem, you can solve it unconstrained > if you use a quadratic penalty term. However, > > that induces spurious long range communication between crack tips. If > you bound the phase field between 0 and 1, > > you can use a linear penalty which has physical behavior. > > > > Matt > > > > Barry > > > > > > > On Sep 12, 2018, at 3:32 PM, Josh L wrote: > > > > > > Hi, > > > > > > my solution is mostly very close to 1. only for a very small area > where solution goes from 0 to 1(a smeared crack). > > > > > > I set -snes_atol 1e-7 and it is converging. > > > > > > I've noticed the following: > > > > > > There is a difference between the function norm. > > > > > > I calculate the function norm in FormFunction, so every time it is > called it gives the function norm > > > , and the result is different from the function norm given by > -snes_monitor if i set > > > > > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > > > SNESVISetVariableBounds(snes,xl,xu,ierr) > > > > > > The function norm calculated in FromFunction is NOT reducing, however, > the function norm given by -snes_monitor is reducing > > > They are the same if I just use regular SNES without setting variable > bounds. > > > > > > > > > Thanks, > > > Josh > > > > > > 2018-09-12 12:02 GMT-05:00 Smith, Barry F. : > > > > > > You have too tight a convergence tolerance for your problem. You > can't expect to get more than 1.e-12 as the minimum residual norm or even > less. > > > > > > How close is your solution to 1 and -1? > > > > > > If you really need much higher convergence you can try > ./configure --with-precision=__float128 > > > > > > > > > Barry > > > > > > > On Sep 11, 2018, at 11:53 PM, Josh L wrote: > > > > > > > > Yes, I initialize all u_i to 1.0 > > > > > > > > > > > > > > > > 2018-09-11 23:37 GMT-05:00 Smith, Barry F. : > > > > > > > > Do you start with initial conditions of 0 <= u_i <= 1 ? > > > > > > > > Run with -snes_monitor -snes_converged_reason > -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the > output > > > > > > > > Barry > > > > > > > > > > > > > On Sep 11, 2018, at 11:33 PM, Josh L wrote: > > > > > > > > > > Hi, > > > > > > > > > > I am using SNES to solve an nonlinear equation f(u), and I know > all the u_i should be 0 and 1. > > > > > > > > > > First, I use SNES without constraint, and it converges. > > > > > > > > > > But, If I set > > > > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > > > > > SNESVISetVariableBounds(snes,xl,xu,ierr) > > > > > > > > > > where xl and xu is vector, and xl_i=0 and xu_i=1 > > > > > > > > > > then SNES fails to converge, because linesearch fails(snes reason > = -6), and the norm of residual is not reducing(the norm of incremental > solution is reducing) > > > > > > > > > > The reason to add constraint is that I want to implement some > irreversibility. > > > > > > > > > > > > > > > Thanks, > > > > > Josh > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellen.price at cfa.harvard.edu Wed Sep 12 20:55:49 2018 From: ellen.price at cfa.harvard.edu (Ellen M. Price) Date: Wed, 12 Sep 2018 21:55:49 -0400 Subject: [petsc-users] Question about HDF5 viewer groups Message-ID: Hi there, I'm running a complex PETSc program that outputs a LOT of data to an HDF5 file. As such, I'd like to separate my output into groups to make it easier to traverse in the analysis phase. I'm either confused about the nomenclature or the usage of PetscViewerHDF5PushGroup/PetscViewerHDF5PopGroup. From a quick skim of the source, it looks like the PushGroup function builds up a linked list of group names? Also, just the name suggests that there should be a "stack" of group names stored somewhere. So I would expect, after calling PetscViewerHDF5PushGroup twice, say, PetscViewerHDF5PushGroup(h5viewer, "group1"); PetscViewerHDF5PushGroup(h5viewer, "group2"); VecView(x, h5viewer); that I would get my vector output in /group1/group2/x. This is obviously not what's actually happening. If I run http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/examples/tutorials/ex19.c, for example, I get, from h5dump: HDF5 "ex19.h5" { FILE_CONTENTS { group / dataset /TestVec group /testBlockSize dataset /testBlockSize/TestVec2 group /testTimestep dataset /testTimestep/TestVec2 } } even though multiple groups are "pushed" in sequence. My question is, have I misunderstood the connotation of pushing/popping groups? How can I properly create a group and then a subgroup? Every attempt at doing this: PetscViewerHDF5PushGroup(h5viewer, "/group1"); PetscViewerHDF5PushGroup(h5viewer, "/group1/group2"); VecView(x, h5viewer); leads to a long string of HDF5 library errors, like: HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 0: #000: ../../../src/H5L.c line 824 in H5Lexists(): unable to get link info major: Symbol table minor: Object not found #001: ../../../src/H5L.c line 2765 in H5L_exists(): path doesn't exist major: Symbol table minor: Object already exists #002: ../../../src/H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed major: Symbol table minor: Object not found #003: ../../../src/H5Gtraverse.c line 755 in H5G_traverse_real(): component not found major: Symbol table minor: Object not found HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 0: #000: ../../../src/H5G.c line 314 in H5Gcreate2(): unable to create group major: Symbol table minor: Unable to initialize object #001: ../../../src/H5Gint.c line 194 in H5G__create_named(): unable to create and link to group major: Symbol table minor: Unable to initialize object #002: ../../../src/H5L.c line 1638 in H5L_link_object(): unable to create new link to object major: Links minor: Unable to initialize object #003: ../../../src/H5L.c line 1882 in H5L_create_real(): can't insert link major: Symbol table minor: Unable to insert object #004: ../../../src/H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed major: Symbol table minor: Object not found #005: ../../../src/H5Gtraverse.c line 755 in H5G_traverse_real(): component not found major: Symbol table minor: Object not found Thanks in advance, Ellen Price From no-reply at webtalk.co Wed Sep 12 21:50:51 2018 From: no-reply at webtalk.co (Webtalk) Date: Thu, 13 Sep 2018 02:50:51 +0000 (UTC) Subject: [petsc-users] Doug Hulse wants to connect... Message-ID: <1266010031.59921.1536807050658@2fc4e2dbcc4b> Join Webtalk You have been invited Doug Hulse. Accept Invitation & Connect with Doug Hulse. https://www.webtalk.co/be/invite/1464904?type=webtalk -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 13 04:56:49 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 13 Sep 2018 05:56:49 -0400 Subject: [petsc-users] Question about HDF5 viewer groups In-Reply-To: References: Message-ID: On Wed, Sep 12, 2018 at 9:56 PM Ellen M. Price wrote: > Hi there, > > I'm running a complex PETSc program that outputs a LOT of data to an > HDF5 file. As such, I'd like to separate my output into groups to make > it easier to traverse in the analysis phase. > > I'm either confused about the nomenclature or the usage of > PetscViewerHDF5PushGroup/PetscViewerHDF5PopGroup. From a quick skim of > the source, it looks like the PushGroup function builds up a linked list > of group names? Also, just the name suggests that there should be a > "stack" of group names stored somewhere. > > So I would expect, after calling PetscViewerHDF5PushGroup twice, say, > > PetscViewerHDF5PushGroup(h5viewer, "group1"); > PetscViewerHDF5PushGroup(h5viewer, "group2"); > VecView(x, h5viewer); > > that I would get my vector output in /group1/group2/x. This is obviously > not what's actually happening. If I run > > http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/examples/tutorials/ex19.c > , > for example, I get, from h5dump: > > HDF5 "ex19.h5" { > FILE_CONTENTS { > group / > dataset /TestVec > group /testBlockSize > dataset /testBlockSize/TestVec2 > group /testTimestep > dataset /testTimestep/TestVec2 > } > } > > even though multiple groups are "pushed" in sequence. > > My question is, have I misunderstood the connotation of pushing/popping > groups? How can I properly create a group and then a subgroup? Every > attempt at doing this: > > PetscViewerHDF5PushGroup(h5viewer, "/group1"); > PetscViewerHDF5PushGroup(h5viewer, "/group1/group2"); > This form works. Here is an example of me doing it: https://bitbucket.org/petsc/petsc/src/624d5184b94bda975877076c18f8637740f06a35/src/dm/impls/plex/plexhdf5.c#lines-430 Note that PushGroup just points us somewhere. If you want that 'directory' created, you have to do something like https://bitbucket.org/petsc/petsc/src/624d5184b94bda975877076c18f8637740f06a35/src/dm/impls/plex/plexhdf5.c#lines-380 HDF5 is terrible, its just that everything else is worse. Thanks, Matt > VecView(x, h5viewer); > > leads to a long string of HDF5 library errors, like: > > HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 0: > #000: ../../../src/H5L.c line 824 in H5Lexists(): unable to get link info > major: Symbol table > minor: Object not found > #001: ../../../src/H5L.c line 2765 in H5L_exists(): path doesn't exist > major: Symbol table > minor: Object already exists > #002: ../../../src/H5Gtraverse.c line 861 in H5G_traverse(): internal > path traversal failed > major: Symbol table > minor: Object not found > #003: ../../../src/H5Gtraverse.c line 755 in H5G_traverse_real(): > component not found > major: Symbol table > minor: Object not found > HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 0: > #000: ../../../src/H5G.c line 314 in H5Gcreate2(): unable to create group > major: Symbol table > minor: Unable to initialize object > #001: ../../../src/H5Gint.c line 194 in H5G__create_named(): unable to > create and link to group > major: Symbol table > minor: Unable to initialize object > #002: ../../../src/H5L.c line 1638 in H5L_link_object(): unable to > create new link to object > major: Links > minor: Unable to initialize object > #003: ../../../src/H5L.c line 1882 in H5L_create_real(): can't insert > link > major: Symbol table > minor: Unable to insert object > #004: ../../../src/H5Gtraverse.c line 861 in H5G_traverse(): internal > path traversal failed > major: Symbol table > minor: Object not found > #005: ../../../src/H5Gtraverse.c line 755 in H5G_traverse_real(): > component not found > major: Symbol table > minor: Object not found > > Thanks in advance, > Ellen Price > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourdin at lsu.edu Thu Sep 13 08:34:48 2018 From: bourdin at lsu.edu (Blaise A Bourdin) Date: Thu, 13 Sep 2018 13:34:48 +0000 Subject: [petsc-users] SNES with constraint diverges In-Reply-To: References: <0B2B682E-2942-444B-B3FC-2EC695577690@anl.gov> <29A829ED-8055-4C68-A751-25C8F27F6A46@mcs.anl.gov> <75D93FDF-8086-416B-A03A-46F372DA718C@mcs.anl.gov> Message-ID: <8934DCB7-CBD1-4A9A-B44D-F1986ABF0B9C@lsu.edu> On Sep 12, 2018, at 4:23 PM, Matthew Knepley > wrote: On Wed, Sep 12, 2018 at 5:16 PM Smith, Barry F. > wrote: The function norm given by the -snes_monitor is the function norm for the DVI problem (which depends on where the solution is constrained) while FormFunction() norm you compute is for the original unconstrained problem. I am still not sure why you solve this as a DVI. If you solve it without the constraints do you get the "wrong" answer? If this is a variational crack problem, you can solve it unconstrained if you use a quadratic penalty term. However, that induces spurious long range communication between crack tips. If you bound the phase field between 0 and 1, you can use a linear penalty which has physical behavior. Not really, you still need the growth constraint from iteration to iteration or you could get healing in unloading cycles, or cack path that change over time. A classical and easy to reproduce examples is to look at a crack tip with two small inclusion at different directions and different orientation relative to the tip. As the loading increase, you could have a crack that goes from tip to inclusion 1, then later from tip to inclusion 2 without going through inclusion 1... @Josh: are you alternating solves in displacement and phase field or trying to implement a monolithic approach? Convergence of monolithic approaches is elusive, and they do not necessarily give you physical path (that is paths that can be interpolated along a path of monotonically decreasing energy) With a staggered (alternate directions) approach, there is no excuse for not using SNESVI. As you mentioned, since you are propagating a crack tip, most of the constraints are saturated and the active set changes very little from step to step. In my experience, SNESVI is very fast in this situation (2-3 iterations), and its cost still negligible compared to the cost of the computation of the equilibrium displacement. In addition, once you can handle growth through SNESVI, you have no excuse not to use a linear damage dissipation potential, which is physically much reasonable (finite support often damage localisation, existence of an elastic limit, and correct nucleation in a wide range of situation as shown in my recent paper (Tann? et al JMPS 2018) My implementation of phase field fracture is available at https://bitbucket.org/bourdin/mef90-sieve It is still based on an old petsc because I can never find the time to catch up with the release, and because I needed some form of I/O in ?natural ordering? that only came in recently. Feel free to ping me directly if you need more details Blaise Bourdin Matt Barry > On Sep 12, 2018, at 3:32 PM, Josh L > wrote: > > Hi, > > my solution is mostly very close to 1. only for a very small area where solution goes from 0 to 1(a smeared crack). > > I set -snes_atol 1e-7 and it is converging. > > I've noticed the following: > > There is a difference between the function norm. > > I calculate the function norm in FormFunction, so every time it is called it gives the function norm > , and the result is different from the function norm given by -snes_monitor if i set > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > SNESVISetVariableBounds(snes,xl,xu,ierr) > > The function norm calculated in FromFunction is NOT reducing, however, the function norm given by -snes_monitor is reducing > They are the same if I just use regular SNES without setting variable bounds. > > > Thanks, > Josh > > 2018-09-12 12:02 GMT-05:00 Smith, Barry F. >: > > You have too tight a convergence tolerance for your problem. You can't expect to get more than 1.e-12 as the minimum residual norm or even less. > > How close is your solution to 1 and -1? > > If you really need much higher convergence you can try ./configure --with-precision=__float128 > > > Barry > > > On Sep 11, 2018, at 11:53 PM, Josh L > wrote: > > > > Yes, I initialize all u_i to 1.0 > > > > > > > > 2018-09-11 23:37 GMT-05:00 Smith, Barry F. >: > > > > Do you start with initial conditions of 0 <= u_i <= 1 ? > > > > Run with -snes_monitor -snes_converged_reason -ksp_monitor_true_residual -info -snes_linesearch_monitor and send all the output > > > > Barry > > > > > > > On Sep 11, 2018, at 11:33 PM, Josh L > wrote: > > > > > > Hi, > > > > > > I am using SNES to solve an nonlinear equation f(u), and I know all the u_i should be 0 and 1. > > > > > > First, I use SNES without constraint, and it converges. > > > > > > But, If I set > > > SNESSetType(snes,SNESVINEWTONRSLS,ierr) > > > SNESVISetVariableBounds(snes,xl,xu,ierr) > > > > > > where xl and xu is vector, and xl_i=0 and xu_i=1 > > > > > > then SNES fails to converge, because linesearch fails(snes reason = -6), and the norm of residual is not reducing(the norm of incremental solution is reducing) > > > > > > The reason to add constraint is that I want to implement some irreversibility. > > > > > > > > > Thanks, > > > Josh > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- Department of Mathematics and Center for Computation & Technology Louisiana State University, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Thu Sep 13 10:24:51 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Thu, 13 Sep 2018 09:24:51 -0600 Subject: [petsc-users] Error 404 for mhypre.c.html Message-ID: http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/aij/hypre/mhypre.c.html#MatHYPRESetPreallocation Fande -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Sep 13 10:36:01 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 13 Sep 2018 10:36:01 -0500 Subject: [petsc-users] Error 404 for mhypre.c.html In-Reply-To: References: Message-ID: balay at asterix /home/balay/petsc (maint=) $ git ls-files |grep mhypre.c src/mat/impls/hypre/mhypre.c So URL should be: http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/hypre/mhypre.c.html#MatHYPRESetPreallocation Satish On Thu, 13 Sep 2018, Fande Kong wrote: > http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/aij/hypre/mhypre.c.html#MatHYPRESetPreallocation > > > Fande > From fdkong.jd at gmail.com Thu Sep 13 10:43:24 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Thu, 13 Sep 2018 09:43:24 -0600 Subject: [petsc-users] Error 404 for mhypre.c.html In-Reply-To: References: Message-ID: Please look at the page http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatHYPRESetPreallocation.html and then click on src/mat/impls/aij/hypre/mhypre.c (right after ``Location") Fande, On Thu, Sep 13, 2018 at 9:37 AM Satish Balay wrote: > balay at asterix /home/balay/petsc (maint=) > $ git ls-files |grep mhypre.c > src/mat/impls/hypre/mhypre.c > > So URL should be: > > > http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/hypre/mhypre.c.html#MatHYPRESetPreallocation > > Satish > > On Thu, 13 Sep 2018, Fande Kong wrote: > > > > http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/aij/hypre/mhypre.c.html#MatHYPRESetPreallocation > > > > > > Fande > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Sep 13 11:17:11 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 13 Sep 2018 11:17:11 -0500 Subject: [petsc-users] Error 404 for mhypre.c.html In-Reply-To: References: Message-ID: Thanks! Fixed in sources. The website will be updated with petsc-3.10.1 Satish On Thu, 13 Sep 2018, Fande Kong wrote: > Please look at the page > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatHYPRESetPreallocation.html > > > and then click on > > src/mat/impls/aij/hypre/mhypre.c > > (right after ``Location") > > Fande, > > On Thu, Sep 13, 2018 at 9:37 AM Satish Balay wrote: > > > balay at asterix /home/balay/petsc (maint=) > > $ git ls-files |grep mhypre.c > > src/mat/impls/hypre/mhypre.c > > > > So URL should be: > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/hypre/mhypre.c.html#MatHYPRESetPreallocation > > > > Satish > > > > On Thu, 13 Sep 2018, Fande Kong wrote: > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/aij/hypre/mhypre.c.html#MatHYPRESetPreallocation > > > > > > > > > Fande > > > > > > > > From iancclin at umich.edu Thu Sep 13 12:46:46 2018 From: iancclin at umich.edu (Ian Lin) Date: Thu, 13 Sep 2018 13:46:46 -0400 Subject: [petsc-users] Function of MatMPIAIJGetSeqAIJ Message-ID: <9BB5BED8-3743-4EAD-9673-7423E0EFBDFD@umich.edu> Hi, I am thinking of using MatMPIAIJGetSeqAIJ to have a reference to the local SeqAIJ Matrix (without actually creating a new matrix). Since I don?t see any description of that function on petsc website, I want to confirm if this functions works similar to MatDenseGetLocalMatrix. Many thanks, Ian From bsmith at mcs.anl.gov Thu Sep 13 12:51:17 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 13 Sep 2018 17:51:17 +0000 Subject: [petsc-users] Function of MatMPIAIJGetSeqAIJ In-Reply-To: <9BB5BED8-3743-4EAD-9673-7423E0EFBDFD@umich.edu> References: <9BB5BED8-3743-4EAD-9673-7423E0EFBDFD@umich.edu> Message-ID: > On Sep 13, 2018, at 12:46 PM, Ian Lin wrote: > > Hi, > > I am thinking of using MatMPIAIJGetSeqAIJ to have a reference to the local SeqAIJ Matrix (without actually creating a new matrix). Since I don?t see any description of that function on petsc website, I want to confirm if this functions works similar to MatDenseGetLocalMatrix. It returns the two SeqAIJ matrices that make up the on-process part of the the parallel MPIAIJ matrix so in that sense is like MatDenseGetLocalMatrix. See MatCreateMPIAIJ for the exact meaning of these two matrices; the final argument maps from the columns of the "off-diagonal" matrix Ao to the global numbering of the original MPIAIJ matrix. Barry > > Many thanks, > Ian From edoardo.alinovi at gmail.com Thu Sep 13 12:58:14 2018 From: edoardo.alinovi at gmail.com (Edoardo alinovi) Date: Thu, 13 Sep 2018 19:58:14 +0200 Subject: [petsc-users] KSPsolve performance tuning Message-ID: Hello PETSc's frieds, It is a couple of weeks that I am trying to enhance the perforamance of my code. Actually I am solving NS equation for a 3D problem of 220k cells with 4procs on my laptop (i7-7800k @ 2.3Ghz with dynamic overclocking). I have installed petsc under linux suse 15 in a virtual machine (I do not know if this is important or not). After some profiling, I can see that the bottle neck is inside KSPSolve while solving pressure equation (solved with cg + hypre pc). For this reason my code is running twice time slower than openFOAM and the gap is only due the solution of pressure. Have you got some hints for me? At this point I am sure I am doing something wrong! I have attached the log of a test simulation. Thank you very much! ------ Edoardo Alinovi, Ph.D. DICCA, Scuola Politecnica Universita' di Genova 1, via Montallegro 16145 Genova, Italy email: edoardo.alinovi at dicca.unige.it Tel: +39 010 353 2540 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 5.6276e+00 100.0% 1.4354e+08 100.0% 3.200e+02 100.0% 6.092e+03 100.0% 9.100e+01 91.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSidedF 23 1.0 5.5043e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatMult 30 1.0 3.8755e-02 1.1 1.38e+07 1.0 3.0e+02 6.4e+03 0.0e+00 1 38 94 98 0 1 38 94 98 0 1415 MatSolve 12 1.0 1.8769e-02 1.5 5.45e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 15 0 0 0 0 15 0 0 0 1158 MatLUFactorNum 3 1.0 2.0071e-02 1.4 2.17e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 6 0 0 0 0 6 0 0 0 430 MatILUFactorSym 1 1.0 2.6951e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatConvert 2 1.0 7.3381e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 8 0 0 0 0 9 0 MatAssemblyBegin 10 1.0 5.3159e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatAssemblyEnd 10 1.0 1.7582e-02 1.1 0.00e+00 0.0 2.0e+01 1.6e+03 1.6e+01 0 0 6 2 16 0 0 6 2 18 0 MatGetRowIJ 5 1.0 1.8690e-06 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 3.3339e-04 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 8 1.0 3.5194e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecDot 6 1.0 1.1794e-03 1.7 6.22e+05 1.0 0.0e+00 0.0e+00 6.0e+00 0 2 0 0 6 0 2 0 0 7 2105 VecDotNorm2 3 1.0 3.6940e-04 1.5 6.22e+05 1.0 0.0e+00 0.0e+00 3.0e+00 0 2 0 0 3 0 2 0 0 3 6720 VecTDot 32 1.0 2.6084e-03 1.1 3.32e+06 1.0 0.0e+00 0.0e+00 3.2e+01 0 9 0 0 32 0 9 0 0 35 5075 VecNorm 29 1.0 1.9163e-02 1.4 3.01e+06 1.0 0.0e+00 0.0e+00 2.9e+01 0 8 0 0 29 0 8 0 0 32 626 VecCopy 6 1.0 5.6139e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 48 1.0 9.9189e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 35 1.0 3.0097e-03 1.1 3.63e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 10 0 0 0 0 10 0 0 0 4811 VecAYPX 16 1.0 1.2960e-03 1.1 1.56e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 4788 VecAXPBYCZ 6 1.0 7.7555e-04 1.0 1.24e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 6401 VecWAXPY 6 1.0 6.5693e-04 1.1 6.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 3778 VecAssemblyBegin 13 1.0 2.1899e-03 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 13 1.0 2.2537e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 30 1.0 7.9062e-04 1.3 0.00e+00 0.0 3.0e+02 6.4e+03 0.0e+00 0 0 94 98 0 0 0 94 98 0 0 VecScatterEnd 30 1.0 1.8332e-02 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 8 1.0 4.3321e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 5 1.0 3.3876e+00 1.0 3.46e+07 1.0 2.7e+02 6.4e+03 7.8e+01 60 96 84 89 78 60 96 84 89 86 41 PCSetUp 8 1.0 1.9194e+00 1.0 2.17e+06 1.0 0.0e+00 0.0e+00 8.0e+00 34 6 0 0 8 34 6 0 0 9 5 PCSetUpOnBlocks 3 1.0 2.3208e-02 1.4 2.17e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 6 0 0 0 0 6 0 0 0 372 PCApply 32 1.0 1.4126e+00 1.0 5.45e+06 1.0 0.0e+00 0.0e+00 0.0e+00 25 15 0 0 0 25 15 0 0 0 15 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 6 6 14642324 0. Vector 31 23 6261400 0. Krylov Solver 5 3 3816 0. Preconditioner 5 3 3424 0. Index Set 5 5 632560 0. Vec Scatter 1 1 1272 0. Viewer 1 0 0 0. ======================================================================================================================== Average time to get PetscTime(): 2.87e-08 Average time for MPI_Barrier(): 9.5102e-06 Average time for zero size MPI_Send(): 1.31203e-05 #PETSc Option Table entries: -log_view #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 --prefix=/home/edo/software/petsc-3.9.3/ --with-debugging=no --with-mpi-dir=/home/edo/software/openMPI-3.1.1/ --download-fblaslapack=1 --download-superlu_dist --download-mumps --download-hypre --download-metis --download-parmetis --download-scalapack ----------------------------------------- Libraries compiled on 2018-08-22 10:01:51 on linux-sypg Machine characteristics: Linux-4.12.14-lp150.11-default-x86_64-with-glibc2.2.5 Using PETSc directory: /home/edo/software/petsc-3.9.3 Using PETSc arch: ----------------------------------------- Using C compiler: /home/edo/software/openMPI-3.1.1/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 Using Fortran compiler: /home/edo/software/openMPI-3.1.1/bin/mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 ----------------------------------------- Using include paths: -I/home/edo/software/petsc-3.9.3/include -I/home/edo/software/petsc-3.9.3//include -I/home/edo/software/petsc-3.9.3-3.9.3/include -I/home/edo/software/openMPI-3.1.1/include ----------------------------------------- Using C linker: /home/edo/software/openMPI-3.1.1/bin/mpicc Using Fortran linker: /home/edo/software/openMPI-3.1.1/bin/mpif90 Using libraries: -Wl,-rpath,/home/edo/software/petsc-3.9.3/lib -L/home/edo/software/petsc-3.9.3/lib -lpetsc -Wl,-rpath,/home/edo/software/petsc-3.9.3/lib -L/home/edo/software/petsc-3.9.3/lib -Wl,-rpath,/home/edo/software/openMPI-3.1.1/lib -L/home/edo/software/openMPI-3.1.1/lib -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/7 -L/usr/lib64/gcc/x86_64-suse-linux/7 -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_dist -lHYPRE -lflapack -lfblas -lparmetis -lmetis -lm -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ -ldl ----------------------------------------- edo at linux-sypg:~/FLUBIO_INC/exec> From bsmith at mcs.anl.gov Thu Sep 13 13:10:13 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 13 Sep 2018 18:10:13 +0000 Subject: [petsc-users] KSPsolve performance tuning In-Reply-To: References: Message-ID: What pressure solver is OpenFOAM using? Are you using the same convergence tolerance for the pressure solver for the two approaches? Have you tried PETSc with a simpler solver: -pc_type bjacobi or -pc_type asm ; the problem is pretty small and maybe at this size hypre is overkill? Barry > On Sep 13, 2018, at 12:58 PM, Edoardo alinovi wrote: > > Hello PETSc's frieds, > > It is a couple of weeks that I am trying to enhance the perforamance of my code. Actually I am solving NS equation for a 3D problem of 220k cells with 4procs on my laptop (i7-7800k @ 2.3Ghz with dynamic overclocking). I have installed petsc under linux suse 15 in a virtual machine (I do not know if this is important or not). > > After some profiling, I can see that the bottle neck is inside KSPSolve while solving pressure equation (solved with cg + hypre pc). For this reason my code is running twice time slower than openFOAM and the gap is only due the solution of pressure. Have you got some hints for me? At this point I am sure I am doing something wrong! I have attached the log of a test simulation. > > Thank you very much! > > ------ > > Edoardo Alinovi, Ph.D. > > DICCA, Scuola Politecnica > Universita' di Genova > 1, via Montallegro > 16145 Genova, Italy > > email: edoardo.alinovi at dicca.unige.it > Tel: +39 010 353 2540 > > > From mvalera-w at sdsu.edu Thu Sep 13 13:33:15 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Thu, 13 Sep 2018 11:33:15 -0700 Subject: [petsc-users] Requesting multi-GPUs In-Reply-To: References: Message-ID: So, from what i get here the round robin assignation to each GPU device is done automatically by PETSc, from mapping the system? or do i have to pass a command line argument to do that? Thanks, On Wed, Sep 12, 2018 at 2:38 PM, Matthew Knepley wrote: > On Wed, Sep 12, 2018 at 5:31 PM Manuel Valera wrote: > >> Ok then, how can i try getting more than one GPU with the same number of >> MPI processes? >> > > I do not believe we handle more than one GPU per MPI process. Is that what > you are asking? > > Thanks, > > Matt > > >> Thanks, >> >> On Wed, Sep 12, 2018 at 2:20 PM, Matthew Knepley >> wrote: >> >>> On Wed, Sep 12, 2018 at 5:13 PM Manuel Valera >>> wrote: >>> >>>> Hello guys, >>>> >>>> I am working in a multi-gpu cluster and i want to request 2 or more >>>> GPUs, how can i do that from PETSc? evidently mpirun -n # is for requesting >>>> processors, but what if i want to use one mpi processor but several GPUs >>>> instead? >>>> >>> >>> We do not do that. You would run the same number of MPI processes as >>> GPUs. Note that >>> you can have more than 1 MPI process on a processor. >>> >>> Matt >>> >>> >>>> Also, i understand the GPU handles the linear system solver, but what >>>> about the data management? can i do DMs for other than the linear solver >>>> using the GPUs >>>> ? >>>> >>>> Thanks once more, >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 13 14:01:07 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 13 Sep 2018 15:01:07 -0400 Subject: [petsc-users] Requesting multi-GPUs In-Reply-To: References: Message-ID: On Thu, Sep 13, 2018 at 2:33 PM Manuel Valera wrote: > So, from what i get here the round robin assignation to each GPU device is > done automatically by PETSc, from mapping the system? or do i have to pass > a command line argument to do that? > It is automatic. Thanks, Matt > Thanks, > > > On Wed, Sep 12, 2018 at 2:38 PM, Matthew Knepley > wrote: > >> On Wed, Sep 12, 2018 at 5:31 PM Manuel Valera wrote: >> >>> Ok then, how can i try getting more than one GPU with the same number of >>> MPI processes? >>> >> >> I do not believe we handle more than one GPU per MPI process. Is that >> what you are asking? >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> On Wed, Sep 12, 2018 at 2:20 PM, Matthew Knepley >>> wrote: >>> >>>> On Wed, Sep 12, 2018 at 5:13 PM Manuel Valera >>>> wrote: >>>> >>>>> Hello guys, >>>>> >>>>> I am working in a multi-gpu cluster and i want to request 2 or more >>>>> GPUs, how can i do that from PETSc? evidently mpirun -n # is for requesting >>>>> processors, but what if i want to use one mpi processor but several GPUs >>>>> instead? >>>>> >>>> >>>> We do not do that. You would run the same number of MPI processes as >>>> GPUs. Note that >>>> you can have more than 1 MPI process on a processor. >>>> >>>> Matt >>>> >>>> >>>>> Also, i understand the GPU handles the linear system solver, but what >>>>> about the data management? can i do DMs for other than the linear solver >>>>> using the GPUs >>>>> ? >>>>> >>>>> Thanks once more, >>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Sep 13 14:07:06 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 13 Sep 2018 19:07:06 +0000 Subject: [petsc-users] KSPsolve performance tuning In-Reply-To: References: Message-ID: <37A4F874-4BE1-4F14-9497-7EFFBD829716@mcs.anl.gov> > On Sep 13, 2018, at 1:55 PM, Edoardo alinovi wrote: > > Hello Barry, > > Thank you very much for your replay! Maybe the best feature of PETSc is this mailing list :) You are right cg + block Jcoby is twice faster than hypre, but still a bit slower than OF. I have definitely to test a bigger case. It seems that hypre loses a lot of time in setting up the PC. Is this normal? Yes, BoomerAMG (or any AMG method) has "large" setup times that don't scale as well as the solve time. Thus if you can reuse the setup for multiple solves you gain a great deal. The best case is, of course, where the same Poisson problem (with a different right hand side obviously) has to be solved many times. I noticed in your log_view that it seems to need to do a new setup for each solve. Barry > > Thank you! > > > > ------ > > Edoardo Alinovi, Ph.D. > > DICCA, Scuola Politecnica > Universita' di Genova > 1, via Montallegro > 16145 Genova, Italy > > email: edoardo.alinovi at dicca.unige.it > Tel: +39 010 353 2540 > > > > 2018-09-13 20:10 GMT+02:00 Smith, Barry F. : > > What pressure solver is OpenFOAM using? Are you using the same convergence tolerance for the pressure solver for the two approaches? Have you tried PETSc with a simpler solver: -pc_type bjacobi or -pc_type asm ; the problem is pretty small and maybe at this size hypre is overkill? > > Barry > > > > On Sep 13, 2018, at 12:58 PM, Edoardo alinovi wrote: > > > > Hello PETSc's frieds, > > > > It is a couple of weeks that I am trying to enhance the perforamance of my code. Actually I am solving NS equation for a 3D problem of 220k cells with 4procs on my laptop (i7-7800k @ 2.3Ghz with dynamic overclocking). I have installed petsc under linux suse 15 in a virtual machine (I do not know if this is important or not). > > > > After some profiling, I can see that the bottle neck is inside KSPSolve while solving pressure equation (solved with cg + hypre pc). For this reason my code is running twice time slower than openFOAM and the gap is only due the solution of pressure. Have you got some hints for me? At this point I am sure I am doing something wrong! I have attached the log of a test simulation. > > > > Thank you very much! > > > > ------ > > > > Edoardo Alinovi, Ph.D. > > > > DICCA, Scuola Politecnica > > Universita' di Genova > > 1, via Montallegro > > 16145 Genova, Italy > > > > email: edoardo.alinovi at dicca.unige.it > > Tel: +39 010 353 2540 > > > > > > > > From bsmith at mcs.anl.gov Thu Sep 13 14:19:45 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 13 Sep 2018 19:19:45 +0000 Subject: [petsc-users] KSPsolve performance tuning In-Reply-To: References: <37A4F874-4BE1-4F14-9497-7EFFBD829716@mcs.anl.gov> Message-ID: > On Sep 13, 2018, at 2:16 PM, Edoardo alinovi wrote: > > Yes, I need it or at least I think so. I am using piso alghorithm which requires to solve the pressure eq twice in a time step. Since the coeffs of the matrix are different in the two cases, then I have to setup the pc every time. What about at the next time-step? Are the coefficients different yet again at each new timestep? Barry > > Thanks Barry! > > Il gio 13 set 2018, 21:08 Smith, Barry F. ha scritto: > > > > On Sep 13, 2018, at 1:55 PM, Edoardo alinovi wrote: > > > > Hello Barry, > > > > Thank you very much for your replay! Maybe the best feature of PETSc is this mailing list :) You are right cg + block Jcoby is twice faster than hypre, but still a bit slower than OF. I have definitely to test a bigger case. It seems that hypre loses a lot of time in setting up the PC. Is this normal? > > Yes, BoomerAMG (or any AMG method) has "large" setup times that don't scale as well as the solve time. Thus if you can reuse the setup for multiple solves you gain a great deal. The best case is, of course, where the same Poisson problem (with a different right hand side obviously) has to be solved many times. I noticed in your log_view that it seems to need to do a new setup for each solve. > > Barry > > > > > Thank you! > > > > > > > > ------ > > > > Edoardo Alinovi, Ph.D. > > > > DICCA, Scuola Politecnica > > Universita' di Genova > > 1, via Montallegro > > 16145 Genova, Italy > > > > email: edoardo.alinovi at dicca.unige.it > > Tel: +39 010 353 2540 > > > > > > > > 2018-09-13 20:10 GMT+02:00 Smith, Barry F. : > > > > What pressure solver is OpenFOAM using? Are you using the same convergence tolerance for the pressure solver for the two approaches? Have you tried PETSc with a simpler solver: -pc_type bjacobi or -pc_type asm ; the problem is pretty small and maybe at this size hypre is overkill? > > > > Barry > > > > > > > On Sep 13, 2018, at 12:58 PM, Edoardo alinovi wrote: > > > > > > Hello PETSc's frieds, > > > > > > It is a couple of weeks that I am trying to enhance the perforamance of my code. Actually I am solving NS equation for a 3D problem of 220k cells with 4procs on my laptop (i7-7800k @ 2.3Ghz with dynamic overclocking). I have installed petsc under linux suse 15 in a virtual machine (I do not know if this is important or not). > > > > > > After some profiling, I can see that the bottle neck is inside KSPSolve while solving pressure equation (solved with cg + hypre pc). For this reason my code is running twice time slower than openFOAM and the gap is only due the solution of pressure. Have you got some hints for me? At this point I am sure I am doing something wrong! I have attached the log of a test simulation. > > > > > > Thank you very much! > > > > > > ------ > > > > > > Edoardo Alinovi, Ph.D. > > > > > > DICCA, Scuola Politecnica > > > Universita' di Genova > > > 1, via Montallegro > > > 16145 Genova, Italy > > > > > > email: edoardo.alinovi at dicca.unige.it > > > Tel: +39 010 353 2540 > > > > > > > > > > > > > > From rupp at iue.tuwien.ac.at Thu Sep 13 23:52:01 2018 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Fri, 14 Sep 2018 06:52:01 +0200 Subject: [petsc-users] [SPAM?] Re: Requesting multi-GPUs In-Reply-To: References: Message-ID: <80b023d1-9612-2050-3f4a-91d6a55d0faa@iue.tuwien.ac.at> Hi Manuel, we do not handle more than one GPU per MPI process. If we did, we would have to duplicate all the existing MPI communication pattern for the multiple GPUs on a single process. This is a significant maintenance burden, yet it would not provide any significant performance benefits. Best regards, Karli On 09/12/2018 11:31 PM, Manuel Valera wrote: > Ok then, how can i try getting more than one GPU with the same number of > MPI processes? > > Thanks, > > On Wed, Sep 12, 2018 at 2:20 PM, Matthew Knepley > wrote: > > On Wed, Sep 12, 2018 at 5:13 PM Manuel Valera > wrote: > > Hello guys, > > I am working in a multi-gpu cluster and i want to request 2 or > more GPUs, how can i do that from PETSc? evidently mpirun -n # > is for requesting processors, but what if i want to use one mpi > processor but several GPUs instead? > > > We do not do that. You would run the same number of MPI processes as > GPUs. Note that > you can have more than 1 MPI process on a processor. > > ? Matt > > Also, i understand the GPU handles the linear system solver, but > what about the data management? can i do DMs for other than the > linear solver using the GPUs > ? > > Thanks once more, > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > From griesser.jan at googlemail.com Fri Sep 14 10:45:42 2018 From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=) Date: Fri, 14 Sep 2018 17:45:42 +0200 Subject: [petsc-users] Retireve eigenvectors from a paralell job/ Spectrum slicing in order to solve big eigenvalue problem Message-ID: Hey there, first i want to say thanks to Satish and Matt for helping with with my last problem with the mpi compilation. I have two questions related to solving a big, hermitian, standard eigenvalue problem using SLEPc4py., compiled with Intel MKL and Intel MPI. - I am using slepc4py with mpi and run it with around -n 20 cores at the moment and how i wanted to ask if there is an easy way to retrieve the eigenvectors? When i run my code and print for i in range(nconv): for i in range(nconv): val = E.getEigenpair(i, vr, vi) Print(vr.getArray()) i get the parts of the eigenvectors according to the partition of the matrix. Is there any easy way to put them together in an array and write them to file ? (I am struggling a little bit with the building them in the correct order) - I need to solve eigenvalue problems up to a dimension of 100000 degrees of freedom and i need all eigenvalues and eigenvectors. I think solving all eigenvalues in one process is far too much and i thought about if it is possible to apply the spectrum slicing described in Chap. 3.4.5. Due to the nature of my problem, i am able to simulate smaller systems of 10000 DOF and extract the biggest eigenvalue, which will be the same for larger systems sizes. Is this in general possible since i have a standard HEP problem or is there a better and faster possibility to do this? Thank you very much! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Fri Sep 14 11:19:34 2018 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 14 Sep 2018 18:19:34 +0200 Subject: [petsc-users] Retireve eigenvectors from a paralell job/ Spectrum slicing in order to solve big eigenvalue problem In-Reply-To: References: Message-ID: <96EE281A-3EC9-4CD7-884B-1AC45D1A6A19@dsic.upv.es> > El 14 sept 2018, a las 17:45, Jan Grie?er escribi?: > > Hey there, > first i want to say thanks to Satish and Matt for helping with with my last problem with the mpi compilation. I have two questions related to solving a big, hermitian, standard eigenvalue problem using SLEPc4py., compiled with Intel MKL and Intel MPI. > - I am using slepc4py with mpi and run it with around -n 20 cores at the moment and how i wanted to ask if there is an easy way to retrieve the eigenvectors? When i run my code and print for i in range(nconv): > for i in range(nconv): > val = E.getEigenpair(i, vr, vi) > Print(vr.getArray()) > i get the parts of the eigenvectors according to the partition of the matrix. Is there any easy way to put them together in an array and write them to file ? (I am struggling a little bit with the building them in the correct order) You need VecScatterCreateToZero. There must be an equivalent in python. > - I need to solve eigenvalue problems up to a dimension of 100000 degrees of freedom and i need all eigenvalues and eigenvectors. I think solving all eigenvalues in one process is far too much and i thought about if it is possible to apply the spectrum slicing described in Chap. 3.4.5. Due to the nature of my problem, i am able to simulate smaller systems of 10000 DOF and extract the biggest eigenvalue, which will be the same for larger systems sizes. Is this in general possible since i have a standard HEP problem or is there a better and faster possibility to do this? In general, SLEPc is not intended for computing the whole spectrum. You can try with spectrum slicing but this will be competitive if computing just a percentage of eigenvalues, 50% say. Jose > > Thank you very much! -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Sep 14 11:34:01 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 14 Sep 2018 12:34:01 -0400 Subject: [petsc-users] Retireve eigenvectors from a paralell job/ Spectrum slicing in order to solve big eigenvalue problem In-Reply-To: <96EE281A-3EC9-4CD7-884B-1AC45D1A6A19@dsic.upv.es> References: <96EE281A-3EC9-4CD7-884B-1AC45D1A6A19@dsic.upv.es> Message-ID: On Fri, Sep 14, 2018 at 12:19 PM Jose E. Roman wrote: > El 14 sept 2018, a las 17:45, Jan Grie?er > escribi?: > > Hey there, > first i want to say thanks to Satish and Matt for helping with with my > last problem with the mpi compilation. I have two questions related to > solving a big, hermitian, standard eigenvalue problem using SLEPc4py., > compiled with Intel MKL and Intel MPI. - I am using slepc4py with mpi and > run it with around -n 20 cores at the moment and how i wanted to ask if > there is an easy way to retrieve the eigenvectors? When i run my code and > print for i in range(nconv): > for i in range(nconv): val = E.getEigenpair(i, vr, vi) Print(vr.getArray > ()) > i get the parts of the eigenvectors according to the partition of the > matrix. Is there any easy way to put them together in an array and write > them to file ? (I am struggling a little bit with the building them in the > correct order) > > > You need VecScatterCreateToZero. There must be an equivalent in python. > An alternative to this which you should consider, because it is simpler, is to write the vector to a file using some format that PETSc understands, Then you just need vr.view(viewer) for a viewer like the binary viewer or some ASCII format you like. Thanks, Matt > - I need to solve eigenvalue problems up to a dimension of 100000 degrees > of freedom and i need all eigenvalues and eigenvectors. I think solving > all eigenvalues in one process is far too much and i thought about if it is > possible to apply the spectrum slicing described in Chap. 3.4.5. Due to the > nature of my problem, i am able to simulate smaller systems of 10000 DOF > and extract the biggest eigenvalue, which will be the same for larger > systems sizes. Is this in general possible since i have a standard HEP > problem or is there a better and faster possibility to do this? > > > In general, SLEPc is not intended for computing the whole spectrum. You > can try with spectrum slicing but this will be competitive if computing > just a percentage of eigenvalues, 50% say. > > Jose > > > Thank you very much! > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From evanum at gmail.com Sat Sep 15 13:19:24 2018 From: evanum at gmail.com (Evan Um) Date: Sat, 15 Sep 2018 11:19:24 -0700 Subject: [petsc-users] Errors during PETSC installation (complex version) Message-ID: Dear PETSC users, My colleague and I found an complex number related error (passing arguments of PetscMax and PetscMin) during PETSC installation. If you have a similar experience, could you suggest a trouble shooting? Thanks, Evan Installation configuration: d=${HOME}/usr/lib/petsc-3.9.3 ./configure PETSC_DIR=$d PETSC_ARCH=linux-intel --prefix=${HOME}/usr/lib/petsc-3.9.3-intel-mumps --with-mpi-dir=/global/software/sl-6.x86_64/modules/ intel/2013_sp1.4.211/openmpi/1.6.5-intel --with-scalar-type=complex --download-fblaslapack=${d}/fblaslapack-3.1.1.tar.gz --download-superlu_dist --download-metis --download-parmetis --download-scotch --download-ptscotch --with-fortran-kernels=yes --with-openmp --download-mumps --with-precision=double --download-superlu --with-cxx-dialect=C++11 --download-scalapack --with-debugging=0 Errors: CC linux-intel/obj/ksp/pc/impls/kaczmarz/kaczmarz.o CC linux-intel/obj/ksp/pc/impls/galerkin/galerkin.o /data1/home/mcommer/usr/lib/petsc-3.9.3/src/ksp/pc/impls/tfs/ivec.c(601): error: complex type is not allowed *arg1 = PetscMax(*arg1,*arg2); ^ /data1/home/mcommer/usr/lib/petsc-3.9.3/src/ksp/pc/impls/tfs/ivec.c(625): error: complex type is not allowed *arg1 = PetscMin(*arg1,*arg2); ^ compilation aborted for /data1/home/mcommer/usr/lib/ petsc-3.9.3/src/ksp/pc/impls/tfs/ivec.c (code 2) gmake[2]: *** [linux-intel/obj/ksp/pc/impls/tfs/ivec.o] Error 2 gmake[2]: *** Waiting for unfinished jobs.... gmake[2]: Leaving directory `/data1/home/mcommer/usr/lib/petsc-3.9.3' gmake[1]: *** [gnumake] Error 2 gmake[1]: Leaving directory `/data1/home/mcommer/usr/lib/petsc-3.9.3' **************************ERROR************************************* Error during compile, check linux-intel/lib/petsc/conf/make.log Send it and linux-intel/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov ******************************************************************** make: *** [all] Error 1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sat Sep 15 13:54:44 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 15 Sep 2018 13:54:44 -0500 Subject: [petsc-users] Errors during PETSC installation (complex version) In-Reply-To: References: Message-ID: hm - this sourcefile should not get built when configured with --with-scalar-type=complex Suggest starting a fresh build and see if it works. Also make sure there is no prior install of petsc in the specified --prefix location. Also - the latest patched version in petsc-3.9 is petsc-3.9.4 - and the latest version is petsc-3.10.0 - suggest using the later versions. [when doing the rebuild] Satish On Sat, 15 Sep 2018, Evan Um wrote: > Dear PETSC users, > > My colleague and I found an complex number related error (passing arguments > of PetscMax and PetscMin) during PETSC installation. If you have a similar > experience, could you suggest a trouble shooting? > > Thanks, > Evan > > > Installation configuration: > > d=${HOME}/usr/lib/petsc-3.9.3 > ./configure PETSC_DIR=$d PETSC_ARCH=linux-intel > --prefix=${HOME}/usr/lib/petsc-3.9.3-intel-mumps > --with-mpi-dir=/global/software/sl-6.x86_64/modules/ > intel/2013_sp1.4.211/openmpi/1.6.5-intel --with-scalar-type=complex > --download-fblaslapack=${d}/fblaslapack-3.1.1.tar.gz > --download-superlu_dist --download-metis --download-parmetis > --download-scotch --download-ptscotch --with-fortran-kernels=yes > --with-openmp --download-mumps --with-precision=double --download-superlu > --with-cxx-dialect=C++11 --download-scalapack --with-debugging=0 > > Errors: > > CC linux-intel/obj/ksp/pc/impls/kaczmarz/kaczmarz.o > CC linux-intel/obj/ksp/pc/impls/galerkin/galerkin.o > /data1/home/mcommer/usr/lib/petsc-3.9.3/src/ksp/pc/impls/tfs/ivec.c(601): > error: complex type is not allowed > *arg1 = PetscMax(*arg1,*arg2); > ^ > > /data1/home/mcommer/usr/lib/petsc-3.9.3/src/ksp/pc/impls/tfs/ivec.c(625): > error: complex type is not allowed > *arg1 = PetscMin(*arg1,*arg2); > ^ > > compilation aborted for /data1/home/mcommer/usr/lib/ > petsc-3.9.3/src/ksp/pc/impls/tfs/ivec.c (code 2) > gmake[2]: *** [linux-intel/obj/ksp/pc/impls/tfs/ivec.o] Error 2 > gmake[2]: *** Waiting for unfinished jobs.... > gmake[2]: Leaving directory `/data1/home/mcommer/usr/lib/petsc-3.9.3' > gmake[1]: *** [gnumake] Error 2 > gmake[1]: Leaving directory `/data1/home/mcommer/usr/lib/petsc-3.9.3' > **************************ERROR************************************* > Error during compile, check linux-intel/lib/petsc/conf/make.log > Send it and linux-intel/lib/petsc/conf/configure.log to > petsc-maint at mcs.anl.gov > ******************************************************************** > make: *** [all] Error 1 > From yjwu16 at gmail.com Mon Sep 17 03:22:13 2018 From: yjwu16 at gmail.com (Yingjie Wu) Date: Mon, 17 Sep 2018 16:22:13 +0800 Subject: [petsc-users] Problems ablout setting up variables and SNES Message-ID: Dear Petsc developer: Hi, Thank you very much for your previous reply, recently I encountered some problems in the implementation of the program. 1. Previously I used DMDA to create distributed variables (Variables are different in each grids). Now I want to add a single variable (independent of the grid). How do I add it? How do I write programs with distributed variables and single variables exist at the same time? I've created three "Field" variables before, and I'd like to add a single variable. How should I describe it in the program? The codes are as below. ierr=DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,10,10,PETSC_DECIDE,PETSC_DECIDE,3,1,0,0,&da);CHKERRQ(ierr); ierr = DMSetFromOptions(da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = SNESSetDM(snes,(DM)da);CHKERRQ(ierr); ierr = DMDASetFieldName(da,0,"Pressure");CHKERRQ(ierr); ierr = DMDASetFieldName(da,1,"Y-Velocity");CHKERRQ(ierr); ierr = DMDASetFieldName(da,2,"Fluid_temperature");CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&x);CHKERRQ(ierr); 2. At present, I am solving nonlinear equations. Now I need to do one-step nonlinear calculation first, and then treat the results as coefficients for the subsequent nonlinear calculation. Now I used to use SNESSolve() directly to solve the problem. How can we calculate step by step and process the result before entering the next nonlinear step? Since I'm a beginner in PETSc, I would be grateful if there were any examples. I'm looking forward to your reply~ Thanks, Yingjie -------------- next part -------------- An HTML attachment was scrubbed... URL: From griesser.jan at googlemail.com Mon Sep 17 03:40:38 2018 From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=) Date: Mon, 17 Sep 2018 10:40:38 +0200 Subject: [petsc-users] Retireve eigenvectors from a paralell job/ Spectrum slicing in order to solve big eigenvalue problem In-Reply-To: References: <96EE281A-3EC9-4CD7-884B-1AC45D1A6A19@dsic.upv.es> Message-ID: I am aware that SLEPc is not supposed to calculate all eigenvalues and eigenvectors, my problem is simply that i want for a physical large enough system all of them before i can make the transition to go to the smallest ones. Competitiveness is of secondary importance at the moment. But ihave a problem connected with spectrum slicing. I followed the instructions in the manual of Chap. 3.4.5 Spectrum Slicing and converted them to the python package. But now i get the following error. It appears to me that it is not able to find the ksp object, but i actually do not know why this is the case. aceback (most recent call last): File "Eigensolver_spectrum_slicing.py", line 216, in solve_eigensystem(DynMatrix_nn, Unity_nn, D_nn.shape, opt_dict.LowLimit, opt_dict.UpperLimit) File "Eigensolver_spectrum_slicing.py", line 121, in solve_eigensystem E.setUp() File "SLEPc/EPS.pyx", line 1099, in slepc4py.SLEPc.EPS.setUp petsc4py.PETSc.Error: error code 92 [14] EPSSetUp() line 165 in /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c [14] EPSSetUp_KrylovSchur() line 146 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c [14] EPSSetUp_KrylovSchur_Slice() line 410 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c [14] EPSSliceGetEPS() line 300 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c [14] EPSSetUp() line 165 in /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c [14] EPSSetUp_KrylovSchur() line 146 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c [14] EPSSetUp_KrylovSchur_Slice() line 461 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c [14] EPSSliceGetInertia() line 331 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c [14] STSetUp() line 271 in /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/interface/stsolve.c [14] STSetUp_Sinvert() line 132 in /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/impls/sinvert/sinvert.c [14] KSPSetUp() line 381 in /tmp/pip-install-xmiaat2t/petsc/src/ksp/ksp/interface/itfunc.c [14] PCSetUp() line 923 in /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/interface/precon.c [14] PCSetUp_Cholesky() line 86 in /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c [14] MatGetFactor() line 4318 in /tmp/pip-install-xmiaat2t/petsc/src/mat/interface/matrix.c [14] See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers [14] Could not locate a solver package. Perhaps you must ./configure with --download- The code i used to solve the problem is Am Fr., 14. Sep. 2018 um 18:34 Uhr schrieb Matthew Knepley < knepley at gmail.com>: > On Fri, Sep 14, 2018 at 12:19 PM Jose E. Roman wrote: > >> El 14 sept 2018, a las 17:45, Jan Grie?er >> escribi?: >> >> Hey there, >> first i want to say thanks to Satish and Matt for helping with with my >> last problem with the mpi compilation. I have two questions related to >> solving a big, hermitian, standard eigenvalue problem using SLEPc4py., >> compiled with Intel MKL and Intel MPI. - I am using slepc4py with mpi >> and run it with around -n 20 cores at the moment and how i wanted to ask if >> there is an easy way to retrieve the eigenvectors? When i run my code >> and print for i in range(nconv): >> for i in range(nconv): val = E.getEigenpair(i, vr, vi) Print(vr.getArray >> ()) >> i get the parts of the eigenvectors according to the partition of the >> matrix. Is there any easy way to put them together in an array and write >> them to file ? (I am struggling a little bit with the building them in the >> correct order) >> >> >> You need VecScatterCreateToZero. There must be an equivalent in python. >> > > An alternative to this which you should consider, because it is simpler, > is to write the vector to a file > using some format that PETSc understands, Then you just need > vr.view(viewer) for a viewer like > the binary viewer or some ASCII format you like. > > Thanks, > > Matt > >> - I need to solve eigenvalue problems up to a dimension of 100000 degrees >> of freedom and i need all eigenvalues and eigenvectors. I think solving >> all eigenvalues in one process is far too much and i thought about if it is >> possible to apply the spectrum slicing described in Chap. 3.4.5. Due to the >> nature of my problem, i am able to simulate smaller systems of 10000 DOF >> and extract the biggest eigenvalue, which will be the same for larger >> systems sizes. Is this in general possible since i have a standard HEP >> problem or is there a better and faster possibility to do this? >> >> >> In general, SLEPc is not intended for computing the whole spectrum. You >> can try with spectrum slicing but this will be competitive if computing >> just a percentage of eigenvalues, 50% say. >> >> Jose >> >> >> Thank you very much! >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From griesser.jan at googlemail.com Mon Sep 17 03:41:30 2018 From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=) Date: Mon, 17 Sep 2018 10:41:30 +0200 Subject: [petsc-users] Retireve eigenvectors from a paralell job/ Spectrum slicing in order to solve big eigenvalue problem References: <96EE281A-3EC9-4CD7-884B-1AC45D1A6A19@dsic.upv.es> Message-ID: def solve_eigensystem(DynMatrix_nn, Unity_nn, Dimension, LowerLimit, UpperLimit): # Create the EPS solver E = SLEPc.EPS().create() # Create the preconditioner and set it to Cholesky pc = PETSc.PC().create() pc.setType(pc.Type.CHOLESKY) # Create the KSP object ksp = PETSc.KSP().create() ksp.setType(ksp.Type.PREONLY) ksp.setPC(pc) # Set up the spectral transformations st = SLEPc.ST().create() st.setType("sinvert") st.setKSP(ksp) # Setup spectral transformation E.setST(st) # Eigenvalues should be real, therefore we start to order them from the smallest real value |l.real| E.setWhichEigenpairs(E.Which.ALL) # Set the interval of spectrum slicing E.setInterval(LowerLimit, UpperLimit) # Since the dynamical matrix is symmetric and real it is hermitian. Use GHEP for the spectrum slicing. Operatormatrix B is just a unit matrix E.setProblemType(SLEPc.EPS.ProblemType.GHEP) # Use the Krylov Schur method to solve the eigenvalue problem E.setType(E.Type.KRYLOVSCHUR) # Partition the Krylov schnur problem in npart procceses E.setKrylovSchurPartitions(10) # Set the convergence criterion to relative to the eigenvalue and the maximal number of iterations E.setConvergenceTest(E.Conv.REL) E.setTolerances(tol = 1e-7, max_it = 1000) # Set the matrix in order to solve E.setOperators(DynMatrix_nn, Unity_nn) # Sets EPS options from the options database. E.setFromOptions() # Sets up all the internal data structures necessary for the execution of the eigensolver. E.setUp() # Solve eigenvalue problem startClock = time.clock() startTime = time.time() E.solve() Has maybe one of you any idea why this happens and where the problem is ? Am Mo., 17. Sep. 2018 um 10:40 Uhr schrieb Jan Grie?er < griesser.jan at googlemail.com>: > I am aware that SLEPc is not supposed to calculate all eigenvalues and > eigenvectors, my problem is simply that i want for a physical large enough > system all of them before i can make the transition to go to the smallest > ones. > Competitiveness is of secondary importance at the moment. > But ihave a problem connected with spectrum slicing. I followed the > instructions in the manual of Chap. 3.4.5 Spectrum Slicing and converted > them to the python package. > But now i get the following error. It appears to me that it is not able to > find the ksp object, but i actually do not know why this is the case. > aceback (most recent call last): > File "Eigensolver_spectrum_slicing.py", line 216, in > solve_eigensystem(DynMatrix_nn, Unity_nn, D_nn.shape, > opt_dict.LowLimit, opt_dict.UpperLimit) > File "Eigensolver_spectrum_slicing.py", line 121, in solve_eigensystem > E.setUp() > File "SLEPc/EPS.pyx", line 1099, in slepc4py.SLEPc.EPS.setUp > petsc4py.PETSc.Error: error code 92 > [14] EPSSetUp() line 165 in > /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c > [14] EPSSetUp_KrylovSchur() line 146 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c > [14] EPSSetUp_KrylovSchur_Slice() line 410 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > [14] EPSSliceGetEPS() line 300 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > [14] EPSSetUp() line 165 in > /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c > [14] EPSSetUp_KrylovSchur() line 146 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c > [14] EPSSetUp_KrylovSchur_Slice() line 461 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > [14] EPSSliceGetInertia() line 331 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > [14] STSetUp() line 271 in > /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/interface/stsolve.c > [14] STSetUp_Sinvert() line 132 in > /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/impls/sinvert/sinvert.c > [14] KSPSetUp() line 381 in > /tmp/pip-install-xmiaat2t/petsc/src/ksp/ksp/interface/itfunc.c > [14] PCSetUp() line 923 in > /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/interface/precon.c > [14] PCSetUp_Cholesky() line 86 in > /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c > [14] MatGetFactor() line 4318 in > /tmp/pip-install-xmiaat2t/petsc/src/mat/interface/matrix.c > [14] See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html > for possible LU and Cholesky solvers > [14] Could not locate a solver package. Perhaps you must ./configure with > --download- > > The code i used to solve the problem is > > Am Fr., 14. Sep. 2018 um 18:34 Uhr schrieb Matthew Knepley < > knepley at gmail.com>: > >> On Fri, Sep 14, 2018 at 12:19 PM Jose E. Roman >> wrote: >> >>> El 14 sept 2018, a las 17:45, Jan Grie?er >>> escribi?: >>> >>> Hey there, >>> first i want to say thanks to Satish and Matt for helping with with my >>> last problem with the mpi compilation. I have two questions related to >>> solving a big, hermitian, standard eigenvalue problem using SLEPc4py., >>> compiled with Intel MKL and Intel MPI. - I am using slepc4py with mpi >>> and run it with around -n 20 cores at the moment and how i wanted to ask if >>> there is an easy way to retrieve the eigenvectors? When i run my code >>> and print for i in range(nconv): >>> for i in range(nconv): val = E.getEigenpair(i, vr, vi) Print(vr.getArray >>> ()) >>> i get the parts of the eigenvectors according to the partition of the >>> matrix. Is there any easy way to put them together in an array and write >>> them to file ? (I am struggling a little bit with the building them in the >>> correct order) >>> >>> >>> You need VecScatterCreateToZero. There must be an equivalent in python. >>> >> >> An alternative to this which you should consider, because it is simpler, >> is to write the vector to a file >> using some format that PETSc understands, Then you just need >> vr.view(viewer) for a viewer like >> the binary viewer or some ASCII format you like. >> >> Thanks, >> >> Matt >> >>> - I need to solve eigenvalue problems up to a dimension of 100000 >>> degrees of freedom and i need all eigenvalues and eigenvectors. I think >>> solving all eigenvalues in one process is far too much and i thought about >>> if it is possible to apply the spectrum slicing described in Chap. 3.4.5. >>> Due to the nature of my problem, i am able to simulate smaller systems of >>> 10000 DOF and extract the biggest eigenvalue, which will be the same >>> for larger systems sizes. Is this in general possible since i have a >>> standard HEP problem or is there a better and faster possibility to do >>> this? >>> >>> >>> In general, SLEPc is not intended for computing the whole spectrum. You >>> can try with spectrum slicing but this will be competitive if computing >>> just a percentage of eigenvalues, 50% say. >>> >>> Jose >>> >>> >>> Thank you very much! >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Sep 17 03:47:25 2018 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 17 Sep 2018 10:47:25 +0200 Subject: [petsc-users] Retireve eigenvectors from a paralell job/ Spectrum slicing in order to solve big eigenvalue problem In-Reply-To: References: <96EE281A-3EC9-4CD7-884B-1AC45D1A6A19@dsic.upv.es> Message-ID: You need a parallel direct solver such as MUMPS. This is explained in section 3.4.5. Jose > El 17 sept 2018, a las 10:41, Jan Grie?er escribi?: > > def solve_eigensystem(DynMatrix_nn, Unity_nn, Dimension, LowerLimit, UpperLimit): > # Create the EPS solver > E = SLEPc.EPS().create() > > # Create the preconditioner and set it to Cholesky > pc = PETSc.PC().create() > pc.setType(pc.Type.CHOLESKY) > > # Create the KSP object > ksp = PETSc.KSP().create() > ksp.setType(ksp.Type.PREONLY) > ksp.setPC(pc) > > # Set up the spectral transformations > st = SLEPc.ST().create() > st.setType("sinvert") > st.setKSP(ksp) > # Setup spectral transformation > E.setST(st) > > # Eigenvalues should be real, therefore we start to order them from the smallest real value |l.real| > E.setWhichEigenpairs(E.Which.ALL) > # Set the interval of spectrum slicing > E.setInterval(LowerLimit, UpperLimit) > # Since the dynamical matrix is symmetric and real it is hermitian. Use GHEP for the spectrum slicing. Operatormatrix B is just a unit matrix > E.setProblemType(SLEPc.EPS.ProblemType.GHEP) > # Use the Krylov Schur method to solve the eigenvalue problem > E.setType(E.Type.KRYLOVSCHUR) > # Partition the Krylov schnur problem in npart procceses > E.setKrylovSchurPartitions(10) > # Set the convergence criterion to relative to the eigenvalue and the maximal number of iterations > E.setConvergenceTest(E.Conv.REL) > E.setTolerances(tol = 1e-7, max_it = 1000) > # Set the matrix in order to solve > E.setOperators(DynMatrix_nn, Unity_nn) > # Sets EPS options from the options database. > E.setFromOptions() > # Sets up all the internal data structures necessary for the execution of the eigensolver. > E.setUp() > > # Solve eigenvalue problem > startClock = time.clock() > startTime = time.time() > E.solve() > > Has maybe one of you any idea why this happens and where the problem is ? > > Am Mo., 17. Sep. 2018 um 10:40 Uhr schrieb Jan Grie?er : > I am aware that SLEPc is not supposed to calculate all eigenvalues and eigenvectors, my problem is simply that i want for a physical large enough system all of them before i can make the transition to go to the smallest ones. > Competitiveness is of secondary importance at the moment. > But ihave a problem connected with spectrum slicing. I followed the instructions in the manual of Chap. 3.4.5 Spectrum Slicing and converted them to the python package. > But now i get the following error. It appears to me that it is not able to find the ksp object, but i actually do not know why this is the case. > aceback (most recent call last): > File "Eigensolver_spectrum_slicing.py", line 216, in > solve_eigensystem(DynMatrix_nn, Unity_nn, D_nn.shape, opt_dict.LowLimit, opt_dict.UpperLimit) > File "Eigensolver_spectrum_slicing.py", line 121, in solve_eigensystem > E.setUp() > File "SLEPc/EPS.pyx", line 1099, in slepc4py.SLEPc.EPS.setUp > petsc4py.PETSc.Error: error code 92 > [14] EPSSetUp() line 165 in /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c > [14] EPSSetUp_KrylovSchur() line 146 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c > [14] EPSSetUp_KrylovSchur_Slice() line 410 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > [14] EPSSliceGetEPS() line 300 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > [14] EPSSetUp() line 165 in /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c > [14] EPSSetUp_KrylovSchur() line 146 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c > [14] EPSSetUp_KrylovSchur_Slice() line 461 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > [14] EPSSliceGetInertia() line 331 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > [14] STSetUp() line 271 in /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/interface/stsolve.c > [14] STSetUp_Sinvert() line 132 in /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/impls/sinvert/sinvert.c > [14] KSPSetUp() line 381 in /tmp/pip-install-xmiaat2t/petsc/src/ksp/ksp/interface/itfunc.c > [14] PCSetUp() line 923 in /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/interface/precon.c > [14] PCSetUp_Cholesky() line 86 in /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c > [14] MatGetFactor() line 4318 in /tmp/pip-install-xmiaat2t/petsc/src/mat/interface/matrix.c > [14] See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers > [14] Could not locate a solver package. Perhaps you must ./configure with --download- > > The code i used to solve the problem is > > Am Fr., 14. Sep. 2018 um 18:34 Uhr schrieb Matthew Knepley : > On Fri, Sep 14, 2018 at 12:19 PM Jose E. Roman wrote: > El 14 sept 2018, a las 17:45, Jan Grie?er escribi?: > >> Hey there, >> first i want to say thanks to Satish and Matt for helping with with my last problem with the mpi compilation. I have two questions related to solving a big, hermitian, standard eigenvalue problem using SLEPc4py., compiled with Intel MKL and Intel MPI. >> >> - I am using slepc4py with >> mpi and run it with around -n 20 cores at the moment and how i wanted to ask if there is an easy way to retrieve the eigenvectors? When i run my code and print for i in range(nconv): >> for i in range(nconv): >> >> val = E. >> getEigenpair(i, vr >> , vi) >> Print( >> vr.getArray()) >> i get the parts of the eigenvectors according to the partition of the matrix. Is there any easy way to put them together in an array and write them to file ? (I am struggling a little bit with the building them in the correct order) > > You need VecScatterCreateToZero. There must be an equivalent in python. > > An alternative to this which you should consider, because it is simpler, is to write the vector to a file > using some format that PETSc understands, Then you just need vr.view(viewer) for a viewer like > the binary viewer or some ASCII format you like. > > Thanks, > > Matt >> - I need to solve eigenvalue problems up to a dimension of 100000 degrees of freedom and i need all eigenvalues and eigenvectors. I think solving all eigenvalues in one process is far too much and i thought about if it is possible to apply the spectrum slicing described in Chap. 3.4.5. Due to the nature of my problem, i am able to simulate smaller systems of 10000 DOF and extract the biggest eigenvalue, which will be the same for larger systems sizes. Is this in general possible since i have a standard HEP problem or is there a better and faster possibility to do this? > > In general, SLEPc is not intended for computing the whole spectrum. You can try with spectrum slicing but this will be competitive if computing just a percentage of eigenvalues, 50% say. > > Jose > >> >> Thank you very much! > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From griesser.jan at googlemail.com Mon Sep 17 03:50:57 2018 From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=) Date: Mon, 17 Sep 2018 10:50:57 +0200 Subject: [petsc-users] Retireve eigenvectors from a paralell job/ Spectrum slicing in order to solve big eigenvalue problem In-Reply-To: References: <96EE281A-3EC9-4CD7-884B-1AC45D1A6A19@dsic.upv.es> Message-ID: Is this relly necessary, because in the last sentences of the chapter it states that: An additional benefit of multi-communicator support is that it enables parallel spectrum slicing runs without the need to install a parallel direct solver (MUMPS). The following commandline example uses sequential linear solves in 4 partitions, one process each: Therefore i assumed that it is not necessary to compile PETsc4py with an external solver e.g. MUMPS Am Mo., 17. Sep. 2018 um 10:47 Uhr schrieb Jose E. Roman : > You need a parallel direct solver such as MUMPS. This is explained in > section 3.4.5. > Jose > > > > El 17 sept 2018, a las 10:41, Jan Grie?er > escribi?: > > > > def solve_eigensystem(DynMatrix_nn, Unity_nn, Dimension, LowerLimit, > UpperLimit): > > # Create the EPS solver > > E = SLEPc.EPS().create() > > > > # Create the preconditioner and set it to Cholesky > > pc = PETSc.PC().create() > > pc.setType(pc.Type.CHOLESKY) > > > > # Create the KSP object > > ksp = PETSc.KSP().create() > > ksp.setType(ksp.Type.PREONLY) > > ksp.setPC(pc) > > > > # Set up the spectral transformations > > st = SLEPc.ST().create() > > st.setType("sinvert") > > st.setKSP(ksp) > > # Setup spectral transformation > > E.setST(st) > > > > # Eigenvalues should be real, therefore we start to order them > from the smallest real value |l.real| > > E.setWhichEigenpairs(E.Which.ALL) > > # Set the interval of spectrum slicing > > E.setInterval(LowerLimit, UpperLimit) > > # Since the dynamical matrix is symmetric and real it is > hermitian. Use GHEP for the spectrum slicing. Operatormatrix B is just a > unit matrix > > E.setProblemType(SLEPc.EPS.ProblemType.GHEP) > > # Use the Krylov Schur method to solve the eigenvalue problem > > E.setType(E.Type.KRYLOVSCHUR) > > # Partition the Krylov schnur problem in npart procceses > > E.setKrylovSchurPartitions(10) > > # Set the convergence criterion to relative to the eigenvalue and > the maximal number of iterations > > E.setConvergenceTest(E.Conv.REL) > > E.setTolerances(tol = 1e-7, max_it = 1000) > > # Set the matrix in order to solve > > E.setOperators(DynMatrix_nn, Unity_nn) > > # Sets EPS options from the options database. > > E.setFromOptions() > > # Sets up all the internal data structures necessary for the > execution of the eigensolver. > > E.setUp() > > > > # Solve eigenvalue problem > > startClock = time.clock() > > startTime = time.time() > > E.solve() > > > > Has maybe one of you any idea why this happens and where the problem is > ? > > > > Am Mo., 17. Sep. 2018 um 10:40 Uhr schrieb Jan Grie?er < > griesser.jan at googlemail.com>: > > I am aware that SLEPc is not supposed to calculate all eigenvalues and > eigenvectors, my problem is simply that i want for a physical large enough > system all of them before i can make the transition to go to the smallest > ones. > > Competitiveness is of secondary importance at the moment. > > But ihave a problem connected with spectrum slicing. I followed the > instructions in the manual of Chap. 3.4.5 Spectrum Slicing and converted > them to the python package. > > But now i get the following error. It appears to me that it is not able > to find the ksp object, but i actually do not know why this is the case. > > aceback (most recent call last): > > File "Eigensolver_spectrum_slicing.py", line 216, in > > solve_eigensystem(DynMatrix_nn, Unity_nn, D_nn.shape, > opt_dict.LowLimit, opt_dict.UpperLimit) > > File "Eigensolver_spectrum_slicing.py", line 121, in solve_eigensystem > > E.setUp() > > File "SLEPc/EPS.pyx", line 1099, in slepc4py.SLEPc.EPS.setUp > > petsc4py.PETSc.Error: error code 92 > > [14] EPSSetUp() line 165 in > /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c > > [14] EPSSetUp_KrylovSchur() line 146 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c > > [14] EPSSetUp_KrylovSchur_Slice() line 410 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > > [14] EPSSliceGetEPS() line 300 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > > [14] EPSSetUp() line 165 in > /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c > > [14] EPSSetUp_KrylovSchur() line 146 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c > > [14] EPSSetUp_KrylovSchur_Slice() line 461 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > > [14] EPSSliceGetInertia() line 331 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > > [14] STSetUp() line 271 in > /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/interface/stsolve.c > > [14] STSetUp_Sinvert() line 132 in > /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/impls/sinvert/sinvert.c > > [14] KSPSetUp() line 381 in > /tmp/pip-install-xmiaat2t/petsc/src/ksp/ksp/interface/itfunc.c > > [14] PCSetUp() line 923 in > /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/interface/precon.c > > [14] PCSetUp_Cholesky() line 86 in > /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c > > [14] MatGetFactor() line 4318 in > /tmp/pip-install-xmiaat2t/petsc/src/mat/interface/matrix.c > > [14] See > http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for > possible LU and Cholesky solvers > > [14] Could not locate a solver package. Perhaps you must ./configure > with --download- > > > > The code i used to solve the problem is > > > > Am Fr., 14. Sep. 2018 um 18:34 Uhr schrieb Matthew Knepley < > knepley at gmail.com>: > > On Fri, Sep 14, 2018 at 12:19 PM Jose E. Roman > wrote: > > El 14 sept 2018, a las 17:45, Jan Grie?er > escribi?: > > > >> Hey there, > >> first i want to say thanks to Satish and Matt for helping with with my > last problem with the mpi compilation. I have two questions related to > solving a big, hermitian, standard eigenvalue problem using SLEPc4py., > compiled with Intel MKL and Intel MPI. > >> > >> - I am using slepc4py with > >> mpi and run it with around -n 20 cores at the moment and how i wanted > to ask if there is an easy way to retrieve the eigenvectors? When i run my > code and print for i in range(nconv): > >> for i in range(nconv): > >> > >> val = E. > >> getEigenpair(i, vr > >> , vi) > >> Print( > >> vr.getArray()) > >> i get the parts of the eigenvectors according to the partition of the > matrix. Is there any easy way to put them together in an array and write > them to file ? (I am struggling a little bit with the building them in the > correct order) > > > > You need VecScatterCreateToZero. There must be an equivalent in python. > > > > An alternative to this which you should consider, because it is simpler, > is to write the vector to a file > > using some format that PETSc understands, Then you just need > vr.view(viewer) for a viewer like > > the binary viewer or some ASCII format you like. > > > > Thanks, > > > > Matt > >> - I need to solve eigenvalue problems up to a dimension of 100000 > degrees of freedom and i need all eigenvalues and eigenvectors. I think > solving all eigenvalues in one process is far too much and i thought about > if it is possible to apply the spectrum slicing described in Chap. 3.4.5. > Due to the nature of my problem, i am able to simulate smaller systems of > 10000 DOF and extract the biggest eigenvalue, which will be the same for > larger systems sizes. Is this in general possible since i have a standard > HEP problem or is there a better and faster possibility to do this? > > > > In general, SLEPc is not intended for computing the whole spectrum. You > can try with spectrum slicing but this will be competitive if computing > just a percentage of eigenvalues, 50% say. > > > > Jose > > > >> > >> Thank you very much! > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Sep 17 03:54:13 2018 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 17 Sep 2018 10:54:13 +0200 Subject: [petsc-users] Retireve eigenvectors from a paralell job/ Spectrum slicing in order to solve big eigenvalue problem In-Reply-To: References: <96EE281A-3EC9-4CD7-884B-1AC45D1A6A19@dsic.upv.es> Message-ID: For this you have to set a number of partitions equal to the number of MPI processes. If you get this error it is because these values are different. Jose > El 17 sept 2018, a las 10:50, Jan Grie?er escribi?: > > Is this relly necessary, because in the last sentences of the chapter it states that: > An additional benefit of multi-communicator support is that it enables parallel spectrum slicing runs without the need to install a parallel direct solver (MUMPS). The following commandline example uses sequential linear solves in 4 partitions, one process each: > Therefore i assumed that it is not necessary to compile PETsc4py with an external solver e.g. MUMPS > > Am Mo., 17. Sep. 2018 um 10:47 Uhr schrieb Jose E. Roman : > You need a parallel direct solver such as MUMPS. This is explained in section 3.4.5. > Jose > > > > El 17 sept 2018, a las 10:41, Jan Grie?er escribi?: > > > > def solve_eigensystem(DynMatrix_nn, Unity_nn, Dimension, LowerLimit, UpperLimit): > > # Create the EPS solver > > E = SLEPc.EPS().create() > > > > # Create the preconditioner and set it to Cholesky > > pc = PETSc.PC().create() > > pc.setType(pc.Type.CHOLESKY) > > > > # Create the KSP object > > ksp = PETSc.KSP().create() > > ksp.setType(ksp.Type.PREONLY) > > ksp.setPC(pc) > > > > # Set up the spectral transformations > > st = SLEPc.ST().create() > > st.setType("sinvert") > > st.setKSP(ksp) > > # Setup spectral transformation > > E.setST(st) > > > > # Eigenvalues should be real, therefore we start to order them from the smallest real value |l.real| > > E.setWhichEigenpairs(E.Which.ALL) > > # Set the interval of spectrum slicing > > E.setInterval(LowerLimit, UpperLimit) > > # Since the dynamical matrix is symmetric and real it is hermitian. Use GHEP for the spectrum slicing. Operatormatrix B is just a unit matrix > > E.setProblemType(SLEPc.EPS.ProblemType.GHEP) > > # Use the Krylov Schur method to solve the eigenvalue problem > > E.setType(E.Type.KRYLOVSCHUR) > > # Partition the Krylov schnur problem in npart procceses > > E.setKrylovSchurPartitions(10) > > # Set the convergence criterion to relative to the eigenvalue and the maximal number of iterations > > E.setConvergenceTest(E.Conv.REL) > > E.setTolerances(tol = 1e-7, max_it = 1000) > > # Set the matrix in order to solve > > E.setOperators(DynMatrix_nn, Unity_nn) > > # Sets EPS options from the options database. > > E.setFromOptions() > > # Sets up all the internal data structures necessary for the execution of the eigensolver. > > E.setUp() > > > > # Solve eigenvalue problem > > startClock = time.clock() > > startTime = time.time() > > E.solve() > > > > Has maybe one of you any idea why this happens and where the problem is ? > > > > Am Mo., 17. Sep. 2018 um 10:40 Uhr schrieb Jan Grie?er : > > I am aware that SLEPc is not supposed to calculate all eigenvalues and eigenvectors, my problem is simply that i want for a physical large enough system all of them before i can make the transition to go to the smallest ones. > > Competitiveness is of secondary importance at the moment. > > But ihave a problem connected with spectrum slicing. I followed the instructions in the manual of Chap. 3.4.5 Spectrum Slicing and converted them to the python package. > > But now i get the following error. It appears to me that it is not able to find the ksp object, but i actually do not know why this is the case. > > aceback (most recent call last): > > File "Eigensolver_spectrum_slicing.py", line 216, in > > solve_eigensystem(DynMatrix_nn, Unity_nn, D_nn.shape, opt_dict.LowLimit, opt_dict.UpperLimit) > > File "Eigensolver_spectrum_slicing.py", line 121, in solve_eigensystem > > E.setUp() > > File "SLEPc/EPS.pyx", line 1099, in slepc4py.SLEPc.EPS.setUp > > petsc4py.PETSc.Error: error code 92 > > [14] EPSSetUp() line 165 in /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c > > [14] EPSSetUp_KrylovSchur() line 146 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c > > [14] EPSSetUp_KrylovSchur_Slice() line 410 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > > [14] EPSSliceGetEPS() line 300 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > > [14] EPSSetUp() line 165 in /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c > > [14] EPSSetUp_KrylovSchur() line 146 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c > > [14] EPSSetUp_KrylovSchur_Slice() line 461 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > > [14] EPSSliceGetInertia() line 331 in /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > > [14] STSetUp() line 271 in /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/interface/stsolve.c > > [14] STSetUp_Sinvert() line 132 in /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/impls/sinvert/sinvert.c > > [14] KSPSetUp() line 381 in /tmp/pip-install-xmiaat2t/petsc/src/ksp/ksp/interface/itfunc.c > > [14] PCSetUp() line 923 in /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/interface/precon.c > > [14] PCSetUp_Cholesky() line 86 in /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c > > [14] MatGetFactor() line 4318 in /tmp/pip-install-xmiaat2t/petsc/src/mat/interface/matrix.c > > [14] See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers > > [14] Could not locate a solver package. Perhaps you must ./configure with --download- > > > > The code i used to solve the problem is > > > > Am Fr., 14. Sep. 2018 um 18:34 Uhr schrieb Matthew Knepley : > > On Fri, Sep 14, 2018 at 12:19 PM Jose E. Roman wrote: > > El 14 sept 2018, a las 17:45, Jan Grie?er escribi?: > > > >> Hey there, > >> first i want to say thanks to Satish and Matt for helping with with my last problem with the mpi compilation. I have two questions related to solving a big, hermitian, standard eigenvalue problem using SLEPc4py., compiled with Intel MKL and Intel MPI. > >> > >> - I am using slepc4py with > >> mpi and run it with around -n 20 cores at the moment and how i wanted to ask if there is an easy way to retrieve the eigenvectors? When i run my code and print for i in range(nconv): > >> for i in range(nconv): > >> > >> val = E. > >> getEigenpair(i, vr > >> , vi) > >> Print( > >> vr.getArray()) > >> i get the parts of the eigenvectors according to the partition of the matrix. Is there any easy way to put them together in an array and write them to file ? (I am struggling a little bit with the building them in the correct order) > > > > You need VecScatterCreateToZero. There must be an equivalent in python. > > > > An alternative to this which you should consider, because it is simpler, is to write the vector to a file > > using some format that PETSc understands, Then you just need vr.view(viewer) for a viewer like > > the binary viewer or some ASCII format you like. > > > > Thanks, > > > > Matt > >> - I need to solve eigenvalue problems up to a dimension of 100000 degrees of freedom and i need all eigenvalues and eigenvectors. I think solving all eigenvalues in one process is far too much and i thought about if it is possible to apply the spectrum slicing described in Chap. 3.4.5. Due to the nature of my problem, i am able to simulate smaller systems of 10000 DOF and extract the biggest eigenvalue, which will be the same for larger systems sizes. Is this in general possible since i have a standard HEP problem or is there a better and faster possibility to do this? > > > > In general, SLEPc is not intended for computing the whole spectrum. You can try with spectrum slicing but this will be competitive if computing just a percentage of eigenvalues, 50% say. > > > > Jose > > > >> > >> Thank you very much! > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > From perceval.desforges at ens-lyon.fr Mon Sep 17 10:24:28 2018 From: perceval.desforges at ens-lyon.fr (Desforges Perceval) Date: Mon, 17 Sep 2018 17:24:28 +0200 Subject: [petsc-users] First shift renders no information Message-ID: Hello PETSc developpers, I have been using SLEPc and PETSc for solving my sparse systems for a small time now, and have been very satisfied with everything. However, I have been running into an error lately, and I have been unable to find any online resource that talks about this specific error. I use SLEPc to find all the eigenvalues of certain tridiagonal and symmetric matrices. I set the solver to find all the eigenvalues in a large interval ([0;10] for example) using the spectrum splicing technique. I use MUMPS because I would like to perform this calculation in parallel later on, but for now I am doing it sequentially. While the solver works very well for small enough matrices (n=2000), when I increase the matrix size to around 10000, I get this error that I do not really understand: [0]PETSC ERROR: First shift renders no information Could someone please enlighten me as to what this error means? I don't know if it's relevant but I am using older version of PETSc (3.8.4) and SLEPc (3.8.2). I apologize in advance if this question has been answered somewhere already. Thank you very much, Sincerely, Desforges Perceval -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Sep 17 10:51:01 2018 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 17 Sep 2018 17:51:01 +0200 Subject: [petsc-users] First shift renders no information In-Reply-To: References: Message-ID: <3B2AE31E-C317-4795-AB92-B36BDDFC274C@dsic.upv.es> This can happen sometimes when the chosen interval for spectrum slicing is to large or the endpoints of the interval are too far away from the eigenvalues. Try moving the endpoints of the interval closer to the eigenvalues. Jose > El 17 sept 2018, a las 17:24, Desforges Perceval escribi?: > > Hello PETSc developpers, > > I have been using SLEPc and PETSc for solving my sparse systems for a small time now, and have been very satisfied with everything. > > However, I have been running into an error lately, and I have been unable to find any online resource that talks about this specific error. > > I use SLEPc to find all the eigenvalues of certain tridiagonal and symmetric matrices. I set the solver to find all the eigenvalues in a large interval ([0;10] for example) using the spectrum splicing technique. I use MUMPS because I would like to perform this calculation in parallel later on, but for now I am doing it sequentially. > > While the solver works very well for small enough matrices (n=2000), when I increase the matrix size to around 10000, I get this error that I do not really understand: > > [0]PETSC ERROR: First shift renders no information > > Could someone please enlighten me as to what this error means? I don't know if it's relevant but I am using older version of PETSc (3.8.4) and SLEPc (3.8.2). > > I apologize in advance if this question has been answered somewhere already. > > Thank you very much, > > Sincerely, > > Desforges Perceval From zakaryah at gmail.com Mon Sep 17 11:12:29 2018 From: zakaryah at gmail.com (zakaryah) Date: Mon, 17 Sep 2018 12:12:29 -0400 Subject: [petsc-users] Basic question about local submatrix from DMComposite Message-ID: I'm still working on implementing a homotopy solver for systems of nonlinear equations. It seems that everything is working in serial, but I'm having at least one issue in parallel. To form the (n+1) x (n+1) Jacobian, I use MatGetLocalSubMatrix, with the ISs from the DMComposite. One of the submatrices is (globally) 1 x n. I need to set one of its values, at global column j, to a particular value. My current attempt to do this involves getting the local to global mappings, with DMCompositeGetISLocalToGlobalMappings, then using ISGlobalToLocalMappingsApply with global index j. If the processor has a locally mapped index, I call MatSetValuesLocal at the local index. So far, this works, in that it seems the value is set at the appropriate point. My issue is that I must use ADD_VALUES, due to the nature of the algorithm for the other submatrices. Therefore, I need a way to ensure that the correct value is set. For example, if two processors map to the same global index, then I need a way to check this and use local values such that the sum is the correct global value. I suspect that this has a simple solution - hopefully someone has some hints. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 17 12:18:52 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Sep 2018 13:18:52 -0400 Subject: [petsc-users] Basic question about local submatrix from DMComposite In-Reply-To: References: Message-ID: On Mon, Sep 17, 2018 at 12:12 PM zakaryah wrote: > I'm still working on implementing a homotopy solver for systems of > nonlinear equations. It seems that everything is working in serial, but > I'm having at least one issue in parallel. > > To form the (n+1) x (n+1) Jacobian, I use MatGetLocalSubMatrix, with the > ISs from the DMComposite. One of the submatrices is (globally) 1 x n. I > need to set one of its values, at global column j, to a particular value. > My current attempt to do this involves getting the local to global > mappings, with DMCompositeGetISLocalToGlobalMappings, then using > ISGlobalToLocalMappingsApply with global index j. If the processor has a > locally mapped index, I call MatSetValuesLocal at the local index. > > So far, this works, in that it seems the value is set at the appropriate > point. My issue is that I must use ADD_VALUES, due to the nature of the > algorithm for the other submatrices. Therefore, I need a way to ensure > that the correct value is set. For example, if two processors map to the > same global index, then I need a way to check this and use local values > such that the sum is the correct global value. > This sounds wrong. I think you should fix the algorithm to use INSERT, which is what you really want. You can usually partition it into and ADD phase, then call Assembly with FLUSH, and then an INSERT phase. Thanks, Matt > I suspect that this has a simple solution - hopefully someone has some > hints. Thanks! > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From olk548 at mail.usask.ca Mon Sep 17 15:31:26 2018 From: olk548 at mail.usask.ca (Oleksandr Koshkarov) Date: Mon, 17 Sep 2018 14:31:26 -0600 Subject: [petsc-users] DMCreatMatrix preallocation Message-ID: <620ee137-041f-6df4-18ae-79bc7d5c9c95@mail.usask.ca> Hello All, I have a question about preallocation of DMDA matrix. As I understand, it preallocates the number of nonzeros using the stencil width info. However, it seems it will not be efficient for me: I have a large dof (around 2000), where only some of those dof's are coupled, so those dof*dof blocks inside the DMMatrix would be also sparse. I also use this DMDA matrix as preconditioner, which has lower coupling than original DMDA array, so stencil is effectively smaller (I precondition higher order finite volume method with lower order FV method). Can I manually specify how many zeros needs to be preallocated for the DMDA matrix? p.s. I do not want o use normal matrix, as I relay on dmda indexing (MatSetValuesStencil). Thank you and best regards, Oleksandr. From bsmith at mcs.anl.gov Mon Sep 17 15:37:20 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 17 Sep 2018 20:37:20 +0000 Subject: [petsc-users] DMCreatMatrix preallocation In-Reply-To: <620ee137-041f-6df4-18ae-79bc7d5c9c95@mail.usask.ca> References: <620ee137-041f-6df4-18ae-79bc7d5c9c95@mail.usask.ca> Message-ID: You can use DMDASetBlockFills() or DMDASetBlockFillsSparse(). If you need any finer scale control then they offer you need to copy the PETSc source that does the preallocation for DMDA generated matrices and customize it exactly for your problem. Barry > On Sep 17, 2018, at 3:31 PM, Oleksandr Koshkarov wrote: > > Hello All, > > I have a question about preallocation of DMDA matrix. As I understand, it preallocates the number of nonzeros using the stencil width info. However, it seems it will not be efficient for me: I have a large dof (around 2000), where only some of those dof's are coupled, so those dof*dof blocks inside the DMMatrix would be also sparse. I also use this DMDA matrix as preconditioner, which has lower coupling than original DMDA array, so stencil is effectively smaller (I precondition higher order finite volume method with lower order FV method). Can I manually specify how many zeros needs to be preallocated for the DMDA matrix? > > p.s. I do not want o use normal matrix, as I relay on dmda indexing (MatSetValuesStencil). > > Thank you and best regards, > > Oleksandr. > From zakaryah at gmail.com Mon Sep 17 15:48:32 2018 From: zakaryah at gmail.com (zakaryah) Date: Mon, 17 Sep 2018 16:48:32 -0400 Subject: [petsc-users] Basic question about local submatrix from DMComposite In-Reply-To: References: Message-ID: That worked - thanks Matt On Mon, Sep 17, 2018 at 1:19 PM Matthew Knepley wrote: > On Mon, Sep 17, 2018 at 12:12 PM zakaryah wrote: > >> I'm still working on implementing a homotopy solver for systems of >> nonlinear equations. It seems that everything is working in serial, but >> I'm having at least one issue in parallel. >> >> To form the (n+1) x (n+1) Jacobian, I use MatGetLocalSubMatrix, with the >> ISs from the DMComposite. One of the submatrices is (globally) 1 x n. I >> need to set one of its values, at global column j, to a particular value. >> My current attempt to do this involves getting the local to global >> mappings, with DMCompositeGetISLocalToGlobalMappings, then using >> ISGlobalToLocalMappingsApply with global index j. If the processor has a >> locally mapped index, I call MatSetValuesLocal at the local index. >> >> So far, this works, in that it seems the value is set at the appropriate >> point. My issue is that I must use ADD_VALUES, due to the nature of the >> algorithm for the other submatrices. Therefore, I need a way to ensure >> that the correct value is set. For example, if two processors map to the >> same global index, then I need a way to check this and use local values >> such that the sum is the correct global value. >> > > This sounds wrong. I think you should fix the algorithm to use INSERT, > which is what you really want. You can usually partition it into > and ADD phase, then call Assembly with FLUSH, and then an INSERT phase. > > Thanks, > > Matt > > >> I suspect that this has a simple solution - hopefully someone has some >> hints. Thanks! >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olk548 at mail.usask.ca Mon Sep 17 17:36:57 2018 From: olk548 at mail.usask.ca (Oleksandr Koshkarov) Date: Mon, 17 Sep 2018 16:36:57 -0600 Subject: [petsc-users] DMCreatMatrix preallocation In-Reply-To: References: <620ee137-041f-6df4-18ae-79bc7d5c9c95@mail.usask.ca> Message-ID: <57a5a106-1492-d832-95de-75ee7c5bf505@mail.usask.ca> Wow, that is neat, thank you! I think it is exactly what I need. However, can you please clarify some details: 1) does "ofill" represent all non diagonal blocs?? meaning they have the same pattern. (e.g., in 1d I will have 1+2*stencil_width blocks) 2) this note: "This only makes sense when you are doing multicomponent problems but using the MPIAIJ matrix format". Sounds like MPIAIJ is wrong choice for multi component problem. Actually, I do not pick the DMDA matrix format myself, and use petsc default (which is probably MPIAIJ). Maybe, I should something else more heavily multi component problem? My matrices appears from discretizing 6D PDE, where I put 3D inside dof... The discretization is all stencil based (finite volume method). p.s. All those functions and flexibility imply the enormous beauty in PETSc structure. I wish to start reading petsc internal code, to be more comfortable using it and to ask more smart questions to you guys. However, it is quite intimidating to start. Maybe you have some pointers, how can I start? Thank you and best regards, Oleksandr. On 09/17/2018 02:37 PM, Smith, Barry F. wrote: > You can use DMDASetBlockFills() or DMDASetBlockFillsSparse(). > > If you need any finer scale control then they offer you need to copy the PETSc source that does the preallocation for DMDA generated matrices and customize it exactly for your problem. > > > Barry > > >> On Sep 17, 2018, at 3:31 PM, Oleksandr Koshkarov wrote: >> >> Hello All, >> >> I have a question about preallocation of DMDA matrix. As I understand, it preallocates the number of nonzeros using the stencil width info. However, it seems it will not be efficient for me: I have a large dof (around 2000), where only some of those dof's are coupled, so those dof*dof blocks inside the DMMatrix would be also sparse. I also use this DMDA matrix as preconditioner, which has lower coupling than original DMDA array, so stencil is effectively smaller (I precondition higher order finite volume method with lower order FV method). Can I manually specify how many zeros needs to be preallocated for the DMDA matrix? >> >> p.s. I do not want o use normal matrix, as I relay on dmda indexing (MatSetValuesStencil). >> >> Thank you and best regards, >> >> Oleksandr. >> From bsmith at mcs.anl.gov Mon Sep 17 17:44:55 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 17 Sep 2018 22:44:55 +0000 Subject: [petsc-users] DMCreatMatrix preallocation In-Reply-To: <57a5a106-1492-d832-95de-75ee7c5bf505@mail.usask.ca> References: <620ee137-041f-6df4-18ae-79bc7d5c9c95@mail.usask.ca> <57a5a106-1492-d832-95de-75ee7c5bf505@mail.usask.ca> Message-ID: <6FCF2838-CCDE-47F7-B9F3-7A5CD9AD5016@mcs.anl.gov> > On Sep 17, 2018, at 5:36 PM, Oleksandr Koshkarov wrote: > > Wow, that is neat, thank you! I think it is exactly what I need. However, can you please clarify some details: > > 1) does "ofill" represent all non diagonal blocs? meaning they have the same pattern. (e.g., in 1d I will have 1+2*stencil_width blocks) Yes, it represents all the non diagonal blocks, this is a weakness in that depending on the discretization different diagonal blocks may have different nonzero structure. You need to use the union of all the off-diagonal block matrix nonzero structure. > > 2) this note: "This only makes sense when you are doing multicomponent problems but using the MPIAIJ matrix format". Sounds like MPIAIJ is wrong choice for multi component problem. Actually, I do not pick the DMDA matrix format myself, and use petsc default (which is probably MPIAIJ). It is MPIAIJ. > Maybe, I should something else more heavily multi component problem? My matrices appears from discretizing 6D PDE, where I put 3D inside dof... The discretization is all stencil based (finite volume method). The BAIJ format is only appropriate if the "blocks" are essentially dense (since BAIJ stores them as dense). BAIJ format is not correct for you. You should be using the MPIAIJ format. Barry > > p.s. All those functions and flexibility imply the enormous beauty in PETSc structure. I wish to start reading petsc internal code, to be more comfortable using it and to ask more smart questions to you guys. However, it is quite intimidating to start. Maybe you have some pointers, how can I start? > > Thank you and best regards, > > Oleksandr. > > > On 09/17/2018 02:37 PM, Smith, Barry F. wrote: >> You can use DMDASetBlockFills() or DMDASetBlockFillsSparse(). >> >> If you need any finer scale control then they offer you need to copy the PETSc source that does the preallocation for DMDA generated matrices and customize it exactly for your problem. >> >> >> Barry >> >> >>> On Sep 17, 2018, at 3:31 PM, Oleksandr Koshkarov wrote: >>> >>> Hello All, >>> >>> I have a question about preallocation of DMDA matrix. As I understand, it preallocates the number of nonzeros using the stencil width info. However, it seems it will not be efficient for me: I have a large dof (around 2000), where only some of those dof's are coupled, so those dof*dof blocks inside the DMMatrix would be also sparse. I also use this DMDA matrix as preconditioner, which has lower coupling than original DMDA array, so stencil is effectively smaller (I precondition higher order finite volume method with lower order FV method). Can I manually specify how many zeros needs to be preallocated for the DMDA matrix? >>> >>> p.s. I do not want o use normal matrix, as I relay on dmda indexing (MatSetValuesStencil). >>> >>> Thank you and best regards, >>> >>> Oleksandr. >>> > From olk548 at mail.usask.ca Mon Sep 17 17:54:00 2018 From: olk548 at mail.usask.ca (Oleksandr Koshkarov) Date: Mon, 17 Sep 2018 16:54:00 -0600 Subject: [petsc-users] DMCreatMatrix preallocation In-Reply-To: <6FCF2838-CCDE-47F7-B9F3-7A5CD9AD5016@mcs.anl.gov> References: <620ee137-041f-6df4-18ae-79bc7d5c9c95@mail.usask.ca> <57a5a106-1492-d832-95de-75ee7c5bf505@mail.usask.ca> <6FCF2838-CCDE-47F7-B9F3-7A5CD9AD5016@mcs.anl.gov> Message-ID: <4ba068b5-0115-c079-7473-bfce9be79e3d@mail.usask.ca> Thank you. One more question: Are DMDA matrices with smaller stencil compatible with state vectors with larger stencil width? If I precondition finite volume method were I use stencil width 2 (or more) with lower order method with stencil width 1, can this still work? (My jacobian is matrix free). Can this work together? Thank you and best regards, Oleksandr. On 09/17/2018 04:44 PM, Smith, Barry F. wrote: > >> On Sep 17, 2018, at 5:36 PM, Oleksandr Koshkarov wrote: >> >> Wow, that is neat, thank you! I think it is exactly what I need. However, can you please clarify some details: >> >> 1) does "ofill" represent all non diagonal blocs? meaning they have the same pattern. (e.g., in 1d I will have 1+2*stencil_width blocks) > Yes, it represents all the non diagonal blocks, this is a weakness in that depending on the discretization different diagonal blocks may have different nonzero structure. You need to use the union of all the off-diagonal block matrix nonzero structure. >> 2) this note: "This only makes sense when you are doing multicomponent problems but using the MPIAIJ matrix format". Sounds like MPIAIJ is wrong choice for multi component problem. Actually, I do not pick the DMDA matrix format myself, and use petsc default (which is probably MPIAIJ). > It is MPIAIJ. > >> Maybe, I should something else more heavily multi component problem? My matrices appears from discretizing 6D PDE, where I put 3D inside dof... The discretization is all stencil based (finite volume method). > The BAIJ format is only appropriate if the "blocks" are essentially dense (since BAIJ stores them as dense). BAIJ format is not correct for you. You should be using the MPIAIJ format. > > Barry > >> p.s. All those functions and flexibility imply the enormous beauty in PETSc structure. I wish to start reading petsc internal code, to be more comfortable using it and to ask more smart questions to you guys. However, it is quite intimidating to start. Maybe you have some pointers, how can I start? >> >> Thank you and best regards, >> >> Oleksandr. >> >> >> On 09/17/2018 02:37 PM, Smith, Barry F. wrote: >>> You can use DMDASetBlockFills() or DMDASetBlockFillsSparse(). >>> >>> If you need any finer scale control then they offer you need to copy the PETSc source that does the preallocation for DMDA generated matrices and customize it exactly for your problem. >>> >>> >>> Barry >>> >>> >>>> On Sep 17, 2018, at 3:31 PM, Oleksandr Koshkarov wrote: >>>> >>>> Hello All, >>>> >>>> I have a question about preallocation of DMDA matrix. As I understand, it preallocates the number of nonzeros using the stencil width info. However, it seems it will not be efficient for me: I have a large dof (around 2000), where only some of those dof's are coupled, so those dof*dof blocks inside the DMMatrix would be also sparse. I also use this DMDA matrix as preconditioner, which has lower coupling than original DMDA array, so stencil is effectively smaller (I precondition higher order finite volume method with lower order FV method). Can I manually specify how many zeros needs to be preallocated for the DMDA matrix? >>>> >>>> p.s. I do not want o use normal matrix, as I relay on dmda indexing (MatSetValuesStencil). >>>> >>>> Thank you and best regards, >>>> >>>> Oleksandr. >>>> From bsmith at mcs.anl.gov Mon Sep 17 18:23:09 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 17 Sep 2018 23:23:09 +0000 Subject: [petsc-users] DMCreatMatrix preallocation In-Reply-To: <4ba068b5-0115-c079-7473-bfce9be79e3d@mail.usask.ca> References: <620ee137-041f-6df4-18ae-79bc7d5c9c95@mail.usask.ca> <57a5a106-1492-d832-95de-75ee7c5bf505@mail.usask.ca> <6FCF2838-CCDE-47F7-B9F3-7A5CD9AD5016@mcs.anl.gov> <4ba068b5-0115-c079-7473-bfce9be79e3d@mail.usask.ca> Message-ID: <1463E25E-320F-438F-92C0-256B03F4FEFF@anl.gov> > On Sep 17, 2018, at 5:54 PM, Oleksandr Koshkarov wrote: > > Thank you. One more question: > > Are DMDA matrices with smaller stencil compatible with state vectors with larger stencil width? If I precondition finite volume method were I use stencil width 2 (or more) with lower order method with stencil width 1, can this still work? (My jacobian is matrix free). Can this work together? Yes > > Thank you and best regards, > > Oleksandr. > > > On 09/17/2018 04:44 PM, Smith, Barry F. wrote: >> >>> On Sep 17, 2018, at 5:36 PM, Oleksandr Koshkarov wrote: >>> >>> Wow, that is neat, thank you! I think it is exactly what I need. However, can you please clarify some details: >>> >>> 1) does "ofill" represent all non diagonal blocs? meaning they have the same pattern. (e.g., in 1d I will have 1+2*stencil_width blocks) >> Yes, it represents all the non diagonal blocks, this is a weakness in that depending on the discretization different diagonal blocks may have different nonzero structure. You need to use the union of all the off-diagonal block matrix nonzero structure. >>> 2) this note: "This only makes sense when you are doing multicomponent problems but using the MPIAIJ matrix format". Sounds like MPIAIJ is wrong choice for multi component problem. Actually, I do not pick the DMDA matrix format myself, and use petsc default (which is probably MPIAIJ). >> It is MPIAIJ. >> >>> Maybe, I should something else more heavily multi component problem? My matrices appears from discretizing 6D PDE, where I put 3D inside dof... The discretization is all stencil based (finite volume method). >> The BAIJ format is only appropriate if the "blocks" are essentially dense (since BAIJ stores them as dense). BAIJ format is not correct for you. You should be using the MPIAIJ format. >> >> Barry >> >>> p.s. All those functions and flexibility imply the enormous beauty in PETSc structure. I wish to start reading petsc internal code, to be more comfortable using it and to ask more smart questions to you guys. However, it is quite intimidating to start. Maybe you have some pointers, how can I start? >>> >>> Thank you and best regards, >>> >>> Oleksandr. >>> >>> >>> On 09/17/2018 02:37 PM, Smith, Barry F. wrote: >>>> You can use DMDASetBlockFills() or DMDASetBlockFillsSparse(). >>>> >>>> If you need any finer scale control then they offer you need to copy the PETSc source that does the preallocation for DMDA generated matrices and customize it exactly for your problem. >>>> >>>> >>>> Barry >>>> >>>> >>>>> On Sep 17, 2018, at 3:31 PM, Oleksandr Koshkarov wrote: >>>>> >>>>> Hello All, >>>>> >>>>> I have a question about preallocation of DMDA matrix. As I understand, it preallocates the number of nonzeros using the stencil width info. However, it seems it will not be efficient for me: I have a large dof (around 2000), where only some of those dof's are coupled, so those dof*dof blocks inside the DMMatrix would be also sparse. I also use this DMDA matrix as preconditioner, which has lower coupling than original DMDA array, so stencil is effectively smaller (I precondition higher order finite volume method with lower order FV method). Can I manually specify how many zeros needs to be preallocated for the DMDA matrix? >>>>> >>>>> p.s. I do not want o use normal matrix, as I relay on dmda indexing (MatSetValuesStencil). >>>>> >>>>> Thank you and best regards, >>>>> >>>>> Oleksandr. >>>>> > From bsmith at mcs.anl.gov Mon Sep 17 18:38:38 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 17 Sep 2018 23:38:38 +0000 Subject: [petsc-users] DMCreatMatrix preallocation In-Reply-To: <911e712d-0cf2-aff4-9c62-a7f66b38f30c@mail.usask.ca> References: <620ee137-041f-6df4-18ae-79bc7d5c9c95@mail.usask.ca> <57a5a106-1492-d832-95de-75ee7c5bf505@mail.usask.ca> <6FCF2838-CCDE-47F7-B9F3-7A5CD9AD5016@mcs.anl.gov> <4ba068b5-0115-c079-7473-bfce9be79e3d@mail.usask.ca> <1463E25E-320F-438F-92C0-256B03F4FEFF@anl.gov> <911e712d-0cf2-aff4-9c62-a7f66b38f30c@mail.usask.ca> Message-ID: The key is to understand that most PETSc functionality is supported by two levels of code: the abstract interface (usually defined in src/xxx/interface) and then one or more implementations (usually defined in src/xxx/impls/*). The abstract interface functions are XXXYYYY while the implementation functions names are usually XXXYYYY_ZZZZ. So for example MatCreate() is the abstract interface function and MatCreate_SeqAIJ() is one particular implementation. You can find the the interface and implementation functions by simply googling the manual page (for example http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreate.html) and then looking at the bottom of the page for the location: Location src/mat/utils/gcreate.c and the implementations Implementations MatCreate_HYPREStruct in src/dm/impls/da/hypre/mhyp.c MatCreate_HYPRESStruct in src/dm/impls/da/hypre/mhyp.c MatCreate_MPIAdj in src/mat/impls/adj/mpi/mpiadj.c ... They are all hyper links that you can click on to go exactly to the routine. Barry If you use EMACS or VI/VIM you can use etags or tags (see the PETSc users manual page pages 235, 236) to jump directly the various functions also. > On Sep 17, 2018, at 6:27 PM, Oleksandr Koshkarov wrote: > > Awesome, thank you. All those functions and flexibility imply the enormous beauty in PETSc structure. I wish to start reading petsc internal code, to be more comfortable using it and to ask less stupid questions. However, it is quite intimidating to start. Maybe you have some pointers, where a petsc user can start reading petsc code? > > Thank you and best regards, > > Oleksandr > > > On 09/17/2018 05:23 PM, Smith, Barry F. wrote: >> >>> On Sep 17, 2018, at 5:54 PM, Oleksandr Koshkarov wrote: >>> >>> Thank you. One more question: >>> >>> Are DMDA matrices with smaller stencil compatible with state vectors with larger stencil width? If I precondition finite volume method were I use stencil width 2 (or more) with lower order method with stencil width 1, can this still work? (My jacobian is matrix free). Can this work together? >> Yes >> >>> Thank you and best regards, >>> >>> Oleksandr. >>> >>> >>> On 09/17/2018 04:44 PM, Smith, Barry F. wrote: >>>>> On Sep 17, 2018, at 5:36 PM, Oleksandr Koshkarov wrote: >>>>> >>>>> Wow, that is neat, thank you! I think it is exactly what I need. However, can you please clarify some details: >>>>> >>>>> 1) does "ofill" represent all non diagonal blocs? meaning they have the same pattern. (e.g., in 1d I will have 1+2*stencil_width blocks) >>>> Yes, it represents all the non diagonal blocks, this is a weakness in that depending on the discretization different diagonal blocks may have different nonzero structure. You need to use the union of all the off-diagonal block matrix nonzero structure. >>>>> 2) this note: "This only makes sense when you are doing multicomponent problems but using the MPIAIJ matrix format". Sounds like MPIAIJ is wrong choice for multi component problem. Actually, I do not pick the DMDA matrix format myself, and use petsc default (which is probably MPIAIJ). >>>> It is MPIAIJ. >>>> >>>>> Maybe, I should something else more heavily multi component problem? My matrices appears from discretizing 6D PDE, where I put 3D inside dof... The discretization is all stencil based (finite volume method). >>>> The BAIJ format is only appropriate if the "blocks" are essentially dense (since BAIJ stores them as dense). BAIJ format is not correct for you. You should be using the MPIAIJ format. >>>> >>>> Barry >>>> >>>>> p.s. All those functions and flexibility imply the enormous beauty in PETSc structure. I wish to start reading petsc internal code, to be more comfortable using it and to ask more smart questions to you guys. However, it is quite intimidating to start. Maybe you have some pointers, how can I start? >>>>> >>>>> Thank you and best regards, >>>>> >>>>> Oleksandr. >>>>> >>>>> >>>>> On 09/17/2018 02:37 PM, Smith, Barry F. wrote: >>>>>> You can use DMDASetBlockFills() or DMDASetBlockFillsSparse(). >>>>>> >>>>>> If you need any finer scale control then they offer you need to copy the PETSc source that does the preallocation for DMDA generated matrices and customize it exactly for your problem. >>>>>> >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>>> On Sep 17, 2018, at 3:31 PM, Oleksandr Koshkarov wrote: >>>>>>> >>>>>>> Hello All, >>>>>>> >>>>>>> I have a question about preallocation of DMDA matrix. As I understand, it preallocates the number of nonzeros using the stencil width info. However, it seems it will not be efficient for me: I have a large dof (around 2000), where only some of those dof's are coupled, so those dof*dof blocks inside the DMMatrix would be also sparse. I also use this DMDA matrix as preconditioner, which has lower coupling than original DMDA array, so stencil is effectively smaller (I precondition higher order finite volume method with lower order FV method). Can I manually specify how many zeros needs to be preallocated for the DMDA matrix? >>>>>>> >>>>>>> p.s. I do not want o use normal matrix, as I relay on dmda indexing (MatSetValuesStencil). >>>>>>> >>>>>>> Thank you and best regards, >>>>>>> >>>>>>> Oleksandr. >>>>>>> > From zonexo at gmail.com Tue Sep 18 00:20:56 2018 From: zonexo at gmail.com (TAY wee-beng) Date: Tue, 18 Sep 2018 13:20:56 +0800 Subject: [petsc-users] Unable to read in values thru namelist in Fortran after using PETSc 64bit in linux In-Reply-To: <0AC1BA91-75F8-4AAB-A803-832B089A2C44@anl.gov> References: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com> <0AC1BA91-75F8-4AAB-A803-832B089A2C44@anl.gov> Message-ID: <22565566-590d-7c89-390a-99557ce05382@gmail.com> Hi, I managed to find the error appearing after using PETSc 64bit in linux - call MPI_COMM_SIZE(MPI_COMM_WORLD, num_procs, ierr) I have assigned num_procs as PetscInt and I got 0 instead of 1 (for 1 procs) Assigning num_procs as integer as the problem. Is this supposed to be the case? Or is it a bug? Thank you very much. Yours sincerely, ================================================ TAY Wee-Beng (Zheng Weiming) ??? Personal research webpage: http://tayweebeng.wixsite.com/website Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA linkedin: www.linkedin.com/in/tay-weebeng ================================================ On 8/9/2018 1:14 AM, Smith, Barry F. wrote: > You can try valgrind http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > Barry > > >> On Sep 7, 2018, at 1:44 AM, TAY wee-beng wrote: >> >> Hi, >> >> I found that I am unable to read in values thru namelist in Fortran after using PETSc 64bit in linux. >> >> I have a parameter txt file which is read in using namelist in Fortran? >> >> namelist /body_input/ no_body, convex_body, motion_type, hover, wing_config ... >> >> open (unit = 44 , FILE = "ibm3d_input.txt" , status = "old", iostat = openstatus(4)) >> >> if (openstatus(4) > 0) then >> >> print *, "ibm3d_input file not present or wrong filename." >> >> stop >> >> end if >> >> read (44,nml = solver_input) >> >> read (44,nml = grid_input) >> >> read (44,nml = body_input)... >> >> >> After using PETSc 64bit, my code aborts and I realise that it is because the values have became NaN. Strangely, it does not occur in windows with VS2008. >> >> I wonder if it's a bug with the Intel Fortran compiler 2018. >> >> Anyone has similar experience? >> >> -- >> Thank you very much. >> >> Yours sincerely, >> >> ================================================ >> TAY Wee-Beng (Zheng Weiming) ??? >> Personal research webpage: http://tayweebeng.wixsite.com/website >> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >> linkedin: www.linkedin.com/in/tay-weebeng >> ================================================ >> From balay at mcs.anl.gov Tue Sep 18 00:39:40 2018 From: balay at mcs.anl.gov (Balay, Satish) Date: Tue, 18 Sep 2018 05:39:40 +0000 Subject: [petsc-users] Unable to read in values thru namelist in Fortran after using PETSc 64bit in linux In-Reply-To: <22565566-590d-7c89-390a-99557ce05382@gmail.com> References: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com> <0AC1BA91-75F8-4AAB-A803-832B089A2C44@anl.gov> <22565566-590d-7c89-390a-99557ce05382@gmail.com> Message-ID: https://www.mpich.org/static/docs/v3.1/www3/MPI_Comm_size.html int MPI_Comm_size( MPI_Comm comm, int *size ) i.e there is no PetscInt here. [MPI does not know about PETSc datatypes] For convinence we provide PetscMPIInt to keep track of such variables [similarly PetscBLASInt]. For eg: Check src/vec/vec/examples/tests/ex2f.F Satish On Tue, 18 Sep 2018, TAY wee-beng wrote: > Hi, > > I managed to find the error appearing after using PETSc 64bit in linux - > > call MPI_COMM_SIZE(MPI_COMM_WORLD, num_procs, ierr) > > I have assigned num_procs as PetscInt and I got 0 instead of 1 (for 1 procs) > > Assigning num_procs as integer as the problem. > > Is this supposed to be the case? Or is it a bug? > > Thank you very much. > > Yours sincerely, > > ================================================ > TAY Wee-Beng (Zheng Weiming) ??? > Personal research webpage: http://tayweebeng.wixsite.com/website > Youtube research showcase: > https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > linkedin: www.linkedin.com/in/tay-weebeng > ================================================ > > On 8/9/2018 1:14 AM, Smith, Barry F. wrote: > > You can try valgrind > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > Barry > > > > > >> On Sep 7, 2018, at 1:44 AM, TAY wee-beng wrote: > >> > >> Hi, > >> > >> I found that I am unable to read in values thru namelist in Fortran after > >> using PETSc 64bit in linux. > >> > >> I have a parameter txt file which is read in using namelist in Fortran? > >> > >> namelist /body_input/ no_body, convex_body, motion_type, hover, wing_config > >> ... > >> > >> open (unit = 44 , FILE = "ibm3d_input.txt" , status = "old", iostat = > >> openstatus(4)) > >> > >> if (openstatus(4) > 0) then > >> > >> print *, "ibm3d_input file not present or wrong filename." > >> > >> stop > >> > >> end if > >> > >> read (44,nml = solver_input) > >> > >> read (44,nml = grid_input) > >> > >> read (44,nml = body_input)... > >> > >> > >> After using PETSc 64bit, my code aborts and I realise that it is because > >> the values have became NaN. Strangely, it does not occur in windows with > >> VS2008. > >> > >> I wonder if it's a bug with the Intel Fortran compiler 2018. > >> > >> Anyone has similar experience? > >> > >> -- > >> Thank you very much. > >> > >> Yours sincerely, > >> > >> ================================================ > >> TAY Wee-Beng (Zheng Weiming) ??? > >> Personal research webpage: http://tayweebeng.wixsite.com/website > >> Youtube research showcase: > >> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > >> linkedin: www.linkedin.com/in/tay-weebeng > >> ================================================ > >> > > From zonexo at gmail.com Tue Sep 18 01:39:14 2018 From: zonexo at gmail.com (TAY wee-beng) Date: Tue, 18 Sep 2018 14:39:14 +0800 Subject: [petsc-users] Unable to read in values thru namelist in Fortran after using PETSc 64bit in linux In-Reply-To: References: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com> <0AC1BA91-75F8-4AAB-A803-832B089A2C44@anl.gov> <22565566-590d-7c89-390a-99557ce05382@gmail.com> Message-ID: Hi, In that case, does it apply to all MPI subroutines such as MPI_ALLGATHER? In other words, must I assign local_array_length etc as PetscMPIInt? call MPI_ALLGATHER(local_array_length,1,MPIU_INTEGER,array_length,1,MPIU_INTEGER,MPI_COMM_WORLD,ierr) Or is it ok to change all integers from PetscInt to PetscMPIInt? With the exception of ierr - PetscErrorCode Thank you very much. Yours sincerely, ================================================ TAY Wee-Beng (Zheng Weiming) ??? Personal research webpage: http://tayweebeng.wixsite.com/website Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA linkedin: www.linkedin.com/in/tay-weebeng ================================================ On 18/9/2018 1:39 PM, Balay, Satish wrote: > https://www.mpich.org/static/docs/v3.1/www3/MPI_Comm_size.html > > int MPI_Comm_size( MPI_Comm comm, int *size ) > > i.e there is no PetscInt here. [MPI does not know about PETSc datatypes] > > For convinence we provide PetscMPIInt to keep track of such variables > [similarly PetscBLASInt]. For eg: Check src/vec/vec/examples/tests/ex2f.F > > Satish > > On Tue, 18 Sep 2018, TAY wee-beng wrote: > >> Hi, >> >> I managed to find the error appearing after using PETSc 64bit in linux - >> >> call MPI_COMM_SIZE(MPI_COMM_WORLD, num_procs, ierr) >> >> I have assigned num_procs as PetscInt and I got 0 instead of 1 (for 1 procs) >> >> Assigning num_procs as integer as the problem. >> >> Is this supposed to be the case? Or is it a bug? >> >> Thank you very much. >> >> Yours sincerely, >> >> ================================================ >> TAY Wee-Beng (Zheng Weiming) ??? >> Personal research webpage: http://tayweebeng.wixsite.com/website >> Youtube research showcase: >> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >> linkedin: www.linkedin.com/in/tay-weebeng >> ================================================ >> >> On 8/9/2018 1:14 AM, Smith, Barry F. wrote: >>> You can try valgrind >>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> >>> Barry >>> >>> >>>> On Sep 7, 2018, at 1:44 AM, TAY wee-beng wrote: >>>> >>>> Hi, >>>> >>>> I found that I am unable to read in values thru namelist in Fortran after >>>> using PETSc 64bit in linux. >>>> >>>> I have a parameter txt file which is read in using namelist in Fortran? >>>> >>>> namelist /body_input/ no_body, convex_body, motion_type, hover, wing_config >>>> ... >>>> >>>> open (unit = 44 , FILE = "ibm3d_input.txt" , status = "old", iostat = >>>> openstatus(4)) >>>> >>>> if (openstatus(4) > 0) then >>>> >>>> print *, "ibm3d_input file not present or wrong filename." >>>> >>>> stop >>>> >>>> end if >>>> >>>> read (44,nml = solver_input) >>>> >>>> read (44,nml = grid_input) >>>> >>>> read (44,nml = body_input)... >>>> >>>> >>>> After using PETSc 64bit, my code aborts and I realise that it is because >>>> the values have became NaN. Strangely, it does not occur in windows with >>>> VS2008. >>>> >>>> I wonder if it's a bug with the Intel Fortran compiler 2018. >>>> >>>> Anyone has similar experience? >>>> >>>> -- >>>> Thank you very much. >>>> >>>> Yours sincerely, >>>> >>>> ================================================ >>>> TAY Wee-Beng (Zheng Weiming) ??? >>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>> Youtube research showcase: >>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>> linkedin: www.linkedin.com/in/tay-weebeng >>>> ================================================ >>>> >> From griesser.jan at googlemail.com Tue Sep 18 02:26:23 2018 From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=) Date: Tue, 18 Sep 2018 09:26:23 +0200 Subject: [petsc-users] Retireve eigenvectors from a paralell job/ Spectrum slicing in order to solve big eigenvalue problem In-Reply-To: References: <96EE281A-3EC9-4CD7-884B-1AC45D1A6A19@dsic.upv.es> Message-ID: Now it works. Thank you very much! Am Mo., 17. Sep. 2018 um 10:54 Uhr schrieb Jose E. Roman : > For this you have to set a number of partitions equal to the number of MPI > processes. If you get this error it is because these values are different. > Jose > > > > El 17 sept 2018, a las 10:50, Jan Grie?er > escribi?: > > > > Is this relly necessary, because in the last sentences of the chapter it > states that: > > An additional benefit of multi-communicator support is that it enables > parallel spectrum slicing runs without the need to install a parallel > direct solver (MUMPS). The following commandline example uses sequential > linear solves in 4 partitions, one process each: > > Therefore i assumed that it is not necessary to compile PETsc4py with an > external solver e.g. MUMPS > > > > Am Mo., 17. Sep. 2018 um 10:47 Uhr schrieb Jose E. Roman < > jroman at dsic.upv.es>: > > You need a parallel direct solver such as MUMPS. This is explained in > section 3.4.5. > > Jose > > > > > > > El 17 sept 2018, a las 10:41, Jan Grie?er > escribi?: > > > > > > def solve_eigensystem(DynMatrix_nn, Unity_nn, Dimension, LowerLimit, > UpperLimit): > > > # Create the EPS solver > > > E = SLEPc.EPS().create() > > > > > > # Create the preconditioner and set it to Cholesky > > > pc = PETSc.PC().create() > > > pc.setType(pc.Type.CHOLESKY) > > > > > > # Create the KSP object > > > ksp = PETSc.KSP().create() > > > ksp.setType(ksp.Type.PREONLY) > > > ksp.setPC(pc) > > > > > > # Set up the spectral transformations > > > st = SLEPc.ST().create() > > > st.setType("sinvert") > > > st.setKSP(ksp) > > > # Setup spectral transformation > > > E.setST(st) > > > > > > # Eigenvalues should be real, therefore we start to order them > from the smallest real value |l.real| > > > E.setWhichEigenpairs(E.Which.ALL) > > > # Set the interval of spectrum slicing > > > E.setInterval(LowerLimit, UpperLimit) > > > # Since the dynamical matrix is symmetric and real it is > hermitian. Use GHEP for the spectrum slicing. Operatormatrix B is just a > unit matrix > > > E.setProblemType(SLEPc.EPS.ProblemType.GHEP) > > > # Use the Krylov Schur method to solve the eigenvalue problem > > > E.setType(E.Type.KRYLOVSCHUR) > > > # Partition the Krylov schnur problem in npart procceses > > > E.setKrylovSchurPartitions(10) > > > # Set the convergence criterion to relative to the eigenvalue > and the maximal number of iterations > > > E.setConvergenceTest(E.Conv.REL) > > > E.setTolerances(tol = 1e-7, max_it = 1000) > > > # Set the matrix in order to solve > > > E.setOperators(DynMatrix_nn, Unity_nn) > > > # Sets EPS options from the options database. > > > E.setFromOptions() > > > # Sets up all the internal data structures necessary for the > execution of the eigensolver. > > > E.setUp() > > > > > > # Solve eigenvalue problem > > > startClock = time.clock() > > > startTime = time.time() > > > E.solve() > > > > > > Has maybe one of you any idea why this happens and where the problem > is ? > > > > > > Am Mo., 17. Sep. 2018 um 10:40 Uhr schrieb Jan Grie?er < > griesser.jan at googlemail.com>: > > > I am aware that SLEPc is not supposed to calculate all eigenvalues and > eigenvectors, my problem is simply that i want for a physical large enough > system all of them before i can make the transition to go to the smallest > ones. > > > Competitiveness is of secondary importance at the moment. > > > But ihave a problem connected with spectrum slicing. I followed the > instructions in the manual of Chap. 3.4.5 Spectrum Slicing and converted > them to the python package. > > > But now i get the following error. It appears to me that it is not > able to find the ksp object, but i actually do not know why this is the > case. > > > aceback (most recent call last): > > > File "Eigensolver_spectrum_slicing.py", line 216, in > > > solve_eigensystem(DynMatrix_nn, Unity_nn, D_nn.shape, > opt_dict.LowLimit, opt_dict.UpperLimit) > > > File "Eigensolver_spectrum_slicing.py", line 121, in > solve_eigensystem > > > E.setUp() > > > File "SLEPc/EPS.pyx", line 1099, in slepc4py.SLEPc.EPS.setUp > > > petsc4py.PETSc.Error: error code 92 > > > [14] EPSSetUp() line 165 in > /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c > > > [14] EPSSetUp_KrylovSchur() line 146 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c > > > [14] EPSSetUp_KrylovSchur_Slice() line 410 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > > > [14] EPSSliceGetEPS() line 300 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > > > [14] EPSSetUp() line 165 in > /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c > > > [14] EPSSetUp_KrylovSchur() line 146 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c > > > [14] EPSSetUp_KrylovSchur_Slice() line 461 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > > > [14] EPSSliceGetInertia() line 331 in > /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > > > [14] STSetUp() line 271 in > /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/interface/stsolve.c > > > [14] STSetUp_Sinvert() line 132 in > /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/impls/sinvert/sinvert.c > > > [14] KSPSetUp() line 381 in > /tmp/pip-install-xmiaat2t/petsc/src/ksp/ksp/interface/itfunc.c > > > [14] PCSetUp() line 923 in > /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/interface/precon.c > > > [14] PCSetUp_Cholesky() line 86 in > /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c > > > [14] MatGetFactor() line 4318 in > /tmp/pip-install-xmiaat2t/petsc/src/mat/interface/matrix.c > > > [14] See > http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for > possible LU and Cholesky solvers > > > [14] Could not locate a solver package. Perhaps you must ./configure > with --download- > > > > > > The code i used to solve the problem is > > > > > > Am Fr., 14. Sep. 2018 um 18:34 Uhr schrieb Matthew Knepley < > knepley at gmail.com>: > > > On Fri, Sep 14, 2018 at 12:19 PM Jose E. Roman > wrote: > > > El 14 sept 2018, a las 17:45, Jan Grie?er > escribi?: > > > > > >> Hey there, > > >> first i want to say thanks to Satish and Matt for helping with with > my last problem with the mpi compilation. I have two questions related to > solving a big, hermitian, standard eigenvalue problem using SLEPc4py., > compiled with Intel MKL and Intel MPI. > > >> > > >> - I am using slepc4py with > > >> mpi and run it with around -n 20 cores at the moment and how i wanted > to ask if there is an easy way to retrieve the eigenvectors? When i run my > code and print for i in range(nconv): > > >> for i in range(nconv): > > >> > > >> val = E. > > >> getEigenpair(i, vr > > >> , vi) > > >> Print( > > >> vr.getArray()) > > >> i get the parts of the eigenvectors according to the partition of > the matrix. Is there any easy way to put them together in an array and > write them to file ? (I am struggling a little bit with the building them > in the correct order) > > > > > > You need VecScatterCreateToZero. There must be an equivalent in python. > > > > > > An alternative to this which you should consider, because it is > simpler, is to write the vector to a file > > > using some format that PETSc understands, Then you just need > vr.view(viewer) for a viewer like > > > the binary viewer or some ASCII format you like. > > > > > > Thanks, > > > > > > Matt > > >> - I need to solve eigenvalue problems up to a dimension of 100000 > degrees of freedom and i need all eigenvalues and eigenvectors. I think > solving all eigenvalues in one process is far too much and i thought about > if it is possible to apply the spectrum slicing described in Chap. 3.4.5. > Due to the nature of my problem, i am able to simulate smaller systems of > 10000 DOF and extract the biggest eigenvalue, which will be the same for > larger systems sizes. Is this in general possible since i have a standard > HEP problem or is there a better and faster possibility to do this? > > > > > > In general, SLEPc is not intended for computing the whole spectrum. > You can try with spectrum slicing but this will be competitive if computing > just a percentage of eigenvalues, 50% say. > > > > > > Jose > > > > > >> > > >> Thank you very much! > > > > > > > > > -- > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > -- Norbert Wiener > > > > > > https://www.cse.buffalo.edu/~knepley/ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From R.A.Astudillo at tudelft.nl Tue Sep 18 02:44:29 2018 From: R.A.Astudillo at tudelft.nl (Reinaldo Astudilo - EWI) Date: Tue, 18 Sep 2018 09:44:29 +0200 Subject: [petsc-users] About SLEPC's EPSSetDeflationSpace implementation Message-ID: <98299e32-b6e8-b849-40ec-77dcee9afe18@tudelft.nl> Dear Petsc/Slepc users, I'd like to know what algorithm SLEPC implements for deflation set by EPSSetDeflationSpace. Could anyone knows this or point out any reference? Thank you in advance Reinaldo From jroman at dsic.upv.es Tue Sep 18 02:55:29 2018 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 18 Sep 2018 09:55:29 +0200 Subject: [petsc-users] About SLEPC's EPSSetDeflationSpace implementation In-Reply-To: <98299e32-b6e8-b849-40ec-77dcee9afe18@tudelft.nl> References: <98299e32-b6e8-b849-40ec-77dcee9afe18@tudelft.nl> Message-ID: <0BA5C0EF-32AF-461C-92AD-A79B414BB435@dsic.upv.es> It is deflation by orthogonalization. The vectors provided with EPSSetDeflationSpace() are converted to an orthonormal basis (orthogonalizing column by column) and then all vectors generated by the solver are orthogonalized against this basis. In this way, the solver operates in the orthogonal complement of the deflation space. Jose > El 18 sept 2018, a las 9:44, Reinaldo Astudilo - EWI escribi?: > > Dear Petsc/Slepc users, > > I'd like to know what algorithm SLEPC implements for deflation set by EPSSetDeflationSpace. > > Could anyone knows this or point out any reference? > > Thank you in advance > > Reinaldo > From knepley at gmail.com Tue Sep 18 03:37:54 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 Sep 2018 04:37:54 -0400 Subject: [petsc-users] Unable to read in values thru namelist in Fortran after using PETSc 64bit in linux In-Reply-To: References: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com> <0AC1BA91-75F8-4AAB-A803-832B089A2C44@anl.gov> <22565566-590d-7c89-390a-99557ce05382@gmail.com> Message-ID: On Tue, Sep 18, 2018 at 2:39 AM TAY wee-beng wrote: > Hi, > > In that case, does it apply to all MPI subroutines such as MPI_ALLGATHER? > > In other words, must I assign local_array_length etc as PetscMPIInt? > No. MPI _data_ arrays can be any type. Here you have specified MPIU_INTEGER, which indicates PetscInt. If you had used MPI_INTEGER, then you would need a PetscMPIInt. Thanks, Matt > call > > MPI_ALLGATHER(local_array_length,1,MPIU_INTEGER,array_length,1,MPIU_INTEGER,MPI_COMM_WORLD,ierr) > > Or is it ok to change all integers from PetscInt to PetscMPIInt? > > With the exception of ierr - PetscErrorCode > > > Thank you very much. > > Yours sincerely, > > ================================================ > TAY Wee-Beng (Zheng Weiming) ??? > Personal research webpage: http://tayweebeng.wixsite.com/website > Youtube research showcase: > https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > linkedin: www.linkedin.com/in/tay-weebeng > ================================================ > > On 18/9/2018 1:39 PM, Balay, Satish wrote: > > https://www.mpich.org/static/docs/v3.1/www3/MPI_Comm_size.html > > > > int MPI_Comm_size( MPI_Comm comm, int *size ) > > > > i.e there is no PetscInt here. [MPI does not know about PETSc datatypes] > > > > For convinence we provide PetscMPIInt to keep track of such variables > > [similarly PetscBLASInt]. For eg: Check src/vec/vec/examples/tests/ex2f.F > > > > Satish > > > > On Tue, 18 Sep 2018, TAY wee-beng wrote: > > > >> Hi, > >> > >> I managed to find the error appearing after using PETSc 64bit in linux - > >> > >> call MPI_COMM_SIZE(MPI_COMM_WORLD, num_procs, ierr) > >> > >> I have assigned num_procs as PetscInt and I got 0 instead of 1 (for 1 > procs) > >> > >> Assigning num_procs as integer as the problem. > >> > >> Is this supposed to be the case? Or is it a bug? > >> > >> Thank you very much. > >> > >> Yours sincerely, > >> > >> ================================================ > >> TAY Wee-Beng (Zheng Weiming) ??? > >> Personal research webpage: http://tayweebeng.wixsite.com/website > >> Youtube research showcase: > >> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > >> linkedin: www.linkedin.com/in/tay-weebeng > >> ================================================ > >> > >> On 8/9/2018 1:14 AM, Smith, Barry F. wrote: > >>> You can try valgrind > >>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > >>> > >>> Barry > >>> > >>> > >>>> On Sep 7, 2018, at 1:44 AM, TAY wee-beng wrote: > >>>> > >>>> Hi, > >>>> > >>>> I found that I am unable to read in values thru namelist in Fortran > after > >>>> using PETSc 64bit in linux. > >>>> > >>>> I have a parameter txt file which is read in using namelist in > Fortran? > >>>> > >>>> namelist /body_input/ no_body, convex_body, motion_type, hover, > wing_config > >>>> ... > >>>> > >>>> open (unit = 44 , FILE = "ibm3d_input.txt" , status = "old", iostat = > >>>> openstatus(4)) > >>>> > >>>> if (openstatus(4) > 0) then > >>>> > >>>> print *, "ibm3d_input file not present or wrong > filename." > >>>> > >>>> stop > >>>> > >>>> end if > >>>> > >>>> read (44,nml = solver_input) > >>>> > >>>> read (44,nml = grid_input) > >>>> > >>>> read (44,nml = body_input)... > >>>> > >>>> > >>>> After using PETSc 64bit, my code aborts and I realise that it is > because > >>>> the values have became NaN. Strangely, it does not occur in windows > with > >>>> VS2008. > >>>> > >>>> I wonder if it's a bug with the Intel Fortran compiler 2018. > >>>> > >>>> Anyone has similar experience? > >>>> > >>>> -- > >>>> Thank you very much. > >>>> > >>>> Yours sincerely, > >>>> > >>>> ================================================ > >>>> TAY Wee-Beng (Zheng Weiming) ??? > >>>> Personal research webpage: http://tayweebeng.wixsite.com/website > >>>> Youtube research showcase: > >>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > >>>> linkedin: www.linkedin.com/in/tay-weebeng > >>>> ================================================ > >>>> > >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Tue Sep 18 08:36:57 2018 From: cpraveen at gmail.com (Praveen C) Date: Tue, 18 Sep 2018 19:06:57 +0530 Subject: [petsc-users] Problem with MPI_Allreduce Message-ID: <05970B6C-EE8A-4FA3-B5A2-AB53B01E80AE@gmail.com> Dear all I use MPI_Allreduce in my petsc code and when compiling I see this type of messages /home/soft/include/petsclog.h:376:57: warning: value computed is not used [-Wunused-value] ((petsc_allreduce_ct += PetscMPIParallelComm(comm),0) || MPI_Allreduce(sendbuf,recvbuf,count,datatype,op,comm)) ^ fdweno.c:564:4: note: in expansion of macro ?MPI_Allreduce? MPI_Allreduce(&lamx, &lambdax, 1, MPI_DOUBLE, MPI_MAX, PETSC_COMM_WORLD); ^~~~~~~~~~~~~ This is with petsc-3.8.3 on Linux. My code crashes due to this error I think, while I dont get this problem on my macbook which uses petsc-3.9.0 Since the error is pointing to petsclog.h I thought I will ask here. Thanks praveen From jed at jedbrown.org Tue Sep 18 09:42:24 2018 From: jed at jedbrown.org (Jed Brown) Date: Tue, 18 Sep 2018 08:42:24 -0600 Subject: [petsc-users] Problem with MPI_Allreduce In-Reply-To: <05970B6C-EE8A-4FA3-B5A2-AB53B01E80AE@gmail.com> References: <05970B6C-EE8A-4FA3-B5A2-AB53B01E80AE@gmail.com> Message-ID: <87sh26vpdb.fsf@jedbrown.org> Which compiler version are you using? This warning is harmless, but we should still clean up these macros to avoid the warning. Praveen C writes: > Dear all > > I use MPI_Allreduce in my petsc code and when compiling I see this type of messages > > /home/soft/include/petsclog.h:376:57: warning: value computed is not used [-Wunused-value] > ((petsc_allreduce_ct += PetscMPIParallelComm(comm),0) || MPI_Allreduce(sendbuf,recvbuf,count,datatype,op,comm)) > ^ > fdweno.c:564:4: note: in expansion of macro ?MPI_Allreduce? > MPI_Allreduce(&lamx, &lambdax, 1, MPI_DOUBLE, MPI_MAX, PETSC_COMM_WORLD); > ^~~~~~~~~~~~~ > > This is with petsc-3.8.3 on Linux. > > My code crashes due to this error I think, This is certainly not why the code crashes. > while I dont get this problem on my macbook which uses petsc-3.9.0 > > Since the error is pointing to petsclog.h I thought I will ask here. > > Thanks > praveen From balay at mcs.anl.gov Tue Sep 18 09:49:21 2018 From: balay at mcs.anl.gov (Balay, Satish) Date: Tue, 18 Sep 2018 14:49:21 +0000 Subject: [petsc-users] Problem with MPI_Allreduce In-Reply-To: <87sh26vpdb.fsf@jedbrown.org> References: <05970B6C-EE8A-4FA3-B5A2-AB53B01E80AE@gmail.com> <87sh26vpdb.fsf@jedbrown.org> Message-ID: This warning will go away if you change the code to: ierr = MPI_Allreduce(&lamx, &lambdax, 1, MPI_DOUBLE, MPI_MAX, PETSC_COMM_WORLD);CHKERRQ(ierr); And its best to run code in gdb or valgrind to track down the crash. Satish On Tue, 18 Sep 2018, Jed Brown wrote: > Which compiler version are you using? This warning is harmless, but we > should still clean up these macros to avoid the warning. > > Praveen C writes: > > > Dear all > > > > I use MPI_Allreduce in my petsc code and when compiling I see this type of messages > > > > /home/soft/include/petsclog.h:376:57: warning: value computed is not used [-Wunused-value] > > ((petsc_allreduce_ct += PetscMPIParallelComm(comm),0) || MPI_Allreduce(sendbuf,recvbuf,count,datatype,op,comm)) > > ^ > > fdweno.c:564:4: note: in expansion of macro ?MPI_Allreduce? > > MPI_Allreduce(&lamx, &lambdax, 1, MPI_DOUBLE, MPI_MAX, PETSC_COMM_WORLD); > > ^~~~~~~~~~~~~ > > > > This is with petsc-3.8.3 on Linux. > > > > My code crashes due to this error I think, > > This is certainly not why the code crashes. > > > while I dont get this problem on my macbook which uses petsc-3.9.0 > > > > Since the error is pointing to petsclog.h I thought I will ask here. > > > > Thanks > > praveen > From cpraveen at gmail.com Tue Sep 18 10:07:01 2018 From: cpraveen at gmail.com (Praveen C) Date: Tue, 18 Sep 2018 20:37:01 +0530 Subject: [petsc-users] Problem with MPI_Allreduce In-Reply-To: References: <05970B6C-EE8A-4FA3-B5A2-AB53B01E80AE@gmail.com> <87sh26vpdb.fsf@jedbrown.org> Message-ID: <362453B8-FE4C-414A-ACE6-5BD1394D2EBA@gmail.com> > On 18-Sep-2018, at 8:19 PM, Balay, Satish wrote: > > This warning will go away if you change the code to: > > ierr = MPI_Allreduce(&lamx, &lambdax, 1, MPI_DOUBLE, MPI_MAX, PETSC_COMM_WORLD);CHKERRQ(ierr); > > And its best to run code in gdb or valgrind to track down the crash. > > Satish Thank you, this fixed the warnings. There was some other bug in my code. Best praveen -------------- next part -------------- An HTML attachment was scrubbed... URL: From perceval.desforges at ens-lyon.fr Tue Sep 18 10:46:57 2018 From: perceval.desforges at ens-lyon.fr (Desforges Perceval) Date: Tue, 18 Sep 2018 17:46:57 +0200 Subject: [petsc-users] First shift renders no information In-Reply-To: <3B2AE31E-C317-4795-AB92-B36BDDFC274C@dsic.upv.es> References: <3B2AE31E-C317-4795-AB92-B36BDDFC274C@dsic.upv.es> Message-ID: Thank you very much, moving around the endpoints of the interval did indeed solve my problem. Best regards, --- Desforges Perceval Le 2018-09-17 17:51, Jose E. Roman a ?crit : > This can happen sometimes when the chosen interval for spectrum slicing is to large or the endpoints of the interval are too far away from the eigenvalues. Try moving the endpoints of the interval closer to the eigenvalues. > > Jose > >> El 17 sept 2018, a las 17:24, Desforges Perceval escribi?: >> >> Hello PETSc developpers, >> >> I have been using SLEPc and PETSc for solving my sparse systems for a small time now, and have been very satisfied with everything. >> >> However, I have been running into an error lately, and I have been unable to find any online resource that talks about this specific error. >> >> I use SLEPc to find all the eigenvalues of certain tridiagonal and symmetric matrices. I set the solver to find all the eigenvalues in a large interval ([0;10] for example) using the spectrum splicing technique. I use MUMPS because I would like to perform this calculation in parallel later on, but for now I am doing it sequentially. >> >> While the solver works very well for small enough matrices (n=2000), when I increase the matrix size to around 10000, I get this error that I do not really understand: >> >> [0]PETSC ERROR: First shift renders no information >> >> Could someone please enlighten me as to what this error means? I don't know if it's relevant but I am using older version of PETSc (3.8.4) and SLEPc (3.8.2). >> >> I apologize in advance if this question has been answered somewhere already. >> >> Thank you very much, >> >> Sincerely, >> >> Desforges Perceval -------------- next part -------------- An HTML attachment was scrubbed... URL: From griesser.jan at googlemail.com Tue Sep 18 10:53:33 2018 From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=) Date: Tue, 18 Sep 2018 17:53:33 +0200 Subject: [petsc-users] Retireve eigenvectors from a paralell job/ Spectrum slicing in order to solve big eigenvalue problem In-Reply-To: References: <96EE281A-3EC9-4CD7-884B-1AC45D1A6A19@dsic.upv.es> Message-ID: But i'am still struggling a little bit with copying the vector entries from the processes to one sequentiel proces. I tried to use the comment with Scatter().toZero() in Python and tried the following code: vecs = np.zeros((69999,nconv)) for i in range(nconv): val = E.getEigenpair(i, vr, vi) scatter, vec_full = PETSc.Scatter().toZero(vr) scatter.scatter(vr, vec_full, False, PETSc.ScatterMode.FORWARD) if rank == 0: vecs[:,i] = vec_full.getArray() but when i take a look at the eigenvectors there are some entries which are strange, because in this exampled i printed 4 eigenvalues and vectors and get nconv=4, but when i print the vectors i get lines with 4 entries but they are all zero and also get lines with strange entries e.g. 6.386079897185445800e-05 3.675310683411162830e-04 9.068495707804999221e-06 7.43627600.000000000000000000e+00 0.000000000000000000e+00 0.000000000000000000e+00 0.000000000000000000e+00 where i have more than 4 entries. Can somebody explain to me why this happens or what is wrong with the code. (Sorry for asking such beginner questions, but i just started to work with PETSc and SLEPc) Am Di., 18. Sep. 2018 um 09:26 Uhr schrieb Jan Grie?er < griesser.jan at googlemail.com>: > Now it works. Thank you very much! > > Am Mo., 17. Sep. 2018 um 10:54 Uhr schrieb Jose E. Roman < > jroman at dsic.upv.es>: > >> For this you have to set a number of partitions equal to the number of >> MPI processes. If you get this error it is because these values are >> different. >> Jose >> >> >> > El 17 sept 2018, a las 10:50, Jan Grie?er >> escribi?: >> > >> > Is this relly necessary, because in the last sentences of the chapter >> it states that: >> > An additional benefit of multi-communicator support is that it enables >> parallel spectrum slicing runs without the need to install a parallel >> direct solver (MUMPS). The following commandline example uses sequential >> linear solves in 4 partitions, one process each: >> > Therefore i assumed that it is not necessary to compile PETsc4py with >> an external solver e.g. MUMPS >> > >> > Am Mo., 17. Sep. 2018 um 10:47 Uhr schrieb Jose E. Roman < >> jroman at dsic.upv.es>: >> > You need a parallel direct solver such as MUMPS. This is explained in >> section 3.4.5. >> > Jose >> > >> > >> > > El 17 sept 2018, a las 10:41, Jan Grie?er < >> griesser.jan at googlemail.com> escribi?: >> > > >> > > def solve_eigensystem(DynMatrix_nn, Unity_nn, Dimension, LowerLimit, >> UpperLimit): >> > > # Create the EPS solver >> > > E = SLEPc.EPS().create() >> > > >> > > # Create the preconditioner and set it to Cholesky >> > > pc = PETSc.PC().create() >> > > pc.setType(pc.Type.CHOLESKY) >> > > >> > > # Create the KSP object >> > > ksp = PETSc.KSP().create() >> > > ksp.setType(ksp.Type.PREONLY) >> > > ksp.setPC(pc) >> > > >> > > # Set up the spectral transformations >> > > st = SLEPc.ST().create() >> > > st.setType("sinvert") >> > > st.setKSP(ksp) >> > > # Setup spectral transformation >> > > E.setST(st) >> > > >> > > # Eigenvalues should be real, therefore we start to order them >> from the smallest real value |l.real| >> > > E.setWhichEigenpairs(E.Which.ALL) >> > > # Set the interval of spectrum slicing >> > > E.setInterval(LowerLimit, UpperLimit) >> > > # Since the dynamical matrix is symmetric and real it is >> hermitian. Use GHEP for the spectrum slicing. Operatormatrix B is just a >> unit matrix >> > > E.setProblemType(SLEPc.EPS.ProblemType.GHEP) >> > > # Use the Krylov Schur method to solve the eigenvalue problem >> > > E.setType(E.Type.KRYLOVSCHUR) >> > > # Partition the Krylov schnur problem in npart procceses >> > > E.setKrylovSchurPartitions(10) >> > > # Set the convergence criterion to relative to the eigenvalue >> and the maximal number of iterations >> > > E.setConvergenceTest(E.Conv.REL) >> > > E.setTolerances(tol = 1e-7, max_it = 1000) >> > > # Set the matrix in order to solve >> > > E.setOperators(DynMatrix_nn, Unity_nn) >> > > # Sets EPS options from the options database. >> > > E.setFromOptions() >> > > # Sets up all the internal data structures necessary for the >> execution of the eigensolver. >> > > E.setUp() >> > > >> > > # Solve eigenvalue problem >> > > startClock = time.clock() >> > > startTime = time.time() >> > > E.solve() >> > > >> > > Has maybe one of you any idea why this happens and where the problem >> is ? >> > > >> > > Am Mo., 17. Sep. 2018 um 10:40 Uhr schrieb Jan Grie?er < >> griesser.jan at googlemail.com>: >> > > I am aware that SLEPc is not supposed to calculate all eigenvalues >> and eigenvectors, my problem is simply that i want for a physical large >> enough system all of them before i can make the transition to go to the >> smallest ones. >> > > Competitiveness is of secondary importance at the moment. >> > > But ihave a problem connected with spectrum slicing. I followed the >> instructions in the manual of Chap. 3.4.5 Spectrum Slicing and converted >> them to the python package. >> > > But now i get the following error. It appears to me that it is not >> able to find the ksp object, but i actually do not know why this is the >> case. >> > > aceback (most recent call last): >> > > File "Eigensolver_spectrum_slicing.py", line 216, in >> > > solve_eigensystem(DynMatrix_nn, Unity_nn, D_nn.shape, >> opt_dict.LowLimit, opt_dict.UpperLimit) >> > > File "Eigensolver_spectrum_slicing.py", line 121, in >> solve_eigensystem >> > > E.setUp() >> > > File "SLEPc/EPS.pyx", line 1099, in slepc4py.SLEPc.EPS.setUp >> > > petsc4py.PETSc.Error: error code 92 >> > > [14] EPSSetUp() line 165 in >> /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c >> > > [14] EPSSetUp_KrylovSchur() line 146 in >> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c >> > > [14] EPSSetUp_KrylovSchur_Slice() line 410 in >> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >> > > [14] EPSSliceGetEPS() line 300 in >> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >> > > [14] EPSSetUp() line 165 in >> /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c >> > > [14] EPSSetUp_KrylovSchur() line 146 in >> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c >> > > [14] EPSSetUp_KrylovSchur_Slice() line 461 in >> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >> > > [14] EPSSliceGetInertia() line 331 in >> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >> > > [14] STSetUp() line 271 in >> /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/interface/stsolve.c >> > > [14] STSetUp_Sinvert() line 132 in >> /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/impls/sinvert/sinvert.c >> > > [14] KSPSetUp() line 381 in >> /tmp/pip-install-xmiaat2t/petsc/src/ksp/ksp/interface/itfunc.c >> > > [14] PCSetUp() line 923 in >> /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/interface/precon.c >> > > [14] PCSetUp_Cholesky() line 86 in >> /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c >> > > [14] MatGetFactor() line 4318 in >> /tmp/pip-install-xmiaat2t/petsc/src/mat/interface/matrix.c >> > > [14] See >> http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for >> possible LU and Cholesky solvers >> > > [14] Could not locate a solver package. Perhaps you must ./configure >> with --download- >> > > >> > > The code i used to solve the problem is >> > > >> > > Am Fr., 14. Sep. 2018 um 18:34 Uhr schrieb Matthew Knepley < >> knepley at gmail.com>: >> > > On Fri, Sep 14, 2018 at 12:19 PM Jose E. Roman >> wrote: >> > > El 14 sept 2018, a las 17:45, Jan Grie?er < >> griesser.jan at googlemail.com> escribi?: >> > > >> > >> Hey there, >> > >> first i want to say thanks to Satish and Matt for helping with with >> my last problem with the mpi compilation. I have two questions related to >> solving a big, hermitian, standard eigenvalue problem using SLEPc4py., >> compiled with Intel MKL and Intel MPI. >> > >> >> > >> - I am using slepc4py with >> > >> mpi and run it with around -n 20 cores at the moment and how i >> wanted to ask if there is an easy way to retrieve the eigenvectors? When i >> run my code and print for i in range(nconv): >> > >> for i in range(nconv): >> > >> >> > >> val = E. >> > >> getEigenpair(i, vr >> > >> , vi) >> > >> Print( >> > >> vr.getArray()) >> > >> i get the parts of the eigenvectors according to the partition of >> the matrix. Is there any easy way to put them together in an array and >> write them to file ? (I am struggling a little bit with the building them >> in the correct order) >> > > >> > > You need VecScatterCreateToZero. There must be an equivalent in >> python. >> > > >> > > An alternative to this which you should consider, because it is >> simpler, is to write the vector to a file >> > > using some format that PETSc understands, Then you just need >> vr.view(viewer) for a viewer like >> > > the binary viewer or some ASCII format you like. >> > > >> > > Thanks, >> > > >> > > Matt >> > >> - I need to solve eigenvalue problems up to a dimension of 100000 >> degrees of freedom and i need all eigenvalues and eigenvectors. I think >> solving all eigenvalues in one process is far too much and i thought about >> if it is possible to apply the spectrum slicing described in Chap. 3.4.5. >> Due to the nature of my problem, i am able to simulate smaller systems of >> 10000 DOF and extract the biggest eigenvalue, which will be the same for >> larger systems sizes. Is this in general possible since i have a standard >> HEP problem or is there a better and faster possibility to do this? >> > > >> > > In general, SLEPc is not intended for computing the whole spectrum. >> You can try with spectrum slicing but this will be competitive if computing >> just a percentage of eigenvalues, 50% say. >> > > >> > > Jose >> > > >> > >> >> > >> Thank you very much! >> > > >> > > >> > > -- >> > > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > > -- Norbert Wiener >> > > >> > > https://www.cse.buffalo.edu/~knepley/ >> > >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From shash_91 at hotmail.com Tue Sep 18 11:09:28 2018 From: shash_91 at hotmail.com (Shashwat Sharma) Date: Tue, 18 Sep 2018 16:09:28 +0000 Subject: [petsc-users] SuperLU_Dist - set options from within code Message-ID: Hello, I'm using Petsc with SuperLU_Dist, and I need to run it with the option -mat_superlu_dist_replacetinypivot. I'd like to be able to set this option (or other SuperLu_Dist options) directly from within my code, rather than the command line during run-time. Is there a way to do this? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 18 11:19:16 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 Sep 2018 12:19:16 -0400 Subject: [petsc-users] Retireve eigenvectors from a paralell job/ Spectrum slicing in order to solve big eigenvalue problem In-Reply-To: References: <96EE281A-3EC9-4CD7-884B-1AC45D1A6A19@dsic.upv.es> Message-ID: On Tue, Sep 18, 2018 at 11:53 AM Jan Grie?er wrote: > But i'am still struggling a little bit with copying the vector entries > from the processes to one sequentiel proces. I tried to use the comment > with Scatter().toZero() in Python and tried the following code: > vecs = np.zeros((69999,nconv)) > for i in range(nconv): > val = E.getEigenpair(i, vr, vi) > scatter, vec_full = PETSc.Scatter().toZero(vr) > scatter.scatter(vr, vec_full, False, PETSc.ScatterMode.FORWARD) > if rank == 0: > vecs[:,i] = vec_full.getArray() > but when i take a look at the eigenvectors there are some entries which > are strange, because in this exampled i printed 4 eigenvalues and vectors > and get nconv=4, but when i print the vectors i get lines > with 4 entries but they are all zero and also get lines with strange > entries e.g. > 6.386079897185445800e-05 3.675310683411162830e-04 9.068495707804999221e-06 > 7.43627600.000000000000000000e+00 0.000000000000000000e+00 > 0.000000000000000000e+00 0.000000000000000000e+00 > where i have more than 4 entries. Can somebody explain to me why this > happens or what is wrong with the code. (Sorry for asking such beginner > questions, but i just started to work with PETSc and SLEPc) > We can definitely figure that out. However, can you try vr.view(PETSC_VIEWER_STDOUT_WORLD) just to check your vectors initially? Thanks, Matt > Am Di., 18. Sep. 2018 um 09:26 Uhr schrieb Jan Grie?er < > griesser.jan at googlemail.com>: > >> Now it works. Thank you very much! >> >> Am Mo., 17. Sep. 2018 um 10:54 Uhr schrieb Jose E. Roman < >> jroman at dsic.upv.es>: >> >>> For this you have to set a number of partitions equal to the number of >>> MPI processes. If you get this error it is because these values are >>> different. >>> Jose >>> >>> >>> > El 17 sept 2018, a las 10:50, Jan Grie?er >>> escribi?: >>> > >>> > Is this relly necessary, because in the last sentences of the chapter >>> it states that: >>> > An additional benefit of multi-communicator support is that it enables >>> parallel spectrum slicing runs without the need to install a parallel >>> direct solver (MUMPS). The following commandline example uses sequential >>> linear solves in 4 partitions, one process each: >>> > Therefore i assumed that it is not necessary to compile PETsc4py with >>> an external solver e.g. MUMPS >>> > >>> > Am Mo., 17. Sep. 2018 um 10:47 Uhr schrieb Jose E. Roman < >>> jroman at dsic.upv.es>: >>> > You need a parallel direct solver such as MUMPS. This is explained in >>> section 3.4.5. >>> > Jose >>> > >>> > >>> > > El 17 sept 2018, a las 10:41, Jan Grie?er < >>> griesser.jan at googlemail.com> escribi?: >>> > > >>> > > def solve_eigensystem(DynMatrix_nn, Unity_nn, Dimension, LowerLimit, >>> UpperLimit): >>> > > # Create the EPS solver >>> > > E = SLEPc.EPS().create() >>> > > >>> > > # Create the preconditioner and set it to Cholesky >>> > > pc = PETSc.PC().create() >>> > > pc.setType(pc.Type.CHOLESKY) >>> > > >>> > > # Create the KSP object >>> > > ksp = PETSc.KSP().create() >>> > > ksp.setType(ksp.Type.PREONLY) >>> > > ksp.setPC(pc) >>> > > >>> > > # Set up the spectral transformations >>> > > st = SLEPc.ST().create() >>> > > st.setType("sinvert") >>> > > st.setKSP(ksp) >>> > > # Setup spectral transformation >>> > > E.setST(st) >>> > > >>> > > # Eigenvalues should be real, therefore we start to order them >>> from the smallest real value |l.real| >>> > > E.setWhichEigenpairs(E.Which.ALL) >>> > > # Set the interval of spectrum slicing >>> > > E.setInterval(LowerLimit, UpperLimit) >>> > > # Since the dynamical matrix is symmetric and real it is >>> hermitian. Use GHEP for the spectrum slicing. Operatormatrix B is just a >>> unit matrix >>> > > E.setProblemType(SLEPc.EPS.ProblemType.GHEP) >>> > > # Use the Krylov Schur method to solve the eigenvalue problem >>> > > E.setType(E.Type.KRYLOVSCHUR) >>> > > # Partition the Krylov schnur problem in npart procceses >>> > > E.setKrylovSchurPartitions(10) >>> > > # Set the convergence criterion to relative to the eigenvalue >>> and the maximal number of iterations >>> > > E.setConvergenceTest(E.Conv.REL) >>> > > E.setTolerances(tol = 1e-7, max_it = 1000) >>> > > # Set the matrix in order to solve >>> > > E.setOperators(DynMatrix_nn, Unity_nn) >>> > > # Sets EPS options from the options database. >>> > > E.setFromOptions() >>> > > # Sets up all the internal data structures necessary for the >>> execution of the eigensolver. >>> > > E.setUp() >>> > > >>> > > # Solve eigenvalue problem >>> > > startClock = time.clock() >>> > > startTime = time.time() >>> > > E.solve() >>> > > >>> > > Has maybe one of you any idea why this happens and where the problem >>> is ? >>> > > >>> > > Am Mo., 17. Sep. 2018 um 10:40 Uhr schrieb Jan Grie?er < >>> griesser.jan at googlemail.com>: >>> > > I am aware that SLEPc is not supposed to calculate all eigenvalues >>> and eigenvectors, my problem is simply that i want for a physical large >>> enough system all of them before i can make the transition to go to the >>> smallest ones. >>> > > Competitiveness is of secondary importance at the moment. >>> > > But ihave a problem connected with spectrum slicing. I followed the >>> instructions in the manual of Chap. 3.4.5 Spectrum Slicing and converted >>> them to the python package. >>> > > But now i get the following error. It appears to me that it is not >>> able to find the ksp object, but i actually do not know why this is the >>> case. >>> > > aceback (most recent call last): >>> > > File "Eigensolver_spectrum_slicing.py", line 216, in >>> > > solve_eigensystem(DynMatrix_nn, Unity_nn, D_nn.shape, >>> opt_dict.LowLimit, opt_dict.UpperLimit) >>> > > File "Eigensolver_spectrum_slicing.py", line 121, in >>> solve_eigensystem >>> > > E.setUp() >>> > > File "SLEPc/EPS.pyx", line 1099, in slepc4py.SLEPc.EPS.setUp >>> > > petsc4py.PETSc.Error: error code 92 >>> > > [14] EPSSetUp() line 165 in >>> /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c >>> > > [14] EPSSetUp_KrylovSchur() line 146 in >>> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c >>> > > [14] EPSSetUp_KrylovSchur_Slice() line 410 in >>> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>> > > [14] EPSSliceGetEPS() line 300 in >>> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>> > > [14] EPSSetUp() line 165 in >>> /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c >>> > > [14] EPSSetUp_KrylovSchur() line 146 in >>> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c >>> > > [14] EPSSetUp_KrylovSchur_Slice() line 461 in >>> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>> > > [14] EPSSliceGetInertia() line 331 in >>> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>> > > [14] STSetUp() line 271 in >>> /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/interface/stsolve.c >>> > > [14] STSetUp_Sinvert() line 132 in >>> /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/impls/sinvert/sinvert.c >>> > > [14] KSPSetUp() line 381 in >>> /tmp/pip-install-xmiaat2t/petsc/src/ksp/ksp/interface/itfunc.c >>> > > [14] PCSetUp() line 923 in >>> /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/interface/precon.c >>> > > [14] PCSetUp_Cholesky() line 86 in >>> /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c >>> > > [14] MatGetFactor() line 4318 in >>> /tmp/pip-install-xmiaat2t/petsc/src/mat/interface/matrix.c >>> > > [14] See >>> http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for >>> possible LU and Cholesky solvers >>> > > [14] Could not locate a solver package. Perhaps you must ./configure >>> with --download- >>> > > >>> > > The code i used to solve the problem is >>> > > >>> > > Am Fr., 14. Sep. 2018 um 18:34 Uhr schrieb Matthew Knepley < >>> knepley at gmail.com>: >>> > > On Fri, Sep 14, 2018 at 12:19 PM Jose E. Roman >>> wrote: >>> > > El 14 sept 2018, a las 17:45, Jan Grie?er < >>> griesser.jan at googlemail.com> escribi?: >>> > > >>> > >> Hey there, >>> > >> first i want to say thanks to Satish and Matt for helping with with >>> my last problem with the mpi compilation. I have two questions related to >>> solving a big, hermitian, standard eigenvalue problem using SLEPc4py., >>> compiled with Intel MKL and Intel MPI. >>> > >> >>> > >> - I am using slepc4py with >>> > >> mpi and run it with around -n 20 cores at the moment and how i >>> wanted to ask if there is an easy way to retrieve the eigenvectors? When i >>> run my code and print for i in range(nconv): >>> > >> for i in range(nconv): >>> > >> >>> > >> val = E. >>> > >> getEigenpair(i, vr >>> > >> , vi) >>> > >> Print( >>> > >> vr.getArray()) >>> > >> i get the parts of the eigenvectors according to the partition of >>> the matrix. Is there any easy way to put them together in an array and >>> write them to file ? (I am struggling a little bit with the building them >>> in the correct order) >>> > > >>> > > You need VecScatterCreateToZero. There must be an equivalent in >>> python. >>> > > >>> > > An alternative to this which you should consider, because it is >>> simpler, is to write the vector to a file >>> > > using some format that PETSc understands, Then you just need >>> vr.view(viewer) for a viewer like >>> > > the binary viewer or some ASCII format you like. >>> > > >>> > > Thanks, >>> > > >>> > > Matt >>> > >> - I need to solve eigenvalue problems up to a dimension of 100000 >>> degrees of freedom and i need all eigenvalues and eigenvectors. I think >>> solving all eigenvalues in one process is far too much and i thought about >>> if it is possible to apply the spectrum slicing described in Chap. 3.4.5. >>> Due to the nature of my problem, i am able to simulate smaller systems of >>> 10000 DOF and extract the biggest eigenvalue, which will be the same for >>> larger systems sizes. Is this in general possible since i have a standard >>> HEP problem or is there a better and faster possibility to do this? >>> > > >>> > > In general, SLEPc is not intended for computing the whole spectrum. >>> You can try with spectrum slicing but this will be competitive if computing >>> just a percentage of eigenvalues, 50% say. >>> > > >>> > > Jose >>> > > >>> > >> >>> > >> Thank you very much! >>> > > >>> > > >>> > > -- >>> > > What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> > > -- Norbert Wiener >>> > > >>> > > https://www.cse.buffalo.edu/~knepley/ >>> > >>> >>> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 18 11:21:46 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 Sep 2018 12:21:46 -0400 Subject: [petsc-users] SuperLU_Dist - set options from within code In-Reply-To: References: Message-ID: On Tue, Sep 18, 2018 at 12:12 PM Shashwat Sharma wrote: > Hello, > > I'm using Petsc with SuperLU_Dist, and I need to run it with the > option -mat_superlu_dist_replacetinypivot. I'd like to be able to set this > option (or other SuperLu_Dist options) directly from within my code, rather > than the command line during run-time. Is there a way to do this? > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscOptionsSetValue.html Thanks, Matt > Thanks! > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Sep 18 11:49:40 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 18 Sep 2018 16:49:40 +0000 Subject: [petsc-users] Unable to read in values thru namelist in Fortran after using PETSc 64bit in linux In-Reply-To: References: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com> <0AC1BA91-75F8-4AAB-A803-832B089A2C44@anl.gov> <22565566-590d-7c89-390a-99557ce05382@gmail.com> Message-ID: PetscMPIInt (or integer) are for all lengths passed to MPI functions; simple look at the prototypes for the MPI function you care about and it will tell you which arguments are integer. The DATA you are passing into the MPI arrays (which are labeled void * in the manual pages) should be PetscInt. > On Sep 18, 2018, at 1:39 AM, TAY wee-beng wrote: > > Hi, > > In that case, does it apply to all MPI subroutines such as MPI_ALLGATHER? > > In other words, must I assign local_array_length etc as PetscMPIInt? > > call MPI_ALLGATHER(local_array_length,1,MPIU_INTEGER,array_length,1,MPIU_INTEGER,MPI_COMM_WORLD,ierr) > > Or is it ok to change all integers from PetscInt to PetscMPIInt? > > With the exception of ierr - PetscErrorCode > > > Thank you very much. > > Yours sincerely, > > ================================================ > TAY Wee-Beng (Zheng Weiming) ??? > Personal research webpage: http://tayweebeng.wixsite.com/website > Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > linkedin: www.linkedin.com/in/tay-weebeng > ================================================ > > On 18/9/2018 1:39 PM, Balay, Satish wrote: >> https://www.mpich.org/static/docs/v3.1/www3/MPI_Comm_size.html >> >> int MPI_Comm_size( MPI_Comm comm, int *size ) >> >> i.e there is no PetscInt here. [MPI does not know about PETSc datatypes] >> >> For convinence we provide PetscMPIInt to keep track of such variables >> [similarly PetscBLASInt]. For eg: Check src/vec/vec/examples/tests/ex2f.F >> >> Satish >> >> On Tue, 18 Sep 2018, TAY wee-beng wrote: >> >>> Hi, >>> >>> I managed to find the error appearing after using PETSc 64bit in linux - >>> >>> call MPI_COMM_SIZE(MPI_COMM_WORLD, num_procs, ierr) >>> >>> I have assigned num_procs as PetscInt and I got 0 instead of 1 (for 1 procs) >>> >>> Assigning num_procs as integer as the problem. >>> >>> Is this supposed to be the case? Or is it a bug? >>> >>> Thank you very much. >>> >>> Yours sincerely, >>> >>> ================================================ >>> TAY Wee-Beng (Zheng Weiming) ??? >>> Personal research webpage: http://tayweebeng.wixsite.com/website >>> Youtube research showcase: >>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>> linkedin: www.linkedin.com/in/tay-weebeng >>> ================================================ >>> >>> On 8/9/2018 1:14 AM, Smith, Barry F. wrote: >>>> You can try valgrind >>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>> >>>> Barry >>>> >>>> >>>>> On Sep 7, 2018, at 1:44 AM, TAY wee-beng wrote: >>>>> >>>>> Hi, >>>>> >>>>> I found that I am unable to read in values thru namelist in Fortran after >>>>> using PETSc 64bit in linux. >>>>> >>>>> I have a parameter txt file which is read in using namelist in Fortran? >>>>> >>>>> namelist /body_input/ no_body, convex_body, motion_type, hover, wing_config >>>>> ... >>>>> >>>>> open (unit = 44 , FILE = "ibm3d_input.txt" , status = "old", iostat = >>>>> openstatus(4)) >>>>> >>>>> if (openstatus(4) > 0) then >>>>> >>>>> print *, "ibm3d_input file not present or wrong filename." >>>>> >>>>> stop >>>>> >>>>> end if >>>>> >>>>> read (44,nml = solver_input) >>>>> >>>>> read (44,nml = grid_input) >>>>> >>>>> read (44,nml = body_input)... >>>>> >>>>> >>>>> After using PETSc 64bit, my code aborts and I realise that it is because >>>>> the values have became NaN. Strangely, it does not occur in windows with >>>>> VS2008. >>>>> >>>>> I wonder if it's a bug with the Intel Fortran compiler 2018. >>>>> >>>>> Anyone has similar experience? >>>>> >>>>> -- >>>>> Thank you very much. >>>>> >>>>> Yours sincerely, >>>>> >>>>> ================================================ >>>>> TAY Wee-Beng (Zheng Weiming) ??? >>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>> Youtube research showcase: >>>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>> ================================================ >>>>> >>> > From griesser.jan at googlemail.com Wed Sep 19 06:57:04 2018 From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=) Date: Wed, 19 Sep 2018 13:57:04 +0200 Subject: [petsc-users] Retireve eigenvectors from a paralell job/ Spectrum slicing in order to solve big eigenvalue problem In-Reply-To: References: <96EE281A-3EC9-4CD7-884B-1AC45D1A6A19@dsic.upv.es> Message-ID: Okay i think i found my error. I now used this code snippet: if nconv > 0: # Create the results vectors vr, wr = DynMatrix_nn.getVecs() vi, wi = DynMatrix_nn.getVecs() # Array for the eigenvectos vecs = np.zeros((69999,nconv)) print("Shape of output array: ", vecs.shape) Print() Print(" k ||Ax-kx||/||kx|| ") Print("-----------------------------------------") for i in range(nconv): k = E.getEigenpair(i, vr, vi) error = E.computeError(i) if k.imag != 0.0: Print("%9f%+9f j %12g" % (k.real, k.imag, error)) else: Print("%12f %12g" %(k.real,error)) # Save all eigenvalues to one array eigenvals_n[i,0] = E.getEigenvalue(i).real eigenvals_n[i,1] = error # Insert eigenvectors scatter, vec_full = PETSc.Scatter().toZero(vr) scatter.scatter(vr, vec_full, PETSc.InsertMode.INSERT, PETSc.ScatterMode.FORWARD) if rank == 0: vecs[:,i] = vec_full.getArray() E.view() # Save eigenvalues to file if rank == 0: np.savetxt("Eigenvalues.txt", eigenvals_n, header="Eigenvalues, ascending ordering due to real part") np.savetxt("Eigenvec2.txt", vecs, header="Eigenvalues, ascending ordering due to real part") The problem was that all MPI processes create the array vecs. The scattering worked fine but in the end every MPI process wrote the eigenvec2.txt file and not only rank0. So i assume that 19 processes writing an empty array and 1 process writing the correct one, was leading to this strange kind of .txt file. Now only rank0 with the correct eigenvectors write the array to file. I still have to check if the rest is correct but i appears to me that it now works and it makes sense. Thank you very much for your help Matt!! Am Di., 18. Sep. 2018 um 18:19 Uhr schrieb Matthew Knepley < knepley at gmail.com>: > On Tue, Sep 18, 2018 at 11:53 AM Jan Grie?er > wrote: > >> But i'am still struggling a little bit with copying the vector entries >> from the processes to one sequentiel proces. I tried to use the comment >> with Scatter().toZero() in Python and tried the following code: >> vecs = np.zeros((69999,nconv)) >> for i in range(nconv): >> val = E.getEigenpair(i, vr, vi) >> scatter, vec_full = PETSc.Scatter().toZero(vr) >> scatter.scatter(vr, vec_full, False, PETSc.ScatterMode.FORWARD) >> if rank == 0: >> vecs[:,i] = vec_full.getArray() >> but when i take a look at the eigenvectors there are some entries which >> are strange, because in this exampled i printed 4 eigenvalues and vectors >> and get nconv=4, but when i print the vectors i get lines >> with 4 entries but they are all zero and also get lines with strange >> entries e.g. >> 6.386079897185445800e-05 3.675310683411162830e-04 >> 9.068495707804999221e-06 7.43627600.000000000000000000e+00 >> 0.000000000000000000e+00 0.000000000000000000e+00 0.000000000000000000e+00 >> where i have more than 4 entries. Can somebody explain to me why this >> happens or what is wrong with the code. (Sorry for asking such beginner >> questions, but i just started to work with PETSc and SLEPc) >> > > We can definitely figure that out. However, can you try > > vr.view(PETSC_VIEWER_STDOUT_WORLD) > > just to check your vectors initially? > > Thanks, > > Matt > > >> Am Di., 18. Sep. 2018 um 09:26 Uhr schrieb Jan Grie?er < >> griesser.jan at googlemail.com>: >> >>> Now it works. Thank you very much! >>> >>> Am Mo., 17. Sep. 2018 um 10:54 Uhr schrieb Jose E. Roman < >>> jroman at dsic.upv.es>: >>> >>>> For this you have to set a number of partitions equal to the number of >>>> MPI processes. If you get this error it is because these values are >>>> different. >>>> Jose >>>> >>>> >>>> > El 17 sept 2018, a las 10:50, Jan Grie?er < >>>> griesser.jan at googlemail.com> escribi?: >>>> > >>>> > Is this relly necessary, because in the last sentences of the chapter >>>> it states that: >>>> > An additional benefit of multi-communicator support is that it >>>> enables parallel spectrum slicing runs without the need to install a >>>> parallel direct solver (MUMPS). The following commandline example uses >>>> sequential linear solves in 4 partitions, one process each: >>>> > Therefore i assumed that it is not necessary to compile PETsc4py with >>>> an external solver e.g. MUMPS >>>> > >>>> > Am Mo., 17. Sep. 2018 um 10:47 Uhr schrieb Jose E. Roman < >>>> jroman at dsic.upv.es>: >>>> > You need a parallel direct solver such as MUMPS. This is explained in >>>> section 3.4.5. >>>> > Jose >>>> > >>>> > >>>> > > El 17 sept 2018, a las 10:41, Jan Grie?er < >>>> griesser.jan at googlemail.com> escribi?: >>>> > > >>>> > > def solve_eigensystem(DynMatrix_nn, Unity_nn, Dimension, >>>> LowerLimit, UpperLimit): >>>> > > # Create the EPS solver >>>> > > E = SLEPc.EPS().create() >>>> > > >>>> > > # Create the preconditioner and set it to Cholesky >>>> > > pc = PETSc.PC().create() >>>> > > pc.setType(pc.Type.CHOLESKY) >>>> > > >>>> > > # Create the KSP object >>>> > > ksp = PETSc.KSP().create() >>>> > > ksp.setType(ksp.Type.PREONLY) >>>> > > ksp.setPC(pc) >>>> > > >>>> > > # Set up the spectral transformations >>>> > > st = SLEPc.ST().create() >>>> > > st.setType("sinvert") >>>> > > st.setKSP(ksp) >>>> > > # Setup spectral transformation >>>> > > E.setST(st) >>>> > > >>>> > > # Eigenvalues should be real, therefore we start to order >>>> them from the smallest real value |l.real| >>>> > > E.setWhichEigenpairs(E.Which.ALL) >>>> > > # Set the interval of spectrum slicing >>>> > > E.setInterval(LowerLimit, UpperLimit) >>>> > > # Since the dynamical matrix is symmetric and real it is >>>> hermitian. Use GHEP for the spectrum slicing. Operatormatrix B is just a >>>> unit matrix >>>> > > E.setProblemType(SLEPc.EPS.ProblemType.GHEP) >>>> > > # Use the Krylov Schur method to solve the eigenvalue problem >>>> > > E.setType(E.Type.KRYLOVSCHUR) >>>> > > # Partition the Krylov schnur problem in npart procceses >>>> > > E.setKrylovSchurPartitions(10) >>>> > > # Set the convergence criterion to relative to the eigenvalue >>>> and the maximal number of iterations >>>> > > E.setConvergenceTest(E.Conv.REL) >>>> > > E.setTolerances(tol = 1e-7, max_it = 1000) >>>> > > # Set the matrix in order to solve >>>> > > E.setOperators(DynMatrix_nn, Unity_nn) >>>> > > # Sets EPS options from the options database. >>>> > > E.setFromOptions() >>>> > > # Sets up all the internal data structures necessary for the >>>> execution of the eigensolver. >>>> > > E.setUp() >>>> > > >>>> > > # Solve eigenvalue problem >>>> > > startClock = time.clock() >>>> > > startTime = time.time() >>>> > > E.solve() >>>> > > >>>> > > Has maybe one of you any idea why this happens and where the >>>> problem is ? >>>> > > >>>> > > Am Mo., 17. Sep. 2018 um 10:40 Uhr schrieb Jan Grie?er < >>>> griesser.jan at googlemail.com>: >>>> > > I am aware that SLEPc is not supposed to calculate all eigenvalues >>>> and eigenvectors, my problem is simply that i want for a physical large >>>> enough system all of them before i can make the transition to go to the >>>> smallest ones. >>>> > > Competitiveness is of secondary importance at the moment. >>>> > > But ihave a problem connected with spectrum slicing. I followed the >>>> instructions in the manual of Chap. 3.4.5 Spectrum Slicing and converted >>>> them to the python package. >>>> > > But now i get the following error. It appears to me that it is not >>>> able to find the ksp object, but i actually do not know why this is the >>>> case. >>>> > > aceback (most recent call last): >>>> > > File "Eigensolver_spectrum_slicing.py", line 216, in >>>> > > solve_eigensystem(DynMatrix_nn, Unity_nn, D_nn.shape, >>>> opt_dict.LowLimit, opt_dict.UpperLimit) >>>> > > File "Eigensolver_spectrum_slicing.py", line 121, in >>>> solve_eigensystem >>>> > > E.setUp() >>>> > > File "SLEPc/EPS.pyx", line 1099, in slepc4py.SLEPc.EPS.setUp >>>> > > petsc4py.PETSc.Error: error code 92 >>>> > > [14] EPSSetUp() line 165 in >>>> /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c >>>> > > [14] EPSSetUp_KrylovSchur() line 146 in >>>> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c >>>> > > [14] EPSSetUp_KrylovSchur_Slice() line 410 in >>>> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>>> > > [14] EPSSliceGetEPS() line 300 in >>>> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>>> > > [14] EPSSetUp() line 165 in >>>> /tmp/pip-install-golhudw7/slepc/src/eps/interface/epssetup.c >>>> > > [14] EPSSetUp_KrylovSchur() line 146 in >>>> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/krylovschur.c >>>> > > [14] EPSSetUp_KrylovSchur_Slice() line 461 in >>>> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>>> > > [14] EPSSliceGetInertia() line 331 in >>>> /tmp/pip-install-golhudw7/slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>>> > > [14] STSetUp() line 271 in >>>> /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/interface/stsolve.c >>>> > > [14] STSetUp_Sinvert() line 132 in >>>> /tmp/pip-install-golhudw7/slepc/src/sys/classes/st/impls/sinvert/sinvert.c >>>> > > [14] KSPSetUp() line 381 in >>>> /tmp/pip-install-xmiaat2t/petsc/src/ksp/ksp/interface/itfunc.c >>>> > > [14] PCSetUp() line 923 in >>>> /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/interface/precon.c >>>> > > [14] PCSetUp_Cholesky() line 86 in >>>> /tmp/pip-install-xmiaat2t/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c >>>> > > [14] MatGetFactor() line 4318 in >>>> /tmp/pip-install-xmiaat2t/petsc/src/mat/interface/matrix.c >>>> > > [14] See >>>> http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for >>>> possible LU and Cholesky solvers >>>> > > [14] Could not locate a solver package. Perhaps you must >>>> ./configure with --download- >>>> > > >>>> > > The code i used to solve the problem is >>>> > > >>>> > > Am Fr., 14. Sep. 2018 um 18:34 Uhr schrieb Matthew Knepley < >>>> knepley at gmail.com>: >>>> > > On Fri, Sep 14, 2018 at 12:19 PM Jose E. Roman >>>> wrote: >>>> > > El 14 sept 2018, a las 17:45, Jan Grie?er < >>>> griesser.jan at googlemail.com> escribi?: >>>> > > >>>> > >> Hey there, >>>> > >> first i want to say thanks to Satish and Matt for helping with >>>> with my last problem with the mpi compilation. I have two questions related >>>> to solving a big, hermitian, standard eigenvalue problem using SLEPc4py., >>>> compiled with Intel MKL and Intel MPI. >>>> > >> >>>> > >> - I am using slepc4py with >>>> > >> mpi and run it with around -n 20 cores at the moment and how i >>>> wanted to ask if there is an easy way to retrieve the eigenvectors? When i >>>> run my code and print for i in range(nconv): >>>> > >> for i in range(nconv): >>>> > >> >>>> > >> val = E. >>>> > >> getEigenpair(i, vr >>>> > >> , vi) >>>> > >> Print( >>>> > >> vr.getArray()) >>>> > >> i get the parts of the eigenvectors according to the partition of >>>> the matrix. Is there any easy way to put them together in an array and >>>> write them to file ? (I am struggling a little bit with the building them >>>> in the correct order) >>>> > > >>>> > > You need VecScatterCreateToZero. There must be an equivalent in >>>> python. >>>> > > >>>> > > An alternative to this which you should consider, because it is >>>> simpler, is to write the vector to a file >>>> > > using some format that PETSc understands, Then you just need >>>> vr.view(viewer) for a viewer like >>>> > > the binary viewer or some ASCII format you like. >>>> > > >>>> > > Thanks, >>>> > > >>>> > > Matt >>>> > >> - I need to solve eigenvalue problems up to a dimension of 100000 >>>> degrees of freedom and i need all eigenvalues and eigenvectors. I think >>>> solving all eigenvalues in one process is far too much and i thought about >>>> if it is possible to apply the spectrum slicing described in Chap. 3.4.5. >>>> Due to the nature of my problem, i am able to simulate smaller systems of >>>> 10000 DOF and extract the biggest eigenvalue, which will be the same for >>>> larger systems sizes. Is this in general possible since i have a standard >>>> HEP problem or is there a better and faster possibility to do this? >>>> > > >>>> > > In general, SLEPc is not intended for computing the whole spectrum. >>>> You can try with spectrum slicing but this will be competitive if computing >>>> just a percentage of eigenvalues, 50% say. >>>> > > >>>> > > Jose >>>> > > >>>> > >> >>>> > >> Thank you very much! >>>> > > >>>> > > >>>> > > -- >>>> > > What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> > > -- Norbert Wiener >>>> > > >>>> > > https://www.cse.buffalo.edu/~knepley/ >>>> > >>>> >>>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yjwu16 at gmail.com Wed Sep 19 10:26:32 2018 From: yjwu16 at gmail.com (Yingjie Wu) Date: Wed, 19 Sep 2018 23:26:32 +0800 Subject: [petsc-users] Problem about DIVERGED_FUNCTION_COUNT Message-ID: Dear Petsc developer: Hi, Thank you very much for your previous reply . I recently wrote an example of neutron diffusion, which shows that the nonlinear residuals are gradually decreasing, but the program terminates in the tenth step of the nonlinear step.The program output information is as follows. -snes_fd -pc_type lu -snes_view -snes_converged_reason -snes_monitor_cancel -ksp_converged_reason -ksp_monitor_true_residual iter = 0, SNES Function norm 0.999045 0 KSP preconditioned resid norm 1.143042621777e+02 true resid norm 9.990445435144e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.250033129078e-12 true resid norm 3.772661181818e-14 ||r(i)||/||b|| 3.776269242757e-14 Linear solve converged due to CONVERGED_RTOL iterations 1 iter = 1, SNES Function norm 0.863398 0 KSP preconditioned resid norm 3.176569252473e+00 true resid norm 8.633978674995e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.360979829512e-14 true resid norm 1.835881897914e-14 ||r(i)||/||b|| 2.126345184557e-14 Linear solve converged due to CONVERGED_RTOL iterations 1 iter = 2, SNES Function norm 0.217885 0 KSP preconditioned resid norm 5.796196387015e+01 true resid norm 2.178851605144e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.643306967470e-13 true resid norm 3.884638811957e-14 ||r(i)||/||b|| 1.782883608405e-13 Linear solve converged due to CONVERGED_RTOL iterations 1 iter = 3, SNES Function norm 0.217871 0 KSP preconditioned resid norm 3.792350832110e+01 true resid norm 2.178711262680e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.369102629758e-12 true resid norm 1.632695466260e-13 ||r(i)||/||b|| 7.493858843196e-13 Linear solve converged due to CONVERGED_RTOL iterations 1 iter = 4, SNES Function norm 0.217839 0 KSP preconditioned resid norm 2.467674567895e+01 true resid norm 2.178387732479e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.336018611676e-12 true resid norm 2.369440379931e-14 ||r(i)||/||b|| 1.087703692324e-13 Linear solve converged due to CONVERGED_RTOL iterations 1 iter = 5, SNES Function norm 0.217793 0 KSP preconditioned resid norm 1.325333377704e+01 true resid norm 2.177933144217e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.663121116220e-13 true resid norm 6.399802508715e-15 ||r(i)||/||b|| 2.938475189519e-14 Linear solve converged due to CONVERGED_RTOL iterations 1 iter = 6, SNES Function norm 0.217546 0 KSP preconditioned resid norm 8.654271429945e+00 true resid norm 2.175456199462e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.103027422078e-12 true resid norm 1.896269373207e-14 ||r(i)||/||b|| 8.716651586351e-14 Linear solve converged due to CONVERGED_RTOL iterations 1 iter = 7, SNES Function norm 0.217024 0 KSP preconditioned resid norm 5.238463146996e+00 true resid norm 2.170240974179e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.538914082201e-13 true resid norm 1.398699322805e-14 ||r(i)||/||b|| 6.444903305426e-14 Linear solve converged due to CONVERGED_RTOL iterations 1 iter = 8, SNES Function norm 0.215661 0 KSP preconditioned resid norm 3.380654310210e+00 true resid norm 2.156611254980e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.676163043758e-14 true resid norm 2.209776803347e-15 ||r(i)||/||b|| 1.024652355979e-14 Linear solve converged due to CONVERGED_RTOL iterations 1 iter = 9, SNES Function norm 0.213573 0 KSP preconditioned resid norm 1.833370627403e+00 true resid norm 2.135728351107e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.237094085371e-13 true resid norm 3.167085089319e-15 ||r(i)||/||b|| 1.482906329205e-14 Linear solve converged due to CONVERGED_RTOL iterations 1 iter = 10, SNES Function norm 0.206951 0 KSP preconditioned resid norm 1.011617991320e+00 true resid norm 2.069514796799e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.789650938118e-13 true resid norm 5.695302032970e-14 ||r(i)||/||b|| 2.751998701231e-13 Linear solve converged due to CONVERGED_RTOL iterations 1 Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT iterations 10 SNES Object: 1 MPI processes type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=11 total number of function evaluations=10623 norm schedule ALWAYS Jacobian is built using finite differences one column at a time SNESLineSearch Object: 1 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 KSP Object: 1 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: lu out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5., needed 5.81047 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=961, cols=961 package used to perform factorization: petsc total: nonzeros=38169, allocated nonzeros=38169 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=961, cols=961 total: nonzeros=6569, allocated nonzeros=15485 total number of mallocs used during MatSetValues calls =712 not using I-node routines I don't know why the program was terminated. I really need your help. Thanks, Yingjie -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Sep 19 11:06:30 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 19 Sep 2018 16:06:30 +0000 Subject: [petsc-users] Problem about DIVERGED_FUNCTION_COUNT In-Reply-To: References: Message-ID: Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT > SNES Object: 1 MPI processes > type: newtonls > maximum iterations=50, maximum function evaluations=10000 You need to set a much higher count for number of function evaluations, say -snes_max_funcs 1.e12 Barry We should probably increase the default in PETSc. > On Sep 19, 2018, at 10:26 AM, Yingjie Wu wrote: > > Dear Petsc developer: > Hi, > > Thank you very much for your previous reply . > I recently wrote an example of neutron diffusion, which shows that the nonlinear residuals are gradually decreasing, but the program terminates in the tenth step of the nonlinear step.The program output information is as follows. > > -snes_fd -pc_type lu -snes_view -snes_converged_reason -snes_monitor_cancel -ksp_converged_reason -ksp_monitor_true_residual > > iter = 0, SNES Function norm 0.999045 > 0 KSP preconditioned resid norm 1.143042621777e+02 true resid norm 9.990445435144e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.250033129078e-12 true resid norm 3.772661181818e-14 ||r(i)||/||b|| 3.776269242757e-14 > Linear solve converged due to CONVERGED_RTOL iterations 1 > iter = 1, SNES Function norm 0.863398 > 0 KSP preconditioned resid norm 3.176569252473e+00 true resid norm 8.633978674995e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 7.360979829512e-14 true resid norm 1.835881897914e-14 ||r(i)||/||b|| 2.126345184557e-14 > Linear solve converged due to CONVERGED_RTOL iterations 1 > iter = 2, SNES Function norm 0.217885 > 0 KSP preconditioned resid norm 5.796196387015e+01 true resid norm 2.178851605144e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.643306967470e-13 true resid norm 3.884638811957e-14 ||r(i)||/||b|| 1.782883608405e-13 > Linear solve converged due to CONVERGED_RTOL iterations 1 > iter = 3, SNES Function norm 0.217871 > 0 KSP preconditioned resid norm 3.792350832110e+01 true resid norm 2.178711262680e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.369102629758e-12 true resid norm 1.632695466260e-13 ||r(i)||/||b|| 7.493858843196e-13 > Linear solve converged due to CONVERGED_RTOL iterations 1 > iter = 4, SNES Function norm 0.217839 > 0 KSP preconditioned resid norm 2.467674567895e+01 true resid norm 2.178387732479e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.336018611676e-12 true resid norm 2.369440379931e-14 ||r(i)||/||b|| 1.087703692324e-13 > Linear solve converged due to CONVERGED_RTOL iterations 1 > iter = 5, SNES Function norm 0.217793 > 0 KSP preconditioned resid norm 1.325333377704e+01 true resid norm 2.177933144217e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.663121116220e-13 true resid norm 6.399802508715e-15 ||r(i)||/||b|| 2.938475189519e-14 > Linear solve converged due to CONVERGED_RTOL iterations 1 > iter = 6, SNES Function norm 0.217546 > 0 KSP preconditioned resid norm 8.654271429945e+00 true resid norm 2.175456199462e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.103027422078e-12 true resid norm 1.896269373207e-14 ||r(i)||/||b|| 8.716651586351e-14 > Linear solve converged due to CONVERGED_RTOL iterations 1 > iter = 7, SNES Function norm 0.217024 > 0 KSP preconditioned resid norm 5.238463146996e+00 true resid norm 2.170240974179e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.538914082201e-13 true resid norm 1.398699322805e-14 ||r(i)||/||b|| 6.444903305426e-14 > Linear solve converged due to CONVERGED_RTOL iterations 1 > iter = 8, SNES Function norm 0.215661 > 0 KSP preconditioned resid norm 3.380654310210e+00 true resid norm 2.156611254980e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.676163043758e-14 true resid norm 2.209776803347e-15 ||r(i)||/||b|| 1.024652355979e-14 > Linear solve converged due to CONVERGED_RTOL iterations 1 > iter = 9, SNES Function norm 0.213573 > 0 KSP preconditioned resid norm 1.833370627403e+00 true resid norm 2.135728351107e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.237094085371e-13 true resid norm 3.167085089319e-15 ||r(i)||/||b|| 1.482906329205e-14 > Linear solve converged due to CONVERGED_RTOL iterations 1 > iter = 10, SNES Function norm 0.206951 > 0 KSP preconditioned resid norm 1.011617991320e+00 true resid norm 2.069514796799e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.789650938118e-13 true resid norm 5.695302032970e-14 ||r(i)||/||b|| 2.751998701231e-13 > Linear solve converged due to CONVERGED_RTOL iterations 1 > Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT iterations 10 > SNES Object: 1 MPI processes > type: newtonls > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=11 > total number of function evaluations=10623 > norm schedule ALWAYS > Jacobian is built using finite differences one column at a time > SNESLineSearch Object: 1 MPI processes > type: bt > interpolation: cubic > alpha=1.000000e-04 > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 > maximum iterations=40 > KSP Object: 1 MPI processes > type: gmres > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 5.81047 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=961, cols=961 > package used to perform factorization: petsc > total: nonzeros=38169, allocated nonzeros=38169 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=961, cols=961 > total: nonzeros=6569, allocated nonzeros=15485 > total number of mallocs used during MatSetValues calls =712 > not using I-node routines > > > > I don't know why the program was terminated. I really need your help. > > Thanks, > Yingjie > > > From david97sommer at gmail.com Thu Sep 20 06:46:16 2018 From: david97sommer at gmail.com (David Sommer) Date: Thu, 20 Sep 2018 13:46:16 +0200 Subject: [petsc-users] CMake error Message-ID: Hi, i am trying to build my project with CMake. For PETSc i use the FindPETSc.cmake Module from Barry's repository. Unfortunately FindPackage(PETSc) fails with "(missing: PETSC_EXECUTABLE_RUNS)" In CMakeError.log i have "error while loading shared libraries: libpetsc.so.3.9: cannot open shared object file: No such file or directory" So i manually set $LD_LIBRARY_PATH and $LIBRARY_PATH, both to $PETSC_DIR/$PETSC_ARCH. Additionally, because of http://jacobmerson.com/2016/01/17/cmake-petsc2.html i ran CMake with cmake .. -D CMAKE_C_COMPILER=$(which mpicc) -D CMAKE_CXX_COMPILER=$(which mpicxx) Where did i go wrong? Thanks David Sommer -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 20 06:56:14 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 20 Sep 2018 07:56:14 -0400 Subject: [petsc-users] CMake error In-Reply-To: References: Message-ID: On Thu, Sep 20, 2018 at 7:47 AM David Sommer wrote: > Hi, > i am trying to build my project with CMake. For PETSc i use the > FindPETSc.cmake Module from Barry's repository. Unfortunately > FindPackage(PETSc) fails with > "(missing: PETSC_EXECUTABLE_RUNS)" > In CMakeError.log i have > "error while loading shared libraries: libpetsc.so.3.9: cannot open shared > object file: No such file or directory" > So i manually set $LD_LIBRARY_PATH and $LIBRARY_PATH, both to > $PETSC_DIR/$PETSC_ARCH. > I do not believe that is correct. I think you need $PETSC_DIR/$PETSC_ARCH/lib Thanks, Matt > Additionally, because of > http://jacobmerson.com/2016/01/17/cmake-petsc2.html > i ran CMake with > cmake .. -D CMAKE_C_COMPILER=$(which mpicc) -D CMAKE_CXX_COMPILER=$(which > mpicxx) > Where did i go wrong? > > Thanks > David Sommer > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Thu Sep 20 08:57:18 2018 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 20 Sep 2018 21:57:18 +0800 Subject: [petsc-users] Unable to read in values thru namelist in Fortran after using PETSc 64bit in linux In-Reply-To: References: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com> <0AC1BA91-75F8-4AAB-A803-832B089A2C44@anl.gov> <22565566-590d-7c89-390a-99557ce05382@gmail.com> Message-ID: <69d01b45-050f-2729-3015-14f3d80202d5@gmail.com> Hi, Sorry I'm still a bit confused. My 64bit code still doesn't work once I use more than 1 procs. It just aborts at some point. I'm been trying to use ARM Forge mpi debugging tool to find the error but it's a bit difficult to back trace. So I should carefully inspect each mpi subroutine or function, is that correct? If it's INT, then I should use PetscMPIInt. If it's labeled void *, I should use PetscInt. Is that so? Thank you very much Yours sincerely, ================================================ TAY Wee-Beng ??? (Zheng Weiming) Personal research webpage: http://tayweebeng.wixsite.com/website Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA linkedin: www.linkedin.com/in/tay-weebeng ================================================ On 19/9/2018 12:49 AM, Smith, Barry F. wrote: > PetscMPIInt (or integer) are for all lengths passed to MPI functions; simple look at the prototypes for the MPI function you care about and it will tell you which arguments are integer. The DATA you are passing into the MPI arrays (which are labeled void * in the manual pages) should be PetscInt. > > > >> On Sep 18, 2018, at 1:39 AM, TAY wee-beng wrote: >> >> Hi, >> >> In that case, does it apply to all MPI subroutines such as MPI_ALLGATHER? >> >> In other words, must I assign local_array_length etc as PetscMPIInt? >> >> call MPI_ALLGATHER(local_array_length,1,MPIU_INTEGER,array_length,1,MPIU_INTEGER,MPI_COMM_WORLD,ierr) >> >> Or is it ok to change all integers from PetscInt to PetscMPIInt? >> >> With the exception of ierr - PetscErrorCode >> >> >> Thank you very much. >> >> Yours sincerely, >> >> ================================================ >> TAY Wee-Beng (Zheng Weiming) ??? >> Personal research webpage: http://tayweebeng.wixsite.com/website >> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >> linkedin: www.linkedin.com/in/tay-weebeng >> ================================================ >> >> On 18/9/2018 1:39 PM, Balay, Satish wrote: >>> https://www.mpich.org/static/docs/v3.1/www3/MPI_Comm_size.html >>> >>> int MPI_Comm_size( MPI_Comm comm, int *size ) >>> >>> i.e there is no PetscInt here. [MPI does not know about PETSc datatypes] >>> >>> For convinence we provide PetscMPIInt to keep track of such variables >>> [similarly PetscBLASInt]. For eg: Check src/vec/vec/examples/tests/ex2f.F >>> >>> Satish >>> >>> On Tue, 18 Sep 2018, TAY wee-beng wrote: >>> >>>> Hi, >>>> >>>> I managed to find the error appearing after using PETSc 64bit in linux - >>>> >>>> call MPI_COMM_SIZE(MPI_COMM_WORLD, num_procs, ierr) >>>> >>>> I have assigned num_procs as PetscInt and I got 0 instead of 1 (for 1 procs) >>>> >>>> Assigning num_procs as integer as the problem. >>>> >>>> Is this supposed to be the case? Or is it a bug? >>>> >>>> Thank you very much. >>>> >>>> Yours sincerely, >>>> >>>> ================================================ >>>> TAY Wee-Beng (Zheng Weiming) ??? >>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>> Youtube research showcase: >>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>> linkedin: www.linkedin.com/in/tay-weebeng >>>> ================================================ >>>> >>>> On 8/9/2018 1:14 AM, Smith, Barry F. wrote: >>>>> You can try valgrind >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>> >>>>> Barry >>>>> >>>>> >>>>>> On Sep 7, 2018, at 1:44 AM, TAY wee-beng wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I found that I am unable to read in values thru namelist in Fortran after >>>>>> using PETSc 64bit in linux. >>>>>> >>>>>> I have a parameter txt file which is read in using namelist in Fortran? >>>>>> >>>>>> namelist /body_input/ no_body, convex_body, motion_type, hover, wing_config >>>>>> ... >>>>>> >>>>>> open (unit = 44 , FILE = "ibm3d_input.txt" , status = "old", iostat = >>>>>> openstatus(4)) >>>>>> >>>>>> if (openstatus(4) > 0) then >>>>>> >>>>>> print *, "ibm3d_input file not present or wrong filename." >>>>>> >>>>>> stop >>>>>> >>>>>> end if >>>>>> >>>>>> read (44,nml = solver_input) >>>>>> >>>>>> read (44,nml = grid_input) >>>>>> >>>>>> read (44,nml = body_input)... >>>>>> >>>>>> >>>>>> After using PETSc 64bit, my code aborts and I realise that it is because >>>>>> the values have became NaN. Strangely, it does not occur in windows with >>>>>> VS2008. >>>>>> >>>>>> I wonder if it's a bug with the Intel Fortran compiler 2018. >>>>>> >>>>>> Anyone has similar experience? >>>>>> >>>>>> -- >>>>>> Thank you very much. >>>>>> >>>>>> Yours sincerely, >>>>>> >>>>>> ================================================ >>>>>> TAY Wee-Beng (Zheng Weiming) ??? >>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>> Youtube research showcase: >>>>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>> ================================================ >>>>>> From knepley at gmail.com Thu Sep 20 09:45:31 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 20 Sep 2018 10:45:31 -0400 Subject: [petsc-users] Unable to read in values thru namelist in Fortran after using PETSc 64bit in linux In-Reply-To: <69d01b45-050f-2729-3015-14f3d80202d5@gmail.com> References: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com> <0AC1BA91-75F8-4AAB-A803-832B089A2C44@anl.gov> <22565566-590d-7c89-390a-99557ce05382@gmail.com> <69d01b45-050f-2729-3015-14f3d80202d5@gmail.com> Message-ID: On Thu, Sep 20, 2018 at 9:58 AM TAY wee-beng wrote: > Hi, > > Sorry I'm still a bit confused. My 64bit code still doesn't work once I > use more than 1 procs. It just aborts at some point. I'm been trying to > use ARM Forge mpi debugging tool to find the error but it's a bit > difficult to back trace. > > So I should carefully inspect each mpi subroutine or function, is that > correct? > > If it's INT, then I should use PetscMPIInt. If it's labeled void *, I > should use PetscInt. Is that so? > MPI routines take void * to data and a datatype. You must use the MPI datatype that matches the data you have in memory. MPIU_INT matches PetscInt. MPI_INT matches int or PetscMPIInt. You just need to use the matching thing. Thanks, Matt > Thank you very much > > Yours sincerely, > > ================================================ > TAY Wee-Beng ??? (Zheng Weiming) > Personal research webpage: http://tayweebeng.wixsite.com/website > Youtube research showcase: > https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > linkedin: www.linkedin.com/in/tay-weebeng > ================================================ > > On 19/9/2018 12:49 AM, Smith, Barry F. wrote: > > PetscMPIInt (or integer) are for all lengths passed to MPI > functions; simple look at the prototypes for the MPI function you care > about and it will tell you which arguments are integer. The DATA you are > passing into the MPI arrays (which are labeled void * in the manual pages) > should be PetscInt. > > > > > > > >> On Sep 18, 2018, at 1:39 AM, TAY wee-beng wrote: > >> > >> Hi, > >> > >> In that case, does it apply to all MPI subroutines such as > MPI_ALLGATHER? > >> > >> In other words, must I assign local_array_length etc as PetscMPIInt? > >> > >> call > MPI_ALLGATHER(local_array_length,1,MPIU_INTEGER,array_length,1,MPIU_INTEGER,MPI_COMM_WORLD,ierr) > >> > >> Or is it ok to change all integers from PetscInt to PetscMPIInt? > >> > >> With the exception of ierr - PetscErrorCode > >> > >> > >> Thank you very much. > >> > >> Yours sincerely, > >> > >> ================================================ > >> TAY Wee-Beng (Zheng Weiming) ??? > >> Personal research webpage: http://tayweebeng.wixsite.com/website > >> Youtube research showcase: > https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > >> linkedin: www.linkedin.com/in/tay-weebeng > >> ================================================ > >> > >> On 18/9/2018 1:39 PM, Balay, Satish wrote: > >>> https://www.mpich.org/static/docs/v3.1/www3/MPI_Comm_size.html > >>> > >>> int MPI_Comm_size( MPI_Comm comm, int *size ) > >>> > >>> i.e there is no PetscInt here. [MPI does not know about PETSc > datatypes] > >>> > >>> For convinence we provide PetscMPIInt to keep track of such variables > >>> [similarly PetscBLASInt]. For eg: Check > src/vec/vec/examples/tests/ex2f.F > >>> > >>> Satish > >>> > >>> On Tue, 18 Sep 2018, TAY wee-beng wrote: > >>> > >>>> Hi, > >>>> > >>>> I managed to find the error appearing after using PETSc 64bit in > linux - > >>>> > >>>> call MPI_COMM_SIZE(MPI_COMM_WORLD, num_procs, ierr) > >>>> > >>>> I have assigned num_procs as PetscInt and I got 0 instead of 1 (for 1 > procs) > >>>> > >>>> Assigning num_procs as integer as the problem. > >>>> > >>>> Is this supposed to be the case? Or is it a bug? > >>>> > >>>> Thank you very much. > >>>> > >>>> Yours sincerely, > >>>> > >>>> ================================================ > >>>> TAY Wee-Beng (Zheng Weiming) ??? > >>>> Personal research webpage: http://tayweebeng.wixsite.com/website > >>>> Youtube research showcase: > >>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > >>>> linkedin: www.linkedin.com/in/tay-weebeng > >>>> ================================================ > >>>> > >>>> On 8/9/2018 1:14 AM, Smith, Barry F. wrote: > >>>>> You can try valgrind > >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > >>>>> > >>>>> Barry > >>>>> > >>>>> > >>>>>> On Sep 7, 2018, at 1:44 AM, TAY wee-beng wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> I found that I am unable to read in values thru namelist in Fortran > after > >>>>>> using PETSc 64bit in linux. > >>>>>> > >>>>>> I have a parameter txt file which is read in using namelist in > Fortran? > >>>>>> > >>>>>> namelist /body_input/ no_body, convex_body, motion_type, hover, > wing_config > >>>>>> ... > >>>>>> > >>>>>> open (unit = 44 , FILE = "ibm3d_input.txt" , status = "old", iostat > = > >>>>>> openstatus(4)) > >>>>>> > >>>>>> if (openstatus(4) > 0) then > >>>>>> > >>>>>> print *, "ibm3d_input file not present or wrong > filename." > >>>>>> > >>>>>> stop > >>>>>> > >>>>>> end if > >>>>>> > >>>>>> read (44,nml = solver_input) > >>>>>> > >>>>>> read (44,nml = grid_input) > >>>>>> > >>>>>> read (44,nml = body_input)... > >>>>>> > >>>>>> > >>>>>> After using PETSc 64bit, my code aborts and I realise that it is > because > >>>>>> the values have became NaN. Strangely, it does not occur in windows > with > >>>>>> VS2008. > >>>>>> > >>>>>> I wonder if it's a bug with the Intel Fortran compiler 2018. > >>>>>> > >>>>>> Anyone has similar experience? > >>>>>> > >>>>>> -- > >>>>>> Thank you very much. > >>>>>> > >>>>>> Yours sincerely, > >>>>>> > >>>>>> ================================================ > >>>>>> TAY Wee-Beng (Zheng Weiming) ??? > >>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website > >>>>>> Youtube research showcase: > >>>>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > >>>>>> linkedin: www.linkedin.com/in/tay-weebeng > >>>>>> ================================================ > >>>>>> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Sep 20 11:35:49 2018 From: jed at jedbrown.org (Jed Brown) Date: Thu, 20 Sep 2018 10:35:49 -0600 Subject: [petsc-users] CMake error In-Reply-To: References: Message-ID: <87efdo9lei.fsf@jedbrown.org> I recommend using pkg-config these days if you're using CMake, but the error from FindPETSc.cmake indicates either an environment problem or an issue with your install. David Sommer writes: > Hi, > i am trying to build my project with CMake. For PETSc i use the > FindPETSc.cmake Module from Barry's repository. Unfortunately > FindPackage(PETSc) fails with > "(missing: PETSC_EXECUTABLE_RUNS)" > In CMakeError.log i have > "error while loading shared libraries: libpetsc.so.3.9: cannot open shared > object file: No such file or directory" > So i manually set $LD_LIBRARY_PATH and $LIBRARY_PATH, both to > $PETSC_DIR/$PETSC_ARCH. Additionally, because of > http://jacobmerson.com/2016/01/17/cmake-petsc2.html > i ran CMake with > cmake .. -D CMAKE_C_COMPILER=$(which mpicc) -D CMAKE_CXX_COMPILER=$(which > mpicxx) > Where did i go wrong? > > Thanks > David Sommer From bsmith at mcs.anl.gov Thu Sep 20 13:57:03 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 20 Sep 2018 18:57:03 +0000 Subject: [petsc-users] Unable to read in values thru namelist in Fortran after using PETSc 64bit in linux In-Reply-To: <69d01b45-050f-2729-3015-14f3d80202d5@gmail.com> References: <03a7cb7c-ec5a-f9ef-e7ed-fa388b9b78b1@gmail.com> <0AC1BA91-75F8-4AAB-A803-832B089A2C44@anl.gov> <22565566-590d-7c89-390a-99557ce05382@gmail.com> <69d01b45-050f-2729-3015-14f3d80202d5@gmail.com> Message-ID: <686A13FC-7B4E-4F7C-84E0-3A0FCFC08FA1@mcs.anl.gov> Yes, you need to go through your code and check each MPI call and make sure you use PetscMPIInt for integer arguments and PetscInt for the void* arguments and also make sure that the data type you use in the MPI calls (when communicating PetscInt) is MPIU_INT. You should not need a fancy debugger to find out the crash point. Just a basic debugger like gdb, lldb, or dbx will Barry > On Sep 20, 2018, at 8:57 AM, TAY wee-beng wrote: > > Hi, > > Sorry I'm still a bit confused. My 64bit code still doesn't work once I use more than 1 procs. It just aborts at some point. I'm been trying to use ARM Forge mpi debugging tool to find the error but it's a bit difficult to back trace. > > So I should carefully inspect each mpi subroutine or function, is that correct? > > If it's INT, then I should use PetscMPIInt. If it's labeled void *, I should use PetscInt. Is that so? > > Thank you very much > > Yours sincerely, > > ================================================ > TAY Wee-Beng ??? (Zheng Weiming) > Personal research webpage: http://tayweebeng.wixsite.com/website > Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > linkedin: www.linkedin.com/in/tay-weebeng > ================================================ > > On 19/9/2018 12:49 AM, Smith, Barry F. wrote: >> PetscMPIInt (or integer) are for all lengths passed to MPI functions; simple look at the prototypes for the MPI function you care about and it will tell you which arguments are integer. The DATA you are passing into the MPI arrays (which are labeled void * in the manual pages) should be PetscInt. >> >> >> >>> On Sep 18, 2018, at 1:39 AM, TAY wee-beng wrote: >>> >>> Hi, >>> >>> In that case, does it apply to all MPI subroutines such as MPI_ALLGATHER? >>> >>> In other words, must I assign local_array_length etc as PetscMPIInt? >>> >>> call MPI_ALLGATHER(local_array_length,1,MPIU_INTEGER,array_length,1,MPIU_INTEGER,MPI_COMM_WORLD,ierr) >>> >>> Or is it ok to change all integers from PetscInt to PetscMPIInt? >>> >>> With the exception of ierr - PetscErrorCode >>> >>> >>> Thank you very much. >>> >>> Yours sincerely, >>> >>> ================================================ >>> TAY Wee-Beng (Zheng Weiming) ??? >>> Personal research webpage: http://tayweebeng.wixsite.com/website >>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>> linkedin: www.linkedin.com/in/tay-weebeng >>> ================================================ >>> >>> On 18/9/2018 1:39 PM, Balay, Satish wrote: >>>> https://www.mpich.org/static/docs/v3.1/www3/MPI_Comm_size.html >>>> >>>> int MPI_Comm_size( MPI_Comm comm, int *size ) >>>> >>>> i.e there is no PetscInt here. [MPI does not know about PETSc datatypes] >>>> >>>> For convinence we provide PetscMPIInt to keep track of such variables >>>> [similarly PetscBLASInt]. For eg: Check src/vec/vec/examples/tests/ex2f.F >>>> >>>> Satish >>>> >>>> On Tue, 18 Sep 2018, TAY wee-beng wrote: >>>> >>>>> Hi, >>>>> >>>>> I managed to find the error appearing after using PETSc 64bit in linux - >>>>> >>>>> call MPI_COMM_SIZE(MPI_COMM_WORLD, num_procs, ierr) >>>>> >>>>> I have assigned num_procs as PetscInt and I got 0 instead of 1 (for 1 procs) >>>>> >>>>> Assigning num_procs as integer as the problem. >>>>> >>>>> Is this supposed to be the case? Or is it a bug? >>>>> >>>>> Thank you very much. >>>>> >>>>> Yours sincerely, >>>>> >>>>> ================================================ >>>>> TAY Wee-Beng (Zheng Weiming) ??? >>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>> Youtube research showcase: >>>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>> ================================================ >>>>> >>>>> On 8/9/2018 1:14 AM, Smith, Barry F. wrote: >>>>>> You can try valgrind >>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>>> On Sep 7, 2018, at 1:44 AM, TAY wee-beng wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I found that I am unable to read in values thru namelist in Fortran after >>>>>>> using PETSc 64bit in linux. >>>>>>> >>>>>>> I have a parameter txt file which is read in using namelist in Fortran? >>>>>>> >>>>>>> namelist /body_input/ no_body, convex_body, motion_type, hover, wing_config >>>>>>> ... >>>>>>> >>>>>>> open (unit = 44 , FILE = "ibm3d_input.txt" , status = "old", iostat = >>>>>>> openstatus(4)) >>>>>>> >>>>>>> if (openstatus(4) > 0) then >>>>>>> >>>>>>> print *, "ibm3d_input file not present or wrong filename." >>>>>>> >>>>>>> stop >>>>>>> >>>>>>> end if >>>>>>> >>>>>>> read (44,nml = solver_input) >>>>>>> >>>>>>> read (44,nml = grid_input) >>>>>>> >>>>>>> read (44,nml = body_input)... >>>>>>> >>>>>>> >>>>>>> After using PETSc 64bit, my code aborts and I realise that it is because >>>>>>> the values have became NaN. Strangely, it does not occur in windows with >>>>>>> VS2008. >>>>>>> >>>>>>> I wonder if it's a bug with the Intel Fortran compiler 2018. >>>>>>> >>>>>>> Anyone has similar experience? >>>>>>> >>>>>>> -- >>>>>>> Thank you very much. >>>>>>> >>>>>>> Yours sincerely, >>>>>>> >>>>>>> ================================================ >>>>>>> TAY Wee-Beng (Zheng Weiming) ??? >>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>>> Youtube research showcase: >>>>>>> https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>>> ================================================ >>>>>>> > From mfadams at lbl.gov Thu Sep 20 14:38:09 2018 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 20 Sep 2018 15:38:09 -0400 Subject: [petsc-users] error make with cude on IBM Message-ID: I am getting this error on SUMMIT. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: application/octet-stream Size: 78700 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log.gz Type: application/x-gzip Size: 703781 bytes Desc: not available URL: From balay at mcs.anl.gov Thu Sep 20 14:51:28 2018 From: balay at mcs.anl.gov (Balay, Satish) Date: Thu, 20 Sep 2018 19:51:28 +0000 Subject: [petsc-users] error make with cude on IBM In-Reply-To: References: Message-ID: > /autofs/nccs-svm1_home1/adams/petsc/include/petscmath.h(389): error: identifier "PETSC_CXX_INLINE" is undefined Somehow nvcc is setting '__cplusplus' flag - and causing he errors.. change --with-cxx=0 to --with-cxx=mpicxx Satish --------- On Thu, 20 Sep 2018, Mark Adams wrote: > I am getting this error on SUMMIT. > From bsmith at mcs.anl.gov Thu Sep 20 16:21:37 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 20 Sep 2018 21:21:37 +0000 Subject: [petsc-users] Problems ablout setting up variables and SNES In-Reply-To: References: Message-ID: <5A0249DA-10A0-4E74-A93E-D10507CB7FFF@anl.gov> Yingjie, Sorry no one responded earlier to your query. What you would like to do (have three field variables plus a single variable) as your unknowns in a nonlinear system is possible in PETSc but cumbersome to implement unfortunately. So first I urge you to consider alternatives that do not require the simultaneous solution of such a system. If you try to setup a system that involves a single degree of freedom plus your field variables you will need to use DMCOMPOSITE. There are three examples that do this in src/snes/examples/tutorials/ ex21.c ex22.c ex28.c Good luck Barry > On Sep 17, 2018, at 3:22 AM, Yingjie Wu wrote: > > Dear Petsc developer: > Hi, > Thank you very much for your previous reply, recently I encountered some problems in the implementation of the program. > > 1. Previously I used DMDA to create distributed variables (Variables are different in each grids). Now I want to add a single variable (independent of the grid). How do I add it? How do I write programs with distributed variables and single variables exist at the same time? > > I've created three "Field" variables before, and I'd like to add a single variable. How should I describe it in the program? The codes are as below. > > ierr=DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,10,10,PETSC_DECIDE,PETSC_DECIDE,3,1,0,0,&da);CHKERRQ(ierr); > > ierr = DMSetFromOptions(da);CHKERRQ(ierr); > > ierr = DMSetUp(da);CHKERRQ(ierr); > > ierr = SNESSetDM(snes,(DM)da);CHKERRQ(ierr); > > ierr = DMDASetFieldName(da,0,"Pressure");CHKERRQ(ierr); > > ierr = DMDASetFieldName(da,1,"Y-Velocity");CHKERRQ(ierr); > > ierr = DMDASetFieldName(da,2,"Fluid_temperature");CHKERRQ(ierr); > > ierr = DMCreateGlobalVector(da,&x);CHKERRQ(ierr); > > > > 2. At present, I am solving nonlinear equations. Now I need to do one-step nonlinear calculation first, and then treat the results as coefficients for the subsequent nonlinear calculation. Now I used to use SNESSolve() directly to solve the problem. How can we calculate step by step and process the result before entering the next nonlinear step? > > Since I'm a beginner in PETSc, I would be grateful if there were any examples. > I'm looking forward to your reply~ > > Thanks, > Yingjie > > From t.appel17 at imperial.ac.uk Thu Sep 20 18:08:56 2018 From: t.appel17 at imperial.ac.uk (Appel, Thibaut) Date: Thu, 20 Sep 2018 23:08:56 +0000 Subject: [petsc-users] Iterative solver and condition number from FDM + fill-in Message-ID: <739CB2F7-9DD2-4E0E-A19A-673524B7BE73@ic.ac.uk> Dear users, I?m having trouble finding a PC/KSP pair that works for my problem in parallel. I?m solving linearized Navier-Stokes PDE?s discretized with a finite difference method in 2D or 3D in a logically rectangular grid, in complex arithmetic. It obviously works fine with a direct solver but also with GMRES + ILU(3) in sequential. I tried different combinations such as -ksp_type gmres -pc_type asm -sub_pc_type ilu -ksp_type gmres -pc_type bjacobi -sub_pc_type ilu but cannot get the relative residuals below 10^(-2), after 2,000 iterations - even with increasing the number of ILU fill-in levels (up to 5), the number of GMRES restarts (300 to 1000), options such as -ksp_initial_guess_nonzero or -ksp_gmres_cgs_refinement_type refine_always. -ksp_monitor_true_residual does not seem to give more information? Maybe there?s room for more experimentation but if you could suggest a way to have a better diagnostic? With the different equation sets I?m working with, the condition numbers estimated with the petsc faq method vary between 10^3 and 10^7. On top of that I have ridiculous fill-in and have to set -pc_factor_fill to 14, up to 35 (!) sometimes. For our application we need a lot of discretization points in one spatial direction and I read somewhere that condition number scales with the square of discretization steps for FD methods. But is there a way to reduce it in my case? I?m also aware that fill-in should be inevitably expected when you have a sparse matrix with a banded structure arising from a FDM. But I was wondering if there?s something more I can do on the numerical side to, on reduce fill-in and/or help the iterative solver to converge faster? I know my discretized PDE?s + boundary conditions are scaled consistently with regards to matrix entries. I?m using natural ordering (if my unknowns are a_ij, b_ij the unknown vector starts with a_00 b_00 a_10 b_10 a_20 b_20 and ends with a_nxny b_nxny?) but I do not think this has any impact? Thanks for your support, Thibaut From jed at jedbrown.org Thu Sep 20 22:13:39 2018 From: jed at jedbrown.org (Jed Brown) Date: Thu, 20 Sep 2018 21:13:39 -0600 Subject: [petsc-users] Iterative solver and condition number from FDM + fill-in In-Reply-To: <739CB2F7-9DD2-4E0E-A19A-673524B7BE73@ic.ac.uk> References: <739CB2F7-9DD2-4E0E-A19A-673524B7BE73@ic.ac.uk> Message-ID: <87y3bv8rvg.fsf@jedbrown.org> "Appel, Thibaut" writes: > Dear users, > > I?m having trouble finding a PC/KSP pair that works for my problem in parallel. > I?m solving linearized Navier-Stokes PDE?s discretized with a finite difference method in 2D or 3D in a logically rectangular grid, in complex arithmetic. Compressible or incompressible? Staggered or centered grid? Why complex arithmetic? > It obviously works fine with a direct solver but also with GMRES + ILU(3) in sequential. > > I tried different combinations such as > -ksp_type gmres -pc_type asm -sub_pc_type ilu > -ksp_type gmres -pc_type bjacobi -sub_pc_type ilu > > but cannot get the relative residuals below 10^(-2), after 2,000 iterations - even with increasing the number of ILU fill-in levels (up to 5), the number of GMRES restarts (300 to 1000), options such as -ksp_initial_guess_nonzero or -ksp_gmres_cgs_refinement_type refine_always. -ksp_monitor_true_residual does not seem to give more information? > Maybe there?s room for more experimentation but if you could suggest a way to have a better diagnostic? > > With the different equation sets I?m working with, the condition numbers estimated with the petsc faq method vary between 10^3 and 10^7. > On top of that I have ridiculous fill-in and have to set -pc_factor_fill to 14, up to 35 (!) sometimes. > > For our application we need a lot of discretization points in one spatial direction and I read somewhere that condition number scales with the square of discretization steps for FD methods. But is there a way to reduce it in my case? > I?m also aware that fill-in should be inevitably expected when you have a sparse matrix with a banded structure arising from a FDM. But I was wondering if there?s something more I can do on the numerical side to, on reduce fill-in and/or help the iterative solver to converge faster? > > I know my discretized PDE?s + boundary conditions are scaled consistently with regards to matrix entries. > I?m using natural ordering (if my unknowns are a_ij, b_ij the unknown vector starts with a_00 b_00 a_10 b_10 a_20 b_20 and ends with a_nxny b_nxny?) but I do not think this has any impact? > > Thanks for your support, > > > Thibaut From t.appel17 at imperial.ac.uk Fri Sep 21 04:16:51 2018 From: t.appel17 at imperial.ac.uk (Appel, Thibaut) Date: Fri, 21 Sep 2018 09:16:51 +0000 Subject: [petsc-users] Iterative solver and condition number from FDM + fill-in In-Reply-To: <87y3bv8rvg.fsf@jedbrown.org> References: <739CB2F7-9DD2-4E0E-A19A-673524B7BE73@ic.ac.uk>, <87y3bv8rvg.fsf@jedbrown.org> Message-ID: <31788D7F-9B66-4170-ADAD-EDD39FFC7E38@imperial.ac.uk> Hi Jed, - It?s incompressible flow but the equations are not singular, we?re using a Poisson equation for the pressure. - It?s a centered/collocated grid. - Complex because we?re seeking solutions under a wave form with complex wavenumbers in the exponential, the right hand side is also complex. Thibaut > On 21 Sep 2018, at 04:13, Jed Brown wrote: > > "Appel, Thibaut" writes: > >> Dear users, >> >> I?m having trouble finding a PC/KSP pair that works for my problem in parallel. >> I?m solving linearized Navier-Stokes PDE?s discretized with a finite difference method in 2D or 3D in a logically rectangular grid, in complex arithmetic. > > Compressible or incompressible? Staggered or centered grid? Why complex arithmetic? > >> It obviously works fine with a direct solver but also with GMRES + ILU(3) in sequential. >> >> I tried different combinations such as >> -ksp_type gmres -pc_type asm -sub_pc_type ilu >> -ksp_type gmres -pc_type bjacobi -sub_pc_type ilu >> >> but cannot get the relative residuals below 10^(-2), after 2,000 iterations - even with increasing the number of ILU fill-in levels (up to 5), the number of GMRES restarts (300 to 1000), options such as -ksp_initial_guess_nonzero or -ksp_gmres_cgs_refinement_type refine_always. -ksp_monitor_true_residual does not seem to give more information? >> Maybe there?s room for more experimentation but if you could suggest a way to have a better diagnostic? >> >> With the different equation sets I?m working with, the condition numbers estimated with the petsc faq method vary between 10^3 and 10^7. >> On top of that I have ridiculous fill-in and have to set -pc_factor_fill to 14, up to 35 (!) sometimes. >> >> For our application we need a lot of discretization points in one spatial direction and I read somewhere that condition number scales with the square of discretization steps for FD methods. But is there a way to reduce it in my case? >> I?m also aware that fill-in should be inevitably expected when you have a sparse matrix with a banded structure arising from a FDM. But I was wondering if there?s something more I can do on the numerical side to, on reduce fill-in and/or help the iterative solver to converge faster? >> >> I know my discretized PDE?s + boundary conditions are scaled consistently with regards to matrix entries. >> I?m using natural ordering (if my unknowns are a_ij, b_ij the unknown vector starts with a_00 b_00 a_10 b_10 a_20 b_20 and ends with a_nxny b_nxny?) but I do not think this has any impact? >> >> Thanks for your support, >> >> >> Thibaut From knepley at gmail.com Fri Sep 21 05:30:00 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 21 Sep 2018 06:30:00 -0400 Subject: [petsc-users] Iterative solver and condition number from FDM + fill-in In-Reply-To: <31788D7F-9B66-4170-ADAD-EDD39FFC7E38@imperial.ac.uk> References: <739CB2F7-9DD2-4E0E-A19A-673524B7BE73@ic.ac.uk> <87y3bv8rvg.fsf@jedbrown.org> <31788D7F-9B66-4170-ADAD-EDD39FFC7E38@imperial.ac.uk> Message-ID: On Fri, Sep 21, 2018 at 5:17 AM Appel, Thibaut wrote: > Hi Jed, > > - It?s incompressible flow but the equations are not singular, we?re using > a Poisson equation for the pressure. > So the only solve is a Poisson solve? This sounds like you have implemented one of the family of Schur complement solvers for Stokes-like equations. We have support (PCFIELDSPLIT) for trying a great range of them. I suggest that this is your best bet. The idea is to create a list of the velocity and pressure unknowns and tell the PC, so that it can split the system into two parts and do intelligent things. This is described in many of the tutorials online. Do you mind writing the equations briefly so we can see them? Thanks, Matt > - It?s a centered/collocated grid. > - Complex because we?re seeking solutions under a wave form with complex > wavenumbers in the exponential, the right hand side is also complex. > > Thibaut > > > On 21 Sep 2018, at 04:13, Jed Brown wrote: > > > > "Appel, Thibaut" writes: > > > >> Dear users, > >> > >> I?m having trouble finding a PC/KSP pair that works for my problem in > parallel. > >> I?m solving linearized Navier-Stokes PDE?s discretized with a finite > difference method in 2D or 3D in a logically rectangular grid, in complex > arithmetic. > > > > Compressible or incompressible? Staggered or centered grid? Why > complex arithmetic? > > > >> It obviously works fine with a direct solver but also with GMRES + > ILU(3) in sequential. > >> > >> I tried different combinations such as > >> -ksp_type gmres -pc_type asm -sub_pc_type ilu > >> -ksp_type gmres -pc_type bjacobi -sub_pc_type ilu > >> > >> but cannot get the relative residuals below 10^(-2), after 2,000 > iterations - even with increasing the number of ILU fill-in levels (up to > 5), the number of GMRES restarts (300 to 1000), options such as > -ksp_initial_guess_nonzero or -ksp_gmres_cgs_refinement_type refine_always. > -ksp_monitor_true_residual does not seem to give more information? > >> Maybe there?s room for more experimentation but if you could suggest a > way to have a better diagnostic? > >> > >> With the different equation sets I?m working with, the condition > numbers estimated with the petsc faq method vary between 10^3 and 10^7. > >> On top of that I have ridiculous fill-in and have to set > -pc_factor_fill to 14, up to 35 (!) sometimes. > >> > >> For our application we need a lot of discretization points in one > spatial direction and I read somewhere that condition number scales with > the square of discretization steps for FD methods. But is there a way to > reduce it in my case? > >> I?m also aware that fill-in should be inevitably expected when you have > a sparse matrix with a banded structure arising from a FDM. But I was > wondering if there?s something more I can do on the numerical side to, on > reduce fill-in and/or help the iterative solver to converge faster? > >> > >> I know my discretized PDE?s + boundary conditions are scaled > consistently with regards to matrix entries. > >> I?m using natural ordering (if my unknowns are a_ij, b_ij the unknown > vector starts with a_00 b_00 a_10 b_10 a_20 b_20 and ends with a_nxny > b_nxny?) but I do not think this has any impact? > >> > >> Thanks for your support, > >> > >> > >> Thibaut > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Fri Sep 21 09:09:31 2018 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 21 Sep 2018 10:09:31 -0400 Subject: [petsc-users] error make with cude on IBM In-Reply-To: References: Message-ID: Thanks, worked. And I get this message, but I set COPTFLAGS="-O2" ... =============================================================================== ***** WARNING: Using default optimization C flags -O You might consider manually setting optimal optimization flags for your system with COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples =============================================================================== On Thu, Sep 20, 2018 at 3:51 PM Balay, Satish wrote: > > /autofs/nccs-svm1_home1/adams/petsc/include/petscmath.h(389): error: > identifier "PETSC_CXX_INLINE" is undefined > > Somehow nvcc is setting '__cplusplus' flag - and causing he errors.. > > change --with-cxx=0 to --with-cxx=mpicxx > > Satish > > --------- > > > > > On Thu, 20 Sep 2018, Mark Adams wrote: > > > I am getting this error on SUMMIT. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Sep 21 09:26:56 2018 From: balay at mcs.anl.gov (Balay, Satish) Date: Fri, 21 Sep 2018 14:26:56 +0000 Subject: [petsc-users] error make with cude on IBM In-Reply-To: References: Message-ID: Can you send the corresponding configure.log? Satish On Fri, 21 Sep 2018, Mark Adams wrote: > Thanks, worked. And I get this message, but I set COPTFLAGS="-O2" ... > > =============================================================================== > ***** > WARNING: Using default optimization C flags -O > You might > consider manually setting optimal optimization flags for your system with > > COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for > examples > =============================================================================== > > > On Thu, Sep 20, 2018 at 3:51 PM Balay, Satish wrote: > > > > /autofs/nccs-svm1_home1/adams/petsc/include/petscmath.h(389): error: > > identifier "PETSC_CXX_INLINE" is undefined > > > > Somehow nvcc is setting '__cplusplus' flag - and causing he errors.. > > > > change --with-cxx=0 to --with-cxx=mpicxx > > > > Satish > > > > --------- > > > > > > > > > > On Thu, 20 Sep 2018, Mark Adams wrote: > > > > > I am getting this error on SUMMIT. > > > > > > > > From mfadams at lbl.gov Fri Sep 21 09:27:30 2018 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 21 Sep 2018 10:27:30 -0400 Subject: [petsc-users] error make with cude on IBM In-Reply-To: References: Message-ID: same one as yesterday I think On Fri, Sep 21, 2018 at 10:27 AM Balay, Satish wrote: > Can you send the corresponding configure.log? > > Satish > > On Fri, 21 Sep 2018, Mark Adams wrote: > > > Thanks, worked. And I get this message, but I set COPTFLAGS="-O2" ... > > > > > =============================================================================== > > > ***** > > WARNING: Using default optimization C flags -O > > You might > > consider manually setting optimal optimization flags for your system with > > > > COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for > > examples > > > =============================================================================== > > > > > > On Thu, Sep 20, 2018 at 3:51 PM Balay, Satish wrote: > > > > > > /autofs/nccs-svm1_home1/adams/petsc/include/petscmath.h(389): error: > > > identifier "PETSC_CXX_INLINE" is undefined > > > > > > Somehow nvcc is setting '__cplusplus' flag - and causing he errors.. > > > > > > change --with-cxx=0 to --with-cxx=mpicxx > > > > > > Satish > > > > > > --------- > > > > > > > > > > > > > > > On Thu, 20 Sep 2018, Mark Adams wrote: > > > > > > > I am getting this error on SUMMIT. > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Sep 21 09:52:42 2018 From: balay at mcs.anl.gov (Balay, Satish) Date: Fri, 21 Sep 2018 14:52:42 +0000 Subject: [petsc-users] error make with cude on IBM In-Reply-To: References: Message-ID: Ah - the same code is used for COPTFLAGS and CUDAOPTFLAGS - but the message printer doesn't know this difference. I think the warning will go away if you also set CUDAOPTFLAGS Will have to see how this can be fixed to print the correct message. Satish On Fri, 21 Sep 2018, Mark Adams wrote: > same one as yesterday I think > > On Fri, Sep 21, 2018 at 10:27 AM Balay, Satish wrote: > > > Can you send the corresponding configure.log? > > > > Satish > > > > On Fri, 21 Sep 2018, Mark Adams wrote: > > > > > Thanks, worked. And I get this message, but I set COPTFLAGS="-O2" ... > > > > > > > > =============================================================================== > > > > > ***** > > > WARNING: Using default optimization C flags -O > > > You might > > > consider manually setting optimal optimization flags for your system with > > > > > > COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for > > > examples > > > > > =============================================================================== > > > > > > > > > On Thu, Sep 20, 2018 at 3:51 PM Balay, Satish wrote: > > > > > > > > /autofs/nccs-svm1_home1/adams/petsc/include/petscmath.h(389): error: > > > > identifier "PETSC_CXX_INLINE" is undefined > > > > > > > > Somehow nvcc is setting '__cplusplus' flag - and causing he errors.. > > > > > > > > change --with-cxx=0 to --with-cxx=mpicxx > > > > > > > > Satish > > > > > > > > --------- > > > > > > > > > > > > > > > > > > > > On Thu, 20 Sep 2018, Mark Adams wrote: > > > > > > > > > I am getting this error on SUMMIT. > > > > > > > > > > > > > > > > > > > > > From phil.tooley at sheffield.ac.uk Mon Sep 24 07:57:33 2018 From: phil.tooley at sheffield.ac.uk (Phil Tooley) Date: Mon, 24 Sep 2018 13:57:33 +0100 Subject: [petsc-users] DMCreateXXXVector vs VecDuplicate Message-ID: <6de11e8d-fa45-7454-28dd-8f00bd5ce64f@sheffield.ac.uk> Hi all, Can I get come clarification regarding? DMCreateXXXVector vs VecDuplicate. The manual says that I can obtain additional vectors with the latter, does this mean that I *must* only call DMCreateXXXVector once each for global and local? I ask because I want multiple instances of an image class all with the same distributed layout. It makes life much tidier if I can just pass the DMDA rather than also having to pass vectors to be copied. Cheers Phil -- Phil Tooley Research Software Engineering University of Sheffield From knepley at gmail.com Mon Sep 24 08:20:59 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 24 Sep 2018 09:20:59 -0400 Subject: [petsc-users] DMCreateXXXVector vs VecDuplicate In-Reply-To: <6de11e8d-fa45-7454-28dd-8f00bd5ce64f@sheffield.ac.uk> References: <6de11e8d-fa45-7454-28dd-8f00bd5ce64f@sheffield.ac.uk> Message-ID: On Mon, Sep 24, 2018 at 8:57 AM Phil Tooley wrote: > Hi all, > > Can I get come clarification regarding DMCreateXXXVector Vectors created from a DM have additional information, such as the DM itself and special View routines. > vs > VecDuplicate. VecDuplicate will retain this special information if present > The manual says that I can obtain additional vectors with > the latter, does this mean that I *must* only call DMCreateXXXVector > once each for global and local? > No, you can call it as much as you want. Thanks, Matt > I ask because I want multiple instances of an image class all with the > same distributed layout. It makes life much tidier if I can just pass > the DMDA rather than also having to pass vectors to be copied. > > Cheers > > > Phil > > -- > Phil Tooley > Research Software Engineering > University of Sheffield > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Mon Sep 24 15:39:14 2018 From: epscodes at gmail.com (Xiangdong) Date: Mon, 24 Sep 2018 16:39:14 -0400 Subject: [petsc-users] number of rows distribution for matrix from MatLoad Message-ID: Hello everyone, I have a quick question on MatLoad. When I use MatLoad to read a binary matrix into AIJ format, how can I specify the distributions of rows? In particular, I want the rows distributed on each processor to be even. More specific, Let's say I have a binary matrix A with 50 rows. I want to read this binary file on np=2. By default, each processor will have 25 rows. If I want the rows on each processor to be even (eg., 26 row on rank 0, and 24 rows on rank 1), how can I achieve this? For this specific matrix, I can do it by specifying m in MatSetSizes. However, for a general matrix A, I do not know the number of rows before calling MatLoad, how can I specify the distribution of the matrix? Thank you. Best, Xiangdong -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 24 15:50:00 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 24 Sep 2018 16:50:00 -0400 Subject: [petsc-users] number of rows distribution for matrix from MatLoad In-Reply-To: References: Message-ID: On Mon, Sep 24, 2018 at 4:41 PM Xiangdong wrote: > Hello everyone, > > I have a quick question on MatLoad. When I use MatLoad to read a binary > matrix into AIJ format, how can I specify the distributions of rows? In > particular, I want the rows distributed on each processor to be even. > The idea is that you pass a new Mat in that you have called MatSetSizes() on. Thanks, Matt > More specific, Let's say I have a binary matrix A with 50 rows. I want to > read this binary file on np=2. By default, each processor will have 25 > rows. If I want the rows on each processor to be even (eg., 26 row on rank > 0, and 24 rows on rank 1), how can I achieve this? For this specific > matrix, I can do it by specifying m in MatSetSizes. However, for a general > matrix A, I do not know the number of rows before calling MatLoad, how can > I specify the distribution of the matrix? > > Thank you. > > Best, > Xiangdong > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From huq2090 at gmail.com Mon Sep 24 20:02:05 2018 From: huq2090 at gmail.com (Fazlul Huq) Date: Mon, 24 Sep 2018 20:02:05 -0500 Subject: [petsc-users] Example 1 of Vector section Message-ID: Hello All, I am looking through the example problems given in PETSc. In the very first example, ex1 of vector section, I don't understand the following part, especially the term "PETSC_SMALL". What does this mean? Here, in the second line we already assign zero to v. What is the meaning of "if" condition then? ierr = VecScale(x,two);CHKERRQ(ierr); ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); v = norm-2.0*PetscSqrtReal((PetscReal)n); if (v > -PETSC_SMALL && v < PETSC_SMALL) v = 0.0; ierr = PetscPrintf(PETSC_COMM_WORLD,"VecScale %g\n",(double)v);CHKERRQ(ierr); All the best. Sincerely, Huq -- Fazlul Huq Graduate Research Assistant Department of Nuclear, Plasma & Radiological Engineering (NPRE) University of Illinois at Urbana-Champaign (UIUC) E-mail: huq2090 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 24 20:26:27 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 24 Sep 2018 21:26:27 -0400 Subject: [petsc-users] Example 1 of Vector section In-Reply-To: References: Message-ID: On Mon, Sep 24, 2018 at 9:02 PM Fazlul Huq wrote: > Hello All, > > I am looking through the example problems given in PETSc. In the very > first example, ex1 of vector section, I don't understand the following > part, especially the term "PETSC_SMALL". What does this mean? > This checks that we get the norm we expect. PETSC_SMALL is a small difference we are willing to accept due to roundoff error. Matt > Here, in the second line we already assign zero to v. What is the meaning > of "if" condition then? > > ierr = VecScale(x,two);CHKERRQ(ierr); > ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); > v = norm-2.0*PetscSqrtReal((PetscReal)n); if (v > -PETSC_SMALL && v < > PETSC_SMALL) v = 0.0; > ierr = PetscPrintf(PETSC_COMM_WORLD,"VecScale > %g\n",(double)v);CHKERRQ(ierr); > > > All the best. > Sincerely, > Huq > -- > > Fazlul Huq > Graduate Research Assistant > Department of Nuclear, Plasma & Radiological Engineering (NPRE) > University of Illinois at Urbana-Champaign (UIUC) > E-mail: huq2090 at gmail.com > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From huq2090 at gmail.com Mon Sep 24 20:29:17 2018 From: huq2090 at gmail.com (Fazlul Huq) Date: Mon, 24 Sep 2018 20:29:17 -0500 Subject: [petsc-users] Example 1 of Vector section In-Reply-To: References: Message-ID: Got it. Thanks a lot. Best, Huq On Mon, Sep 24, 2018 at 8:26 PM Matthew Knepley wrote: > On Mon, Sep 24, 2018 at 9:02 PM Fazlul Huq wrote: > >> Hello All, >> >> I am looking through the example problems given in PETSc. In the very >> first example, ex1 of vector section, I don't understand the following >> part, especially the term "PETSC_SMALL". What does this mean? >> > > This checks that we get the norm we expect. PETSC_SMALL is a small > difference we are willing to accept > due to roundoff error. > > Matt > > >> Here, in the second line we already assign zero to v. What is the meaning >> of "if" condition then? >> >> ierr = VecScale(x,two);CHKERRQ(ierr); >> ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); >> v = norm-2.0*PetscSqrtReal((PetscReal)n); if (v > -PETSC_SMALL && v >> < PETSC_SMALL) v = 0.0; >> ierr = PetscPrintf(PETSC_COMM_WORLD,"VecScale >> %g\n",(double)v);CHKERRQ(ierr); >> >> >> All the best. >> Sincerely, >> Huq >> -- >> >> Fazlul Huq >> Graduate Research Assistant >> Department of Nuclear, Plasma & Radiological Engineering (NPRE) >> University of Illinois at Urbana-Champaign (UIUC) >> E-mail: huq2090 at gmail.com >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- Fazlul Huq Graduate Research Assistant Department of Nuclear, Plasma & Radiological Engineering (NPRE) University of Illinois at Urbana-Champaign (UIUC) E-mail: huq2090 at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From phil.tooley at sheffield.ac.uk Tue Sep 25 05:49:43 2018 From: phil.tooley at sheffield.ac.uk (Phil Tooley) Date: Tue, 25 Sep 2018 11:49:43 +0100 Subject: [petsc-users] Checking if a vector is a localvector of a given DMDA Message-ID: <4d911f2e-cab6-db48-feae-5634051291e4@sheffield.ac.uk> Hi all, Given a vector I know I can get an associated DM (if there is one) by calling VecGetDM, but I need to also be able to check that a) the vector is the localvector of that DM rather than the global b) the DM is a DMDA rather than some other subclass I can't immediately see routines that do what I need, but I am likely missing something obvious. Is there a way to achieve the above? Thanks Phil -- Phil Tooley Research Software Engineering University of Sheffield From knepley at gmail.com Tue Sep 25 05:55:07 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Sep 2018 06:55:07 -0400 Subject: [petsc-users] Checking if a vector is a localvector of a given DMDA In-Reply-To: <4d911f2e-cab6-db48-feae-5634051291e4@sheffield.ac.uk> References: <4d911f2e-cab6-db48-feae-5634051291e4@sheffield.ac.uk> Message-ID: On Tue, Sep 25, 2018 at 6:49 AM Phil Tooley wrote: > Hi all, > > Given a vector I know I can get an associated DM (if there is one) by > calling VecGetDM, but I need to also be able to check that > > a) the vector is the localvector of that DM rather than the global > Get a local vector from the DM and check the sizes. > b) the DM is a DMDA rather than some other subclass > You can do this (PetscObjectTypeCompare), but it is not recommended. Why would you want to do that? Thanks, Matt > I can't immediately see routines that do what I need, but I am likely > missing something obvious. Is there a way to achieve the above? > > Thanks > > Phil > > -- > Phil Tooley > Research Software Engineering > University of Sheffield > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Sep 25 06:02:37 2018 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 25 Sep 2018 12:02:37 +0100 Subject: [petsc-users] Checking if a vector is a localvector of a given DMDA In-Reply-To: <4d911f2e-cab6-db48-feae-5634051291e4@sheffield.ac.uk> References: <4d911f2e-cab6-db48-feae-5634051291e4@sheffield.ac.uk> Message-ID: On Tue, 25 Sep 2018 at 11:49, Phil Tooley wrote: > Hi all, > > Given a vector I know I can get an associated DM (if there is one) by > calling VecGetDM, but I need to also be able to check that > > a) the vector is the localvector of that DM rather than the global > Given the vector, you can check the communicator size via PetscObjectGetComm() https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectGetComm.html and then MPI_Comm_size() If the comm size 1, it is local vector. You can check the size matches your local DMDA space by using DMDAGetGhostCorners() https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetGhostCorners.html and return the quantities m, n, and p. You also need to use DMDAGetInfo() https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetInfo.html The important quantity you want returned is "dof" If m x n x p x dof matches the number returned by VecGetSize() (assuming you know the vector is sequential) then you know the local space will fit within your vector. > > b) the DM is a DMDA rather than some other subclass > See Matt's answer > > I can't immediately see routines that do what I need, but I am likely > missing something obvious. Is there a way to achieve the above? > > Thanks > > Phil > > -- > Phil Tooley > Research Software Engineering > University of Sheffield > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phil.tooley at sheffield.ac.uk Tue Sep 25 06:12:10 2018 From: phil.tooley at sheffield.ac.uk (Phil Tooley) Date: Tue, 25 Sep 2018 12:12:10 +0100 Subject: [petsc-users] Checking if a vector is a localvector of a given DMDA In-Reply-To: References: <4d911f2e-cab6-db48-feae-5634051291e4@sheffield.ac.uk> Message-ID: <01ac5772-bf38-5424-d68c-75d67c7630b2@sheffield.ac.uk> Hi Matt, Thanks for the rapid response.? The reason I want to ensure I have a DMDA is that this is a custom gradient function, the results of which are meaningless for anything but a regular grid. I would rather raise an exception upfront with the user than allow the program to continue with garbage data. Phil On 25/09/18 11:55, Matthew Knepley wrote: > On Tue, Sep 25, 2018 at 6:49 AM Phil Tooley > > wrote: > > Hi all, > > Given a vector I know I can get an associated DM (if there is one) by > calling VecGetDM, but I need to also be able to check that > > a) the vector is the localvector of that DM rather than the global > > > Get a local vector from the DM and check the sizes. > ? > > b) the DM is a DMDA rather than some other subclass > > > You can do this (PetscObjectTypeCompare), but it is not recommended. > Why would > you want to do that? > > ? Thanks, > > ? ? ?Matt > ? > > I can't immediately see routines that do what I need, but I am likely > missing something obvious. Is there a way to achieve the above? > > Thanks > > Phil > > -- > Phil Tooley > Research Software Engineering > University of Sheffield > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- Phil Tooley Research Software Engineering University of Sheffield -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 25 06:20:04 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Sep 2018 07:20:04 -0400 Subject: [petsc-users] Checking if a vector is a localvector of a given DMDA In-Reply-To: References: <4d911f2e-cab6-db48-feae-5634051291e4@sheffield.ac.uk> Message-ID: On Tue, Sep 25, 2018 at 7:03 AM Dave May wrote: > On Tue, 25 Sep 2018 at 11:49, Phil Tooley > wrote: > >> Hi all, >> >> Given a vector I know I can get an associated DM (if there is one) by >> calling VecGetDM, but I need to also be able to check that >> >> a) the vector is the localvector of that DM rather than the global >> > > Given the vector, you can check the communicator size via > PetscObjectGetComm() > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectGetComm.html > and then MPI_Comm_size() > If the comm size 1, it is local vector. > In serial, both vectors have comm size 1. Matt > You can check the size matches your local DMDA space by using > DMDAGetGhostCorners() > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetGhostCorners.html > > and return the quantities m, n, and p. > > You also need to use DMDAGetInfo() > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetInfo.html > > The important quantity you want returned is "dof" > > If m x n x p x dof matches the number returned by VecGetSize() (assuming > you know the vector is sequential) then you know the local space will fit > within your vector. > > > >> >> b) the DM is a DMDA rather than some other subclass >> > > See Matt's answer > > >> >> I can't immediately see routines that do what I need, but I am likely >> missing something obvious. Is there a way to achieve the above? >> >> Thanks >> >> Phil >> >> -- >> Phil Tooley >> Research Software Engineering >> University of Sheffield >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Sep 25 07:07:57 2018 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 25 Sep 2018 14:07:57 +0200 Subject: [petsc-users] Checking if a vector is a localvector of a given DMDA In-Reply-To: References: <4d911f2e-cab6-db48-feae-5634051291e4@sheffield.ac.uk> Message-ID: On Tue, 25 Sep 2018 at 13:20, Matthew Knepley wrote: > On Tue, Sep 25, 2018 at 7:03 AM Dave May wrote: > >> On Tue, 25 Sep 2018 at 11:49, Phil Tooley >> wrote: >> >>> Hi all, >>> >>> Given a vector I know I can get an associated DM (if there is one) by >>> calling VecGetDM, but I need to also be able to check that >>> >>> a) the vector is the localvector of that DM rather than the global >>> >> >> Given the vector, you can check the communicator size via >> PetscObjectGetComm() >> >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectGetComm.html >> and then MPI_Comm_size() >> If the comm size 1, it is local vector. >> > > In serial, both vectors have comm size 1. > Right - and the local and global sizes are the same. My point was to check the comm size first. If it's 1 then you have a candidate for a local vector. Then you'd check the vec global size matches the dmda local size. If the commsize is anything other than 1 then it cannot be a local vector > Matt > > >> You can check the size matches your local DMDA space by using >> DMDAGetGhostCorners() >> >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetGhostCorners.html >> >> and return the quantities m, n, and p. >> >> You also need to use DMDAGetInfo() >> >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetInfo.html >> >> The important quantity you want returned is "dof" >> >> If m x n x p x dof matches the number returned by VecGetSize() (assuming >> you know the vector is sequential) then you know the local space will fit >> within your vector. >> >> >> >>> >>> b) the DM is a DMDA rather than some other subclass >>> >> >> See Matt's answer >> >> >>> >>> I can't immediately see routines that do what I need, but I am likely >>> missing something obvious. Is there a way to achieve the above? >>> >>> Thanks >>> >>> Phil >>> >>> -- >>> Phil Tooley >>> Research Software Engineering >>> University of Sheffield >>> >>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phil.tooley at sheffield.ac.uk Tue Sep 25 07:31:14 2018 From: phil.tooley at sheffield.ac.uk (Phil Tooley) Date: Tue, 25 Sep 2018 13:31:14 +0100 Subject: [petsc-users] Checking if a vector is a localvector of a given DMDA In-Reply-To: References: <4d911f2e-cab6-db48-feae-5634051291e4@sheffield.ac.uk> Message-ID: Thanks both, I now have what I need.? For now I am checking that the vector I am passed has the same local size, global size, and Comm as the vector provided by DMGetLocalVector, mostly because I already have a compatibility check function written.? (I assume this requires a malloc and free behind the scenes) At some point I will likely change to explicitly checking for comm size of one and appropriate global and local sizes based on the DMDA properties instead, for now I want to get to an alpha version I can let people play with. Phil On 25/09/18 13:07, Dave May wrote: > > > On Tue, 25 Sep 2018 at 13:20, Matthew Knepley > wrote: > > On Tue, Sep 25, 2018 at 7:03 AM Dave May > wrote: > > On Tue, 25 Sep 2018 at 11:49, Phil Tooley > > wrote: > > Hi all, > > Given a vector I know I can get an associated DM (if there > is one) by > calling VecGetDM, but I need to also be able to check that > > a) the vector is the localvector of that DM rather than > the global > > > Given the vector, you can check the communicator size via > PetscObjectGetComm() > > ? > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectGetComm.html > and then MPI_Comm_size() > If the comm size 1, it is local vector. > > > In serial, both vectors have comm size 1. > > > Right - and the local and global sizes are the same. > > ?My point was to check the comm size first. If it's 1 then you have a > candidate for a local vector. Then you'd check the vec global size > matches the dmda local size. If the commsize is anything other than 1 > then it cannot be a local vector? > > > ? ?Matt > ? > > You can check the size matches your local DMDA space by using > DMDAGetGhostCorners() > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetGhostCorners.html > > and return the quantities m, n, and p. > > You also need to use? DMDAGetInfo() > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetInfo.html > > The important quantity you want returned is "dof" > > If m x n x p x dof matches the number returned by VecGetSize() > (assuming you know the vector is sequential) then you know the > local space will fit within your vector. > > ? > > > b) the DM is a DMDA rather than some other subclass > > > See Matt's answer? > ? > > > I can't immediately see routines that do what I need, but > I am likely > missing something obvious. Is there a way to achieve the > above? > > Thanks > > Phil > > -- > Phil Tooley > Research Software Engineering > University of Sheffield > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to > which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- Phil Tooley Research Software Engineering University of Sheffield -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Sep 25 07:52:21 2018 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 25 Sep 2018 13:52:21 +0100 Subject: [petsc-users] Checking if a vector is a localvector of a given DMDA In-Reply-To: References: <4d911f2e-cab6-db48-feae-5634051291e4@sheffield.ac.uk> Message-ID: On Tue, 25 Sep 2018 at 13:31, Phil Tooley wrote: > Thanks both, > > I now have what I need. For now I am checking that the vector I am passed > has the same local size, global size, and Comm as the vector provided by > DMGetLocalVector, mostly because I already have a compatibility check > function written. (I assume this requires a malloc and free behind the > scenes) > Not necessarily. The Get/Restore strategy will re-use internally cached vectors. > At some point I will likely change to explicitly checking for comm size of > one and appropriate global and local sizes based on the DMDA properties > instead, for now I want to get to an alpha version I can let people play > with. > > Phil > > On 25/09/18 13:07, Dave May wrote: > > > > On Tue, 25 Sep 2018 at 13:20, Matthew Knepley wrote: > >> On Tue, Sep 25, 2018 at 7:03 AM Dave May wrote: >> >>> On Tue, 25 Sep 2018 at 11:49, Phil Tooley >>> wrote: >>> >>>> Hi all, >>>> >>>> Given a vector I know I can get an associated DM (if there is one) by >>>> calling VecGetDM, but I need to also be able to check that >>>> >>>> a) the vector is the localvector of that DM rather than the global >>>> >>> >>> Given the vector, you can check the communicator size via >>> PetscObjectGetComm() >>> >>> >>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectGetComm.html >>> and then MPI_Comm_size() >>> If the comm size 1, it is local vector. >>> >> >> In serial, both vectors have comm size 1. >> > > Right - and the local and global sizes are the same. > > My point was to check the comm size first. If it's 1 then you have a > candidate for a local vector. Then you'd check the vec global size matches > the dmda local size. If the commsize is anything other than 1 then it > cannot be a local vector > > >> Matt >> >> >>> You can check the size matches your local DMDA space by using >>> DMDAGetGhostCorners() >>> >>> >>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetGhostCorners.html >>> >>> and return the quantities m, n, and p. >>> >>> You also need to use DMDAGetInfo() >>> >>> >>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetInfo.html >>> >>> The important quantity you want returned is "dof" >>> >>> If m x n x p x dof matches the number returned by VecGetSize() (assuming >>> you know the vector is sequential) then you know the local space will fit >>> within your vector. >>> >>> >>> >>>> >>>> b) the DM is a DMDA rather than some other subclass >>>> >>> >>> See Matt's answer >>> >>> >>>> >>>> I can't immediately see routines that do what I need, but I am likely >>>> missing something obvious. Is there a way to achieve the above? >>>> >>>> Thanks >>>> >>>> Phil >>>> >>>> -- >>>> Phil Tooley >>>> Research Software Engineering >>>> University of Sheffield >>>> >>>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > -- > Phil Tooley > Research Software Engineering > University of Sheffield > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yjwu16 at gmail.com Wed Sep 26 11:00:56 2018 From: yjwu16 at gmail.com (Yingjie Wu) Date: Thu, 27 Sep 2018 00:00:56 +0800 Subject: [petsc-users] Implementation of Power Iteration method in PETSc Message-ID: Dear Petsc developer: Hi, Thank you very much for your previous reply, they really helped me a lot. I've been doing neutron calculations recently, and I want to use the power Iteration method to calculate the neutron flux, essentially to solve linear equations, but after each solution is done, a scalar *K* is updated to form a new right end term *b*, and the next linear equation system is solved until the convergence criterion is satisfied. The flow of the source iteration algorithm is as follows: 1: *?*(0) = arbitrary nonzero vector 2: K(0) = arbitrary nonzero constant 3: *for* n = 1; 2; 3; ::: do 4: *b *= 1 / K(n-1) *F * ?(n-1) 5: *?*(n) * M = *b* 6: K(n) = k(n-1)* (F * *?*(n) ,F *?*(n) ) / (F *?*(n-1) ,F *?* (n-1) ) 7: check convergence of eigenvalue and eigenvector 8: *end for* (F , M are the coefficient matrix.) The difficulty is that I need to set up an extra loop to regulate KSP. Does PETSc have such an example? How do I implement this algorithm in PETsc? Please tell me the key functions if possible. Thanks, Yingjie -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Wed Sep 26 11:27:00 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Wed, 26 Sep 2018 10:27:00 -0600 Subject: [petsc-users] Implementation of Power Iteration method in PETSc In-Reply-To: References: Message-ID: I have implemented this algorithm in SLEPC. Take a look at this example http://slepc.upv.es/documentation/current/src/eps/examples/tutorials/ex34.c.html The motivation of the algorithm is also for neutron calculations (a moose-based application). Fande, On Wed, Sep 26, 2018 at 10:02 AM Yingjie Wu wrote: > Dear Petsc developer: > Hi, > > Thank you very much for your previous reply, they really helped me a lot. > > I've been doing neutron calculations recently, and I want to use the power > Iteration method to calculate the neutron flux, essentially to solve linear > equations, but after each solution is done, a scalar *K* is updated to > form a new right end term *b*, and the next linear equation system is > solved until the convergence criterion is satisfied. The flow of the source > iteration algorithm is as follows: > > > 1: *?*(0) = arbitrary nonzero vector 2: K(0) = arbitrary nonzero constant > 3: *for* n = 1; 2; 3; ::: do 4: *b *= 1 / K(n-1) *F * ?(n-1) 5: *?*(n) > * M = *b* 6: K(n) = k(n-1)* (F * *?*(n) ,F *?*(n) ) / (F *?*(n-1) ,F > *?*(n-1) ) > > 7: check convergence of eigenvalue and eigenvector 8: *end for* > > > (F , M are the coefficient matrix.) > > > The difficulty is that I need to set up an extra loop to regulate KSP. > Does PETSc have such an example? How do I implement this algorithm in > PETsc? Please tell me the key functions if possible. > > > Thanks, > > Yingjie > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yjwu16 at gmail.com Wed Sep 26 21:17:22 2018 From: yjwu16 at gmail.com (Yingjie Wu) Date: Thu, 27 Sep 2018 10:17:22 +0800 Subject: [petsc-users] Implementation of Power Iteration method in PETSc In-Reply-To: References: Message-ID: Thank you, Fande. I've seen you in moose usergroup before. I've just learned about SLEPC, and I wonder if I want to do neutron eigenvalue calculations and use finite difference to discrete grids, would it be difficult to implement it in SLEPC? Is there such an example(finite difference + linear eigenvalue)?Because I don't know much about finite element.Or am I still using a loop of KSP in PETSc?I'm a newcomer to petsc, please give me some advice Thanks, Yingjie Fande Kong ?2018?9?27??? ??12:25??? > I have implemented this algorithm in SLEPC. Take a look at this example > http://slepc.upv.es/documentation/current/src/eps/examples/tutorials/ex34.c.html > > The motivation of the algorithm is also for neutron calculations (a > moose-based application). > > Fande, > > On Wed, Sep 26, 2018 at 10:02 AM Yingjie Wu wrote: > >> Dear Petsc developer: >> Hi, >> >> Thank you very much for your previous reply, they really helped me a >> lot. >> >> I've been doing neutron calculations recently, and I want to use the >> power Iteration method to calculate the neutron flux, essentially to solve >> linear equations, but after each solution is done, a scalar *K* is >> updated to form a new right end term *b*, and the next linear equation >> system is solved until the convergence criterion is satisfied. The flow of >> the source iteration algorithm is as follows: >> >> >> 1: *?*(0) = arbitrary nonzero vector 2: K(0) = arbitrary nonzero >> constant 3: *for* n = 1; 2; 3; ::: do 4: *b *= 1 / K(n-1) *F * ?(n-1) >> 5: *?*(n) * M = *b* 6: K(n) = k(n-1)* (F * *?*(n) ,F *?*(n) ) / (F >> *?*(n-1) ,F *?*(n-1) ) >> >> 7: check convergence of eigenvalue and eigenvector 8: *end for* >> >> >> (F , M are the coefficient matrix.) >> >> >> The difficulty is that I need to set up an extra loop to regulate KSP. >> Does PETSc have such an example? How do I implement this algorithm in >> PETsc? Please tell me the key functions if possible. >> >> >> Thanks, >> >> Yingjie >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Thu Sep 27 10:12:26 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Thu, 27 Sep 2018 09:12:26 -0600 Subject: [petsc-users] Implementation of Power Iteration method in PETSc In-Reply-To: References: Message-ID: Hi Yingjie, For finite difference method, there are a lot of example in PETSc. For instance, https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex29.c.html For linear eigenvalue problems, there is a list of examples at http://slepc.upv.es/documentation/current/src/eps/examples/tutorials/index.html Before you go to implement your inver power algorithm, I would like to suggest you go though PETSc manual and SLEPc manual. https://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf and http://slepc.upv.es/documentation/slepc.pdf SLEPc has both nonlinear and linear inverse power iteration methods, and you can pick any version you want. It is a good idea to have big picture before you do anything using PETSc or SLEPc. Fande, On Wed, Sep 26, 2018 at 8:17 PM Yingjie Wu wrote: > Thank you, Fande. > > I've seen you in moose usergroup before. > > > I've just learned about SLEPC, and I wonder if I want to do neutron > eigenvalue calculations and use finite difference to discrete grids, would > it be difficult to implement it in SLEPC? Is there such an example(finite > difference + linear eigenvalue)?Because I don't know much about finite > element.Or am I still using a loop of KSP in PETSc?I'm a newcomer to petsc, > please give me some advice > > > Thanks, > > Yingjie > > > Fande Kong ?2018?9?27??? ??12:25??? > >> I have implemented this algorithm in SLEPC. Take a look at this example >> http://slepc.upv.es/documentation/current/src/eps/examples/tutorials/ex34.c.html >> >> The motivation of the algorithm is also for neutron calculations (a >> moose-based application). >> >> Fande, >> >> On Wed, Sep 26, 2018 at 10:02 AM Yingjie Wu wrote: >> >>> Dear Petsc developer: >>> Hi, >>> >>> Thank you very much for your previous reply, they really helped me a >>> lot. >>> >>> I've been doing neutron calculations recently, and I want to use the >>> power Iteration method to calculate the neutron flux, essentially to solve >>> linear equations, but after each solution is done, a scalar *K* is >>> updated to form a new right end term *b*, and the next linear equation >>> system is solved until the convergence criterion is satisfied. The flow of >>> the source iteration algorithm is as follows: >>> >>> >>> 1: *?*(0) = arbitrary nonzero vector 2: K(0) = arbitrary nonzero >>> constant 3: *for* n = 1; 2; 3; ::: do 4: *b *= 1 / K(n-1) *F * ?(n-1) >>> 5: *?*(n) * M = *b* 6: K(n) = k(n-1)* (F * *?*(n) ,F *?*(n) ) / (F >>> *?*(n-1) ,F *?*(n-1) ) >>> >>> 7: check convergence of eigenvalue and eigenvector 8: *end for* >>> >>> >>> (F , M are the coefficient matrix.) >>> >>> >>> The difficulty is that I need to set up an extra loop to regulate KSP. >>> Does PETSc have such an example? How do I implement this algorithm in >>> PETsc? Please tell me the key functions if possible. >>> >>> >>> Thanks, >>> >>> Yingjie >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.werner at dlr.de Fri Sep 28 02:23:27 2018 From: michael.werner at dlr.de (Michael Werner) Date: Fri, 28 Sep 2018 09:23:27 +0200 Subject: [petsc-users] Usage of AMG as preconditioner Message-ID: Hello, I'm having trouble with getting the AMG preconditioners working. I tried all of them (gamg, ml, hypre-boomeramg), with varying degrees of "success": - GAMG: CMD options: -ksp_rtol 1e-8 -ksp_monitor_true_residual -ksp_max_it 20 -ksp_type fgmres -pc_type gamg -pc_gamg_sym_graph TRUE -pc_gamg_agg_nsmooths 1 -ksp_view always crashes with the following error: Error: error code 77 [0] KSPSolve() line 780 in /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c [0] KSPSolve_GMRES() line 233 in /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/gmres/gmres.c [0] KSPInitialResidual() line 67 in /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itres.c [0] KSP_PCApply() line 281 in /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/include/petsc/private/kspimpl.h [0] PCApply() line 462 in /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c [0] PCApply_MG() line 377 in /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c [0] PCMGMCycle_Private() line 20 in /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c [0] KSPSolve() line 780 in /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c [0] KSPSolve_Chebyshev() line 381 in /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/cheby/cheby.c [0] Petsc has generated inconsistent data [0] Eigen estimator failed: DIVERGED_NANORINF at iteration 0 When I'm using a different solver for -mg_levels_ksp_type, such as gmres, GAMG no longer crashes, but I don't see convergence of the problem (convergence history and ksp_view output are attached below). - Hypre With the default settings, BoomerAMG just returns a Vector of all zeros after one iteration. When I change the relaxation type -pc_hypre_boomeramg_relax_type_all to Jacobi, I get similar results than with GAMG: the solver works without errors, but doesn't converge. The output for Hypre is also attached below. - ML With default settings the result is just like Boomeramg: a vector of all zeros after one iteration. When I change -mg_levels_ksp_type the behaviour is identical to GAMG. Since none of the packages worked, I'm assuming that the error lies with me/ my code, so I'll give a short overview over what I'm trying to do. The matrix I'm trying to precondition is the Jacobian of a flow field originating from an unstructured finite-volume CFD code. It has a blocked structure as each node of the original mesh holds several variables (pressure, density, velocities). However, I'm not using a DM-Plex since I get the assembled Jacobian in binary format from the external CFD code. When I'm using direct (lu) I get correct results, so the basic code should be fine. However, for larger problems lu is not feasible due the very large memory requirements, therefore I wanted to switch over to multigrid. Currently, PETSc has no information about the original geometry. I tried using setCoordinates, but for Hypre and ml it didn't make a difference, and GAMG crashed with an error: [0] PCSetCoordinates() line 1883 in /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c [0] PCSetCoordinates_AGG() line 199 in /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/gamg/agg.c [0] Petsc has generated inconsistent data [0] Don't know how to create null space for ndm=2, ndf=4. Use MatSetNearNullSpace. I would be glad if you could give me some advice on how to deal with this. Kind regards, Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: GAMG_KSP_view Type: application/octet-stream Size: 8847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Hypre_KSP_view Type: application/octet-stream Size: 7364 bytes Desc: not available URL: From knepley at gmail.com Fri Sep 28 05:47:58 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 28 Sep 2018 06:47:58 -0400 Subject: [petsc-users] Usage of AMG as preconditioner In-Reply-To: References: Message-ID: On Fri, Sep 28, 2018 at 3:23 AM Michael Werner wrote: > Hello, > > I'm having trouble with getting the AMG preconditioners working. I > tried all of them (gamg, ml, hypre-boomeramg), with varying > degrees of "success": > > - GAMG: > CMD options: -ksp_rtol 1e-8 -ksp_monitor_true_residual -ksp_max_it > 20 -ksp_type fgmres -pc_type gamg -pc_gamg_sym_graph TRUE > -pc_gamg_agg_nsmooths 1 -ksp_view > always crashes with the following error: > Error: error code 77 > [0] KSPSolve() line 780 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c > [0] KSPSolve_GMRES() line 233 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/gmres/gmres.c > [0] KSPInitialResidual() line 67 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itres.c > [0] KSP_PCApply() line 281 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/include/petsc/private/kspimpl.h > [0] PCApply() line 462 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c > [0] PCApply_MG() line 377 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c > [0] PCMGMCycle_Private() line 20 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c > [0] KSPSolve() line 780 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c > [0] KSPSolve_Chebyshev() line 381 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/cheby/cheby.c > [0] Petsc has generated inconsistent data > [0] Eigen estimator failed: DIVERGED_NANORINF at iteration 0 > > When I'm using a different solver for -mg_levels_ksp_type, such as > gmres, GAMG no longer crashes, but I don't see convergence of the > problem (convergence history and ksp_view output are attached > below). > It uses unpreconditioned GMRES to estimate spectral bounds for the operator before using a Chebychev smoother. If your matrix does not have a nice, connected, positive spectrum, Chebychev will not work. However, the fact that you get DIVERGED_NANORINF in the estimator tells me that you have a problem in the matrix. > - Hypre > With the default settings, BoomerAMG just returns a Vector of all > zeros after one iteration. > When I change the relaxation type > -pc_hypre_boomeramg_relax_type_all to Jacobi, I get similar > results than with GAMG: the solver works without errors, but > doesn't converge. The output for Hypre is also attached below. > > - ML > With default settings the result is just like Boomeramg: a vector > of all zeros after one iteration. > When I change -mg_levels_ksp_type the behaviour is identical to > GAMG. > > > Since none of the packages worked, I'm assuming that the error > lies with me/ my code, It looks like a value in the matrix might be bad. > so I'll give a short overview over what I'm > trying to do. > The matrix I'm trying to precondition is the Jacobian of a flow > field originating from an unstructured finite-volume CFD code. It > has a blocked structure as each node of the original mesh holds > several variables (pressure, density, velocities). However, I'm > not using a DM-Plex since I get the assembled Jacobian in binary > format from the external CFD code. > When I'm using direct (lu) I get correct results, so the basic > code should be fine. So LU works for the same problem that gave the NANORINF in GMRES? I would recommend using a Manufactured Solution to check the operation. Then you can start by feeding in the exact solution, and see that nothing fails. Also, I would make a very small test case, so that you can send the matrix and/or code to us to check. > However, for larger problems lu is not > feasible due the very large memory requirements, therefore I > wanted to switch over to multigrid. > > Currently, PETSc has no information about the original geometry. I > tried using setCoordinates, but for Hypre and ml it didn't make a > difference, and GAMG crashed with an error: > This is only an optimization, and currently GAMG does not know what to do for multiple fields (it handles single field problems by building the nullspace of the symbol for the coarse grid). You could provide this when you want to optimize things. Thanks, Matt > [0] PCSetCoordinates() line 1883 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c > [0] PCSetCoordinates_AGG() line 199 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/gamg/agg.c > [0] Petsc has generated inconsistent data > [0] Don't know how to create null space for ndm=2, ndf=4. Use > MatSetNearNullSpace. > > I would be glad if you could give me some advice on how to deal > with this. > > Kind regards, > Michael > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.werner at dlr.de Fri Sep 28 06:43:23 2018 From: michael.werner at dlr.de (Michael Werner) Date: Fri, 28 Sep 2018 13:43:23 +0200 Subject: [petsc-users] Usage of AMG as preconditioner In-Reply-To: References: Message-ID: Matthew Knepley writes: > On Fri, Sep 28, 2018 at 3:23 AM Michael Werner > > wrote: > >> Hello, >> >> I'm having trouble with getting the AMG preconditioners >> working. I >> tried all of them (gamg, ml, hypre-boomeramg), with varying >> degrees of "success": >> >> - GAMG: >> CMD options: -ksp_rtol 1e-8 -ksp_monitor_true_residual >> -ksp_max_it >> 20 -ksp_type fgmres -pc_type gamg -pc_gamg_sym_graph TRUE >> -pc_gamg_agg_nsmooths 1 -ksp_view >> always crashes with the following error: >> Error: error code 77 >> [0] KSPSolve() line 780 in >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c >> [0] KSPSolve_GMRES() line 233 in >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/gmres/gmres.c >> [0] KSPInitialResidual() line 67 in >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itres.c >> [0] KSP_PCApply() line 281 in >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/include/petsc/private/kspimpl.h >> [0] PCApply() line 462 in >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c >> [0] PCApply_MG() line 377 in >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c >> [0] PCMGMCycle_Private() line 20 in >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c >> [0] KSPSolve() line 780 in >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c >> [0] KSPSolve_Chebyshev() line 381 in >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/cheby/cheby.c >> [0] Petsc has generated inconsistent data >> [0] Eigen estimator failed: DIVERGED_NANORINF at iteration 0 >> >> When I'm using a different solver for -mg_levels_ksp_type, such >> as >> gmres, GAMG no longer crashes, but I don't see convergence of >> the >> problem (convergence history and ksp_view output are attached >> below). >> > > It uses unpreconditioned GMRES to estimate spectral bounds for > the operator > before using a Chebychev smoother. > If your matrix does not have a nice, connected, positive > spectrum, > Chebychev will not work. However, the fact that > you get DIVERGED_NANORINF in the estimator tells me that you > have a problem > in the matrix. > The error above (DIVERGED_NANORINF) only appears for -mg_levels_ksp_type chebyshev. When I use GMRES (-mg_levels_ksp_type gmres) there are no errors, the KSP just never converges. > >> - Hypre >> With the default settings, BoomerAMG just returns a Vector of >> all >> zeros after one iteration. >> When I change the relaxation type >> -pc_hypre_boomeramg_relax_type_all to Jacobi, I get similar >> results than with GAMG: the solver works without errors, but >> doesn't converge. The output for Hypre is also attached below. >> >> - ML >> With default settings the result is just like Boomeramg: a >> vector >> of all zeros after one iteration. >> When I change -mg_levels_ksp_type the behaviour is identical to >> GAMG. >> >> >> Since none of the packages worked, I'm assuming that the error >> lies with me/ my code, > > > It looks like a value in the matrix might be bad. > > >> so I'll give a short overview over what I'm >> trying to do. >> The matrix I'm trying to precondition is the Jacobian of a flow >> field originating from an unstructured finite-volume CFD >> code. It >> has a blocked structure as each node of the original mesh holds >> several variables (pressure, density, velocities). However, I'm >> not using a DM-Plex since I get the assembled Jacobian in >> binary >> format from the external CFD code. >> When I'm using direct (lu) I get correct results, so the basic >> code should be fine. > > > So LU works for the same problem that gave the NANORINF in > GMRES? > > I would recommend using a Manufactured Solution to check the > operation. Then > you can start by feeding in the exact solution, and see that > nothing fails. > Also, I > would make a very small test case, so that you can send the > matrix and/or > code > to us to check. > Yes, LU works for the same matrix/ problem. I didn't change anything besides using -pc_type lu instead of -pc_type gamg. What do you mean by "Manufactured Solution"? Concerning the test case: I'll set one up and send you the matrix in binary format. > >> However, for larger problems lu is not >> feasible due the very large memory requirements, therefore I >> wanted to switch over to multigrid. >> >> Currently, PETSc has no information about the original >> geometry. I >> tried using setCoordinates, but for Hypre and ml it didn't make >> a >> difference, and GAMG crashed with an error: >> > > This is only an optimization, and currently GAMG does not know > what to > do for multiple fields (it handles single field problems by > building the > nullspace of the symbol for the coarse grid). You could provide > this when > you want to optimize things. > > Thanks, > > Matt Ok, thanks for the clarification. In that case I won't bother with it for now. No need to optimize the code before the other problems are solved. Thank you for your answer! Kind regards, Michael > > >> [0] PCSetCoordinates() line 1883 in >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c >> [0] PCSetCoordinates_AGG() line 199 in >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/gamg/agg.c >> [0] Petsc has generated inconsistent data >> [0] Don't know how to create null space for ndm=2, ndf=4. Use >> MatSetNearNullSpace. >> >> I would be glad if you could give me some advice on how to deal >> with this. >> >> Kind regards, >> Michael >> >> From knepley at gmail.com Fri Sep 28 06:55:08 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 28 Sep 2018 07:55:08 -0400 Subject: [petsc-users] Usage of AMG as preconditioner In-Reply-To: References: Message-ID: On Fri, Sep 28, 2018 at 7:43 AM Michael Werner wrote: > > Matthew Knepley writes: > > > On Fri, Sep 28, 2018 at 3:23 AM Michael Werner > > > > wrote: > > > >> Hello, > >> > >> I'm having trouble with getting the AMG preconditioners > >> working. I > >> tried all of them (gamg, ml, hypre-boomeramg), with varying > >> degrees of "success": > >> > >> - GAMG: > >> CMD options: -ksp_rtol 1e-8 -ksp_monitor_true_residual > >> -ksp_max_it > >> 20 -ksp_type fgmres -pc_type gamg -pc_gamg_sym_graph TRUE > >> -pc_gamg_agg_nsmooths 1 -ksp_view > >> always crashes with the following error: > >> Error: error code 77 > >> [0] KSPSolve() line 780 in > >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c > >> [0] KSPSolve_GMRES() line 233 in > >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/gmres/gmres.c > >> [0] KSPInitialResidual() line 67 in > >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itres.c > >> [0] KSP_PCApply() line 281 in > >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/include/petsc/private/kspimpl.h > >> [0] PCApply() line 462 in > >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c > >> [0] PCApply_MG() line 377 in > >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c > >> [0] PCMGMCycle_Private() line 20 in > >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c > >> [0] KSPSolve() line 780 in > >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c > >> [0] KSPSolve_Chebyshev() line 381 in > >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/cheby/cheby.c > >> [0] Petsc has generated inconsistent data > >> [0] Eigen estimator failed: DIVERGED_NANORINF at iteration 0 > >> > >> When I'm using a different solver for -mg_levels_ksp_type, such > >> as > >> gmres, GAMG no longer crashes, but I don't see convergence of > >> the > >> problem (convergence history and ksp_view output are attached > >> below). > >> > > > > It uses unpreconditioned GMRES to estimate spectral bounds for > > the operator > > before using a Chebychev smoother. > > If your matrix does not have a nice, connected, positive > > spectrum, > > Chebychev will not work. However, the fact that > > you get DIVERGED_NANORINF in the estimator tells me that you > > have a problem > > in the matrix. > > > > The error above (DIVERGED_NANORINF) only appears for > -mg_levels_ksp_type chebyshev. When I use GMRES > (-mg_levels_ksp_type gmres) there are no errors, the KSP just > never converges. > > > > >> - Hypre > >> With the default settings, BoomerAMG just returns a Vector of > >> all > >> zeros after one iteration. > >> When I change the relaxation type > >> -pc_hypre_boomeramg_relax_type_all to Jacobi, I get similar > >> results than with GAMG: the solver works without errors, but > >> doesn't converge. The output for Hypre is also attached below. > >> > >> - ML > >> With default settings the result is just like Boomeramg: a > >> vector > >> of all zeros after one iteration. > >> When I change -mg_levels_ksp_type the behaviour is identical to > >> GAMG. > >> > >> > >> Since none of the packages worked, I'm assuming that the error > >> lies with me/ my code, > > > > > > It looks like a value in the matrix might be bad. > > > > > >> so I'll give a short overview over what I'm > >> trying to do. > >> The matrix I'm trying to precondition is the Jacobian of a flow > >> field originating from an unstructured finite-volume CFD > >> code. Compressible or incompressible? Mstt > It > >> has a blocked structure as each node of the original mesh holds > >> several variables (pressure, density, velocities). However, I'm > >> not using a DM-Plex since I get the assembled Jacobian in > >> binary > >> format from the external CFD code. > >> When I'm using direct (lu) I get correct results, so the basic > >> code should be fine. > > > > > > So LU works for the same problem that gave the NANORINF in > > GMRES? > > > > I would recommend using a Manufactured Solution to check the > > operation. Then > > you can start by feeding in the exact solution, and see that > > nothing fails. > > Also, I > > would make a very small test case, so that you can send the > > matrix and/or > > code > > to us to check. > > > > Yes, LU works for the same matrix/ problem. I didn't change > anything besides using -pc_type lu instead of -pc_type gamg. > What do you mean by "Manufactured Solution"? > Concerning the test case: I'll set one up and send you the matrix > in binary format. > > > > >> However, for larger problems lu is not > >> feasible due the very large memory requirements, therefore I > >> wanted to switch over to multigrid. > >> > >> Currently, PETSc has no information about the original > >> geometry. I > >> tried using setCoordinates, but for Hypre and ml it didn't make > >> a > >> difference, and GAMG crashed with an error: > >> > > > > This is only an optimization, and currently GAMG does not know > > what to > > do for multiple fields (it handles single field problems by > > building the > > nullspace of the symbol for the coarse grid). You could provide > > this when > > you want to optimize things. > > > > Thanks, > > > > Matt > > Ok, thanks for the clarification. In that case I won't bother with > it for now. No need to optimize the code before the other problems > are solved. > > Thank you for your answer! > > Kind regards, > Michael > > > > > >> [0] PCSetCoordinates() line 1883 in > >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c > >> [0] PCSetCoordinates_AGG() line 199 in > >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/gamg/agg.c > >> [0] Petsc has generated inconsistent data > >> [0] Don't know how to create null space for ndm=2, ndf=4. Use > >> MatSetNearNullSpace. > >> > >> I would be glad if you could give me some advice on how to deal > >> with this. > >> > >> Kind regards, > >> Michael > >> > >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.werner at dlr.de Fri Sep 28 07:13:24 2018 From: michael.werner at dlr.de (Michael Werner) Date: Fri, 28 Sep 2018 14:13:24 +0200 Subject: [petsc-users] Usage of AMG as preconditioner In-Reply-To: References: Message-ID: Matthew Knepley writes: > On Fri, Sep 28, 2018 at 7:43 AM Michael Werner > > wrote: > >> >> Matthew Knepley writes: >> >> > On Fri, Sep 28, 2018 at 3:23 AM Michael Werner >> > >> > wrote: >> > >> >> Hello, >> >> >> >> I'm having trouble with getting the AMG preconditioners >> >> working. I >> >> tried all of them (gamg, ml, hypre-boomeramg), with varying >> >> degrees of "success": >> >> >> >> - GAMG: >> >> CMD options: -ksp_rtol 1e-8 -ksp_monitor_true_residual >> >> -ksp_max_it >> >> 20 -ksp_type fgmres -pc_type gamg -pc_gamg_sym_graph TRUE >> >> -pc_gamg_agg_nsmooths 1 -ksp_view >> >> always crashes with the following error: >> >> Error: error code 77 >> >> [0] KSPSolve() line 780 in >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c >> >> [0] KSPSolve_GMRES() line 233 in >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/gmres/gmres.c >> >> [0] KSPInitialResidual() line 67 in >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itres.c >> >> [0] KSP_PCApply() line 281 in >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/include/petsc/private/kspimpl.h >> >> [0] PCApply() line 462 in >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c >> >> [0] PCApply_MG() line 377 in >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c >> >> [0] PCMGMCycle_Private() line 20 in >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c >> >> [0] KSPSolve() line 780 in >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c >> >> [0] KSPSolve_Chebyshev() line 381 in >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/cheby/cheby.c >> >> [0] Petsc has generated inconsistent data >> >> [0] Eigen estimator failed: DIVERGED_NANORINF at iteration 0 >> >> >> >> When I'm using a different solver for -mg_levels_ksp_type, >> >> such >> >> as >> >> gmres, GAMG no longer crashes, but I don't see convergence >> >> of >> >> the >> >> problem (convergence history and ksp_view output are >> >> attached >> >> below). >> >> >> > >> > It uses unpreconditioned GMRES to estimate spectral bounds >> > for >> > the operator >> > before using a Chebychev smoother. >> > If your matrix does not have a nice, connected, positive >> > spectrum, >> > Chebychev will not work. However, the fact that >> > you get DIVERGED_NANORINF in the estimator tells me that you >> > have a problem >> > in the matrix. >> > >> >> The error above (DIVERGED_NANORINF) only appears for >> -mg_levels_ksp_type chebyshev. When I use GMRES >> (-mg_levels_ksp_type gmres) there are no errors, the KSP just >> never converges. >> >> > >> >> - Hypre >> >> With the default settings, BoomerAMG just returns a Vector >> >> of >> >> all >> >> zeros after one iteration. >> >> When I change the relaxation type >> >> -pc_hypre_boomeramg_relax_type_all to Jacobi, I get similar >> >> results than with GAMG: the solver works without errors, but >> >> doesn't converge. The output for Hypre is also attached >> >> below. >> >> >> >> - ML >> >> With default settings the result is just like Boomeramg: a >> >> vector >> >> of all zeros after one iteration. >> >> When I change -mg_levels_ksp_type the behaviour is identical >> >> to >> >> GAMG. >> >> >> >> >> >> Since none of the packages worked, I'm assuming that the >> >> error >> >> lies with me/ my code, >> > >> > >> > It looks like a value in the matrix might be bad. >> > >> > >> >> so I'll give a short overview over what I'm >> >> trying to do. >> >> The matrix I'm trying to precondition is the Jacobian of a >> >> flow >> >> field originating from an unstructured finite-volume CFD >> >> code. > > > Compressible or incompressible? > > Mstt > The matrix is generated by a compressible CFD code. The current test case is quasi-incompressible due to very low flow velocity (flow around a cylinder). Kind regards, Michael > >> It >> >> has a blocked structure as each node of the original mesh >> >> holds >> >> several variables (pressure, density, velocities). However, >> >> I'm >> >> not using a DM-Plex since I get the assembled Jacobian in >> >> binary >> >> format from the external CFD code. >> >> When I'm using direct (lu) I get correct results, so the >> >> basic >> >> code should be fine. >> > >> > >> > So LU works for the same problem that gave the NANORINF in >> > GMRES? >> > >> > I would recommend using a Manufactured Solution to check the >> > operation. Then >> > you can start by feeding in the exact solution, and see that >> > nothing fails. >> > Also, I >> > would make a very small test case, so that you can send the >> > matrix and/or >> > code >> > to us to check. >> > >> >> Yes, LU works for the same matrix/ problem. I didn't change >> anything besides using -pc_type lu instead of -pc_type gamg. >> What do you mean by "Manufactured Solution"? >> Concerning the test case: I'll set one up and send you the >> matrix >> in binary format. >> >> > >> >> However, for larger problems lu is not >> >> feasible due the very large memory requirements, therefore I >> >> wanted to switch over to multigrid. >> >> >> >> Currently, PETSc has no information about the original >> >> geometry. I >> >> tried using setCoordinates, but for Hypre and ml it didn't >> >> make >> >> a >> >> difference, and GAMG crashed with an error: >> >> >> > >> > This is only an optimization, and currently GAMG does not >> > know >> > what to >> > do for multiple fields (it handles single field problems by >> > building the >> > nullspace of the symbol for the coarse grid). You could >> > provide >> > this when >> > you want to optimize things. >> > >> > Thanks, >> > >> > Matt >> >> Ok, thanks for the clarification. In that case I won't bother >> with >> it for now. No need to optimize the code before the other >> problems >> are solved. >> >> Thank you for your answer! >> >> Kind regards, >> Michael >> > >> > >> >> [0] PCSetCoordinates() line 1883 in >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c >> >> [0] PCSetCoordinates_AGG() line 199 in >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/gamg/agg.c >> >> [0] Petsc has generated inconsistent data >> >> [0] Don't know how to create null space for ndm=2, ndf=4. >> >> Use >> >> MatSetNearNullSpace. >> >> >> >> I would be glad if you could give me some advice on how to >> >> deal >> >> with this. >> >> >> >> Kind regards, >> >> Michael >> >> >> >> >> >> -- ____________________________________________________ Deutsches Zentrum f?r Luft- und Raumfahrt e.V. (DLR) Institut f?r Aerodynamik und Str?mungstechnik | Bunsenstr. 10 | 37073 G?ttingen Michael Werner Telefon 0551 709-2627 | Telefax 0551 709-2811 | Michael.Werner at dlr.de DLR.de From knepley at gmail.com Fri Sep 28 07:18:51 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 28 Sep 2018 08:18:51 -0400 Subject: [petsc-users] Usage of AMG as preconditioner In-Reply-To: References: Message-ID: On Fri, Sep 28, 2018 at 8:13 AM Michael Werner wrote: > > Matthew Knepley writes: > > > On Fri, Sep 28, 2018 at 7:43 AM Michael Werner > > > > wrote: > > > >> > >> Matthew Knepley writes: > >> > >> > On Fri, Sep 28, 2018 at 3:23 AM Michael Werner > >> > > >> > wrote: > >> > > >> >> Hello, > >> >> > >> >> I'm having trouble with getting the AMG preconditioners > >> >> working. I > >> >> tried all of them (gamg, ml, hypre-boomeramg), with varying > >> >> degrees of "success": > >> >> > >> >> - GAMG: > >> >> CMD options: -ksp_rtol 1e-8 -ksp_monitor_true_residual > >> >> -ksp_max_it > >> >> 20 -ksp_type fgmres -pc_type gamg -pc_gamg_sym_graph TRUE > >> >> -pc_gamg_agg_nsmooths 1 -ksp_view > >> >> always crashes with the following error: > >> >> Error: error code 77 > >> >> [0] KSPSolve() line 780 in > >> >> > >> >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c > >> >> [0] KSPSolve_GMRES() line 233 in > >> >> > >> >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/gmres/gmres.c > >> >> [0] KSPInitialResidual() line 67 in > >> >> > >> >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itres.c > >> >> [0] KSP_PCApply() line 281 in > >> >> > >> >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/include/petsc/private/kspimpl.h > >> >> [0] PCApply() line 462 in > >> >> > >> >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c > >> >> [0] PCApply_MG() line 377 in > >> >> > >> >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c > >> >> [0] PCMGMCycle_Private() line 20 in > >> >> > >> >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c > >> >> [0] KSPSolve() line 780 in > >> >> > >> >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c > >> >> [0] KSPSolve_Chebyshev() line 381 in > >> >> > >> >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/cheby/cheby.c > >> >> [0] Petsc has generated inconsistent data > >> >> [0] Eigen estimator failed: DIVERGED_NANORINF at iteration 0 > >> >> > >> >> When I'm using a different solver for -mg_levels_ksp_type, > >> >> such > >> >> as > >> >> gmres, GAMG no longer crashes, but I don't see convergence > >> >> of > >> >> the > >> >> problem (convergence history and ksp_view output are > >> >> attached > >> >> below). > >> >> > >> > > >> > It uses unpreconditioned GMRES to estimate spectral bounds > >> > for > >> > the operator > >> > before using a Chebychev smoother. > >> > If your matrix does not have a nice, connected, positive > >> > spectrum, > >> > Chebychev will not work. However, the fact that > >> > you get DIVERGED_NANORINF in the estimator tells me that you > >> > have a problem > >> > in the matrix. > >> > > >> > >> The error above (DIVERGED_NANORINF) only appears for > >> -mg_levels_ksp_type chebyshev. When I use GMRES > >> (-mg_levels_ksp_type gmres) there are no errors, the KSP just > >> never converges. > >> > >> > > >> >> - Hypre > >> >> With the default settings, BoomerAMG just returns a Vector > >> >> of > >> >> all > >> >> zeros after one iteration. > >> >> When I change the relaxation type > >> >> -pc_hypre_boomeramg_relax_type_all to Jacobi, I get similar > >> >> results than with GAMG: the solver works without errors, but > >> >> doesn't converge. The output for Hypre is also attached > >> >> below. > >> >> > >> >> - ML > >> >> With default settings the result is just like Boomeramg: a > >> >> vector > >> >> of all zeros after one iteration. > >> >> When I change -mg_levels_ksp_type the behaviour is identical > >> >> to > >> >> GAMG. > >> >> > >> >> > >> >> Since none of the packages worked, I'm assuming that the > >> >> error > >> >> lies with me/ my code, > >> > > >> > > >> > It looks like a value in the matrix might be bad. > >> > > >> > > >> >> so I'll give a short overview over what I'm > >> >> trying to do. > >> >> The matrix I'm trying to precondition is the Jacobian of a > >> >> flow > >> >> field originating from an unstructured finite-volume CFD > >> >> code. > > > > > > Compressible or incompressible? > > > > Mstt > > > > The matrix is generated by a compressible CFD code. The current > test case is quasi-incompressible due to very low flow velocity > (flow around a cylinder). > Ah. I do not believe Multigrid is going to work for compressible flow, at least not the straightforward kind of multigrid that we have in PETSc. MG is designed for elliptic problems. Hyprebolic problems require extensive changes and its still a research problem to get it working. Definitely no freely available package has it working, and others cannot be tested by us. Moreover, in the incompressible limit, the compressible formulation can become very stiff, leading to failure of solvers. That is why you get the slightly compressible (adiabatic?) formulations. Thanks, Matt > Kind regards, > Michael > > > > >> It > >> >> has a blocked structure as each node of the original mesh > >> >> holds > >> >> several variables (pressure, density, velocities). However, > >> >> I'm > >> >> not using a DM-Plex since I get the assembled Jacobian in > >> >> binary > >> >> format from the external CFD code. > >> >> When I'm using direct (lu) I get correct results, so the > >> >> basic > >> >> code should be fine. > >> > > >> > > >> > So LU works for the same problem that gave the NANORINF in > >> > GMRES? > >> > > >> > I would recommend using a Manufactured Solution to check the > >> > operation. Then > >> > you can start by feeding in the exact solution, and see that > >> > nothing fails. > >> > Also, I > >> > would make a very small test case, so that you can send the > >> > matrix and/or > >> > code > >> > to us to check. > >> > > >> > >> Yes, LU works for the same matrix/ problem. I didn't change > >> anything besides using -pc_type lu instead of -pc_type gamg. > >> What do you mean by "Manufactured Solution"? > >> Concerning the test case: I'll set one up and send you the > >> matrix > >> in binary format. > >> > >> > > >> >> However, for larger problems lu is not > >> >> feasible due the very large memory requirements, therefore I > >> >> wanted to switch over to multigrid. > >> >> > >> >> Currently, PETSc has no information about the original > >> >> geometry. I > >> >> tried using setCoordinates, but for Hypre and ml it didn't > >> >> make > >> >> a > >> >> difference, and GAMG crashed with an error: > >> >> > >> > > >> > This is only an optimization, and currently GAMG does not > >> > know > >> > what to > >> > do for multiple fields (it handles single field problems by > >> > building the > >> > nullspace of the symbol for the coarse grid). You could > >> > provide > >> > this when > >> > you want to optimize things. > >> > > >> > Thanks, > >> > > >> > Matt > >> > >> Ok, thanks for the clarification. In that case I won't bother > >> with > >> it for now. No need to optimize the code before the other > >> problems > >> are solved. > >> > >> Thank you for your answer! > >> > >> Kind regards, > >> Michael > >> > > >> > > >> >> [0] PCSetCoordinates() line 1883 in > >> >> > >> >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c > >> >> [0] PCSetCoordinates_AGG() line 199 in > >> >> > >> >> > >> > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/gamg/agg.c > >> >> [0] Petsc has generated inconsistent data > >> >> [0] Don't know how to create null space for ndm=2, ndf=4. > >> >> Use > >> >> MatSetNearNullSpace. > >> >> > >> >> I would be glad if you could give me some advice on how to > >> >> deal > >> >> with this. > >> >> > >> >> Kind regards, > >> >> Michael > >> >> > >> >> > >> > >> > > > -- > ____________________________________________________ > > Deutsches Zentrum f?r Luft- und Raumfahrt e.V. (DLR) > Institut f?r Aerodynamik und Str?mungstechnik | Bunsenstr. 10 | > 37073 G?ttingen > > Michael Werner > Telefon 0551 709-2627 | Telefax 0551 709-2811 | > Michael.Werner at dlr.de > DLR.de > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.werner at dlr.de Fri Sep 28 07:27:30 2018 From: michael.werner at dlr.de (Michael Werner) Date: Fri, 28 Sep 2018 14:27:30 +0200 Subject: [petsc-users] Usage of AMG as preconditioner In-Reply-To: References: Message-ID: Matthew Knepley writes: > On Fri, Sep 28, 2018 at 8:13 AM Michael Werner > > wrote: > >> >> Matthew Knepley writes: >> >> > On Fri, Sep 28, 2018 at 7:43 AM Michael Werner >> > >> > wrote: >> > >> >> >> >> Matthew Knepley writes: >> >> >> >> > On Fri, Sep 28, 2018 at 3:23 AM Michael Werner >> >> > >> >> > wrote: >> >> > >> >> >> Hello, >> >> >> >> >> >> I'm having trouble with getting the AMG preconditioners >> >> >> working. I >> >> >> tried all of them (gamg, ml, hypre-boomeramg), with >> >> >> varying >> >> >> degrees of "success": >> >> >> >> >> >> - GAMG: >> >> >> CMD options: -ksp_rtol 1e-8 -ksp_monitor_true_residual >> >> >> -ksp_max_it >> >> >> 20 -ksp_type fgmres -pc_type gamg -pc_gamg_sym_graph TRUE >> >> >> -pc_gamg_agg_nsmooths 1 -ksp_view >> >> >> always crashes with the following error: >> >> >> Error: error code 77 >> >> >> [0] KSPSolve() line 780 in >> >> >> >> >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c >> >> >> [0] KSPSolve_GMRES() line 233 in >> >> >> >> >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/gmres/gmres.c >> >> >> [0] KSPInitialResidual() line 67 in >> >> >> >> >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itres.c >> >> >> [0] KSP_PCApply() line 281 in >> >> >> >> >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/include/petsc/private/kspimpl.h >> >> >> [0] PCApply() line 462 in >> >> >> >> >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c >> >> >> [0] PCApply_MG() line 377 in >> >> >> >> >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c >> >> >> [0] PCMGMCycle_Private() line 20 in >> >> >> >> >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/mg/mg.c >> >> >> [0] KSPSolve() line 780 in >> >> >> >> >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/interface/itfunc.c >> >> >> [0] KSPSolve_Chebyshev() line 381 in >> >> >> >> >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/ksp/impls/cheby/cheby.c >> >> >> [0] Petsc has generated inconsistent data >> >> >> [0] Eigen estimator failed: DIVERGED_NANORINF at >> >> >> iteration 0 >> >> >> >> >> >> When I'm using a different solver for >> >> >> -mg_levels_ksp_type, >> >> >> such >> >> >> as >> >> >> gmres, GAMG no longer crashes, but I don't see >> >> >> convergence >> >> >> of >> >> >> the >> >> >> problem (convergence history and ksp_view output are >> >> >> attached >> >> >> below). >> >> >> >> >> > >> >> > It uses unpreconditioned GMRES to estimate spectral bounds >> >> > for >> >> > the operator >> >> > before using a Chebychev smoother. >> >> > If your matrix does not have a nice, connected, positive >> >> > spectrum, >> >> > Chebychev will not work. However, the fact that >> >> > you get DIVERGED_NANORINF in the estimator tells me that >> >> > you >> >> > have a problem >> >> > in the matrix. >> >> > >> >> >> >> The error above (DIVERGED_NANORINF) only appears for >> >> -mg_levels_ksp_type chebyshev. When I use GMRES >> >> (-mg_levels_ksp_type gmres) there are no errors, the KSP >> >> just >> >> never converges. >> >> >> >> > >> >> >> - Hypre >> >> >> With the default settings, BoomerAMG just returns a >> >> >> Vector >> >> >> of >> >> >> all >> >> >> zeros after one iteration. >> >> >> When I change the relaxation type >> >> >> -pc_hypre_boomeramg_relax_type_all to Jacobi, I get >> >> >> similar >> >> >> results than with GAMG: the solver works without errors, >> >> >> but >> >> >> doesn't converge. The output for Hypre is also attached >> >> >> below. >> >> >> >> >> >> - ML >> >> >> With default settings the result is just like Boomeramg: >> >> >> a >> >> >> vector >> >> >> of all zeros after one iteration. >> >> >> When I change -mg_levels_ksp_type the behaviour is >> >> >> identical >> >> >> to >> >> >> GAMG. >> >> >> >> >> >> >> >> >> Since none of the packages worked, I'm assuming that the >> >> >> error >> >> >> lies with me/ my code, >> >> > >> >> > >> >> > It looks like a value in the matrix might be bad. >> >> > >> >> > >> >> >> so I'll give a short overview over what I'm >> >> >> trying to do. >> >> >> The matrix I'm trying to precondition is the Jacobian of >> >> >> a >> >> >> flow >> >> >> field originating from an unstructured finite-volume CFD >> >> >> code. >> > >> > >> > Compressible or incompressible? >> > >> > Mstt >> > >> >> The matrix is generated by a compressible CFD code. The current >> test case is quasi-incompressible due to very low flow velocity >> (flow around a cylinder). >> > > Ah. I do not believe Multigrid is going to work for compressible > flow, at > least not the > straightforward kind of multigrid that we have in PETSc. MG is > designed for > elliptic > problems. Hyprebolic problems require extensive changes and its > still a > research problem > to get it working. Definitely no freely available package has it > working, > and others cannot be > tested by us. > > Moreover, in the incompressible limit, the compressible > formulation can > become very stiff, > leading to failure of solvers. That is why you get the slightly > compressible (adiabatic?) formulations. > > Thanks, > > Matt > Ah, too bad, I was already expecting something like that, because most of the applications I found so far were FEM/ elasticity problems. Then I'll have to see if I can find some other suitable solution. Kind regards, Michael > >> Kind regards, >> Michael >> >> > >> >> It >> >> >> has a blocked structure as each node of the original mesh >> >> >> holds >> >> >> several variables (pressure, density, >> >> >> velocities). However, >> >> >> I'm >> >> >> not using a DM-Plex since I get the assembled Jacobian in >> >> >> binary >> >> >> format from the external CFD code. >> >> >> When I'm using direct (lu) I get correct results, so the >> >> >> basic >> >> >> code should be fine. >> >> > >> >> > >> >> > So LU works for the same problem that gave the NANORINF in >> >> > GMRES? >> >> > >> >> > I would recommend using a Manufactured Solution to check >> >> > the >> >> > operation. Then >> >> > you can start by feeding in the exact solution, and see >> >> > that >> >> > nothing fails. >> >> > Also, I >> >> > would make a very small test case, so that you can send >> >> > the >> >> > matrix and/or >> >> > code >> >> > to us to check. >> >> > >> >> >> >> Yes, LU works for the same matrix/ problem. I didn't change >> >> anything besides using -pc_type lu instead of -pc_type gamg. >> >> What do you mean by "Manufactured Solution"? >> >> Concerning the test case: I'll set one up and send you the >> >> matrix >> >> in binary format. >> >> >> >> > >> >> >> However, for larger problems lu is not >> >> >> feasible due the very large memory requirements, >> >> >> therefore I >> >> >> wanted to switch over to multigrid. >> >> >> >> >> >> Currently, PETSc has no information about the original >> >> >> geometry. I >> >> >> tried using setCoordinates, but for Hypre and ml it >> >> >> didn't >> >> >> make >> >> >> a >> >> >> difference, and GAMG crashed with an error: >> >> >> >> >> > >> >> > This is only an optimization, and currently GAMG does not >> >> > know >> >> > what to >> >> > do for multiple fields (it handles single field problems >> >> > by >> >> > building the >> >> > nullspace of the symbol for the coarse grid). You could >> >> > provide >> >> > this when >> >> > you want to optimize things. >> >> > >> >> > Thanks, >> >> > >> >> > Matt >> >> >> >> Ok, thanks for the clarification. In that case I won't >> >> bother >> >> with >> >> it for now. No need to optimize the code before the other >> >> problems >> >> are solved. >> >> >> >> Thank you for your answer! >> >> >> >> Kind regards, >> >> Michael >> >> > >> >> > >> >> >> [0] PCSetCoordinates() line 1883 in >> >> >> >> >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c >> >> >> [0] PCSetCoordinates_AGG() line 199 in >> >> >> >> >> >> >> >> >> /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/gamg/agg.c >> >> >> [0] Petsc has generated inconsistent data >> >> >> [0] Don't know how to create null space for ndm=2, ndf=4. >> >> >> Use >> >> >> MatSetNearNullSpace. >> >> >> >> >> >> I would be glad if you could give me some advice on how >> >> >> to >> >> >> deal >> >> >> with this. >> >> >> >> >> >> Kind regards, >> >> >> Michael >> >> >> >> >> >> >> >> >> >> >> From jed at jedbrown.org Fri Sep 28 08:15:04 2018 From: jed at jedbrown.org (Jed Brown) Date: Fri, 28 Sep 2018 07:15:04 -0600 Subject: [petsc-users] Usage of AMG as preconditioner In-Reply-To: References: Message-ID: <87zhw1n4pz.fsf@jedbrown.org> Michael Werner writes: >>> >> > It uses unpreconditioned GMRES to estimate spectral bounds for >>> >> > the operator before using a Chebychev smoother. It is GMRES preconditioned by the diagonal. >> Moreover, in the incompressible limit, the compressible >> formulation can >> become very stiff, >> leading to failure of solvers. That is why you get the slightly >> compressible (adiabatic?) formulations. > > Ah, too bad, I was already expecting something like that, because > most of the applications I found so far were FEM/ elasticity > problems. Then I'll have to see if I can find some other suitable > solution. You can implement a low-Mach preconditioner that would use multigrid. If you're working in conservative variables, then it requires a nonlinear change of variables to isolate the pressure space. From mfadams at lbl.gov Fri Sep 28 09:10:04 2018 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 28 Sep 2018 10:10:04 -0400 Subject: [petsc-users] Usage of AMG as preconditioner In-Reply-To: References: Message-ID: > > > [0] PCSetCoordinates() line 1883 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/interface/precon.c > [0] PCSetCoordinates_AGG() line 199 in > > /home/yoda/wern_mc/Programme/Anaconda/envs/GSA_27/weitere_programme/petsc-3.10.0/src/ksp/pc/impls/gamg/agg.c > [0] Petsc has generated inconsistent data > [0] Don't know how to create null space for ndm=2, ndf=4. Use > MatSetNearNullSpace. > > You have more important things to deal with but PCSetCoordinates is just for elasticity, where the null space can be constructed from the coordinates. As the message says you need to use MatSetNearNullSpace, which means that you need to create the null space manually. You seem to have a 2D problem and 4 dof/cell. For pressure and density you can use a vector of all 1.0. For the velocity you put the rotational mode, the x translation, and the y translational mode. So 5 vectors total. MatNullSpaceCreateRigidBody is a helper function that can create the three rigid body modes for you (same process as PCSetCoordinates), then use that with the two trivial 1 vectors to get a 5 field null space. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.werner at dlr.de Fri Sep 28 11:05:27 2018 From: michael.werner at dlr.de (Michael Werner) Date: Fri, 28 Sep 2018 18:05:27 +0200 Subject: [petsc-users] Usage of AMG as preconditioner In-Reply-To: <87zhw1n4pz.fsf@jedbrown.org> References: <87zhw1n4pz.fsf@jedbrown.org> Message-ID: Jed Brown writes: > Michael Werner writes: > >>>> >> > It uses unpreconditioned GMRES to estimate spectral >>>> >> > bounds for >>>> >> > the operator before using a Chebychev smoother. > > It is GMRES preconditioned by the diagonal. > >>> Moreover, in the incompressible limit, the compressible >>> formulation can >>> become very stiff, >>> leading to failure of solvers. That is why you get the >>> slightly >>> compressible (adiabatic?) formulations. >> >> Ah, too bad, I was already expecting something like that, >> because >> most of the applications I found so far were FEM/ elasticity >> problems. Then I'll have to see if I can find some other >> suitable >> solution. > > You can implement a low-Mach preconditioner that would use > multigrid. > If you're working in conservative variables, then it requires a > nonlinear change of variables to isolate the pressure space. A low-Mach preconditioner might help with the current test cases, however I also intend to apply this code to high-Mach number flows, so I need to find a more general solution. Actually, the high-Mach number applications are more important, since so far all the low-Mach cases are small enough to be solved with direct solvers. I was also thinking about using a geometric multigrid approach via DMPlex. As far as I understood, hyperbolic problems are difficult to solve with AMG because the solver isn't aware of the underlying structure of the problem. Therefore I would think that a geometric multigrid approach should produce better results, right? Do you think it would be worthwile to implement a DMPlex, or would I still run into the same problems? Kind regards, Michael From jed at jedbrown.org Fri Sep 28 11:42:34 2018 From: jed at jedbrown.org (Jed Brown) Date: Fri, 28 Sep 2018 10:42:34 -0600 Subject: [petsc-users] Usage of AMG as preconditioner In-Reply-To: References: <87zhw1n4pz.fsf@jedbrown.org> Message-ID: <87tvm9mv45.fsf@jedbrown.org> Michael Werner writes: > A low-Mach preconditioner might help with the current test cases, > however I also intend to apply this code to high-Mach number > flows, so I need to find a more general solution. Actually, the > high-Mach number applications are more important, since so far all > the low-Mach cases are small enough to be solved with direct > solvers. > > I was also thinking about using a geometric multigrid approach via > DMPlex. As far as I understood, hyperbolic problems are difficult > to solve with AMG because the solver isn't aware of the underlying > structure of the problem. Therefore I would think that a geometric > multigrid approach should produce better results, right? Do you > think it would be worthwile to implement a DMPlex, or would I > still run into the same problems? Agglomeration-based geometric multigrid with block relaxation is a common technique for unstructured compressible CFD, but there are a lot of subtleties. See FUN3D papers (Diskin, Thomas, sometimes Brandt) for examples of this approach. From mfadams at lbl.gov Fri Sep 28 14:41:20 2018 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 28 Sep 2018 15:41:20 -0400 Subject: [petsc-users] Usage of AMG as preconditioner In-Reply-To: <87tvm9mv45.fsf@jedbrown.org> References: <87zhw1n4pz.fsf@jedbrown.org> <87tvm9mv45.fsf@jedbrown.org> Message-ID: As Jed says hyperbolic dominated systems are hard. For geometric papers you can look at Jameson papers and Adams, Samtaney & Brandt (MHD). But be prepared to work. Good stuff but hard. On Fri, Sep 28, 2018 at 12:42 PM Jed Brown wrote: > Michael Werner writes: > > > A low-Mach preconditioner might help with the current test cases, > > however I also intend to apply this code to high-Mach number > > flows, so I need to find a more general solution. Actually, the > > high-Mach number applications are more important, since so far all > > the low-Mach cases are small enough to be solved with direct > > solvers. > > > > I was also thinking about using a geometric multigrid approach via > > DMPlex. As far as I understood, hyperbolic problems are difficult > > to solve with AMG because the solver isn't aware of the underlying > > structure of the problem. Therefore I would think that a geometric > > multigrid approach should produce better results, right? Do you > > think it would be worthwile to implement a DMPlex, or would I > > still run into the same problems? > > Agglomeration-based geometric multigrid with block relaxation is a > common technique for unstructured compressible CFD, but there are a lot > of subtleties. See FUN3D papers (Diskin, Thomas, sometimes Brandt) for > examples of this approach. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Fri Sep 28 18:07:00 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Fri, 28 Sep 2018 17:07:00 -0600 Subject: [petsc-users] MatPtAPNumeric_MPIAIJ_MPIAIJ_scalable Message-ID: Hi All, I was wondering how much memory is required to get PtAP done? Do you have any simple formula to this? So that I can have an estimate. Fande, [132]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [132]PETSC ERROR: Out of memory. This could be due to allocating [132]PETSC ERROR: too large an object or bleeding by not properly [132]PETSC ERROR: destroying unneeded objects. [132]PETSC ERROR: Memory allocated 0 Memory used by process 3249920 [132]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [132]PETSC ERROR: Memory requested 89148704 [132]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [132]PETSC ERROR: Petsc Release Version 3.9.4, unknown [132]PETSC ERROR: ../../rattlesnake-opt on a arch-theta-avx512-64-opt named nid03830 by fdkong Fri Sep 28 22:43:45 2018 [132]PETSC ERROR: Configure options --LIBS=-lstdc++ --known-64-bit-blas-indices=0 --known-bits-per-byte=8 --known-has-attribute-aligned=1 --known-level1-dcache-assoc=8 --known-level1-dcache-linesize=64 --known-level1-dcache-size=32768 --known-memcmp-ok=1 --known-mklspblas-supports-zero-based=0 --known-mpi-c-double-complex=1 --known-mpi-int64_t=1 --known-mpi-long-double=1 --known-mpi-shared-libraries=0 --known-sdot-returns-double=0 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8 --known-sizeof-void-p=8 --known-snrm2-returns-double=0 --with-batch=1 --with-blaslapack-lib="-mkl -L/opt/intel/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64" --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn --with-fortranlib-autodetect=0 --with-hdf5=0 --with-memalign=64 --with-mpiexec=aprun --with-shared-libraries=0 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-hypre=1 --download-ptscotch=1 COPTFLAGS="-O3 -xMIC-AVX512" CXXOPTFLAGS="-O3 -xMIC-AVX512" FOPTFLAGS="-O3 -xMIC-AVX512" PETSC_ARCH=arch-theta-avx512-64-opt --with-64-bit-indices=1 [132]PETSC ERROR: #1 PetscSegBufferCreate() line 64 in /gpfs/mira-home/fdkong/petsc/src/sys/utils/segbuffer.c [132]PETSC ERROR: #2 PetscSegBufferCreate() line 64 in /gpfs/mira-home/fdkong/petsc/src/sys/utils/segbuffer.c [132]PETSC ERROR: #3 PetscSegBufferExtractInPlace() line 227 in /gpfs/mira-home/fdkong/petsc/src/sys/utils/segbuffer.c [132]PETSC ERROR: #4 MatStashScatterBegin_BTS() line 854 in /gpfs/mira-home/fdkong/petsc/src/mat/utils/matstash.c [132]PETSC ERROR: #5 MatStashScatterBegin_Private() line 461 in /gpfs/mira-home/fdkong/petsc/src/mat/utils/matstash.c [132]PETSC ERROR: #6 MatAssemblyBegin_MPIAIJ() line 683 in /gpfs/mira-home/fdkong/petsc/src/mat/impls/aij/mpi/mpiaij.c [132]PETSC ERROR: #7 MatAssemblyBegin() line 5158 in /gpfs/mira-home/fdkong/petsc/src/mat/interface/matrix.c [132]PETSC ERROR: #8 MatPtAPNumeric_MPIAIJ_MPIAIJ_scalable() line 262 in /gpfs/mira-home/fdkong/petsc/src/mat/impls/aij/mpi/mpiptap.c [132]PETSC ERROR: #9 MatPtAP_MPIAIJ_MPIAIJ() line 172 in /gpfs/mira-home/fdkong/petsc/src/mat/impls/aij/mpi/mpiptap.c [132]PETSC ERROR: #10 MatPtAP() line 9182 in /gpfs/mira-home/fdkong/petsc/src/mat/interface/matrix.c [132]PETSC ERROR: #11 MatGalerkin() line 10615 in /gpfs/mira-home/fdkong/petsc/src/mat/interface/matrix.c [132]PETSC ERROR: #12 PCSetUp_MG() line 730 in /gpfs/mira-home/fdkong/petsc/src/ksp/pc/impls/mg/mg.c [132]PETSC ERROR: #13 PCSetUp_HMG() line 336 in /gpfs/mira-home/fdkong/petsc/src/ksp/pc/impls/hmg/hmg.c [132]PETSC ERROR: #14 PCSetUp() line 923 in /gpfs/mira-home/fdkong/petsc/src/ksp/pc/interface/precon.c [132]PETSC ERROR: #15 KSPSetUp() line 381 in /gpfs/mira-home/fdkong/petsc/src/ksp/ksp/interface/itfunc.c [136]PETSC ERROR: --------------------- Error Message ---------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysjosh.lo at gmail.com Fri Sep 28 20:57:28 2018 From: ysjosh.lo at gmail.com (Josh L) Date: Fri, 28 Sep 2018 20:57:28 -0500 Subject: [petsc-users] DMPlexDistribute segmentation fault Message-ID: Hi, I am implementing DMPlex to handle my mesh in parallel, but I am getting segmentation fault during DMPlexDistribute. i am testing with 2 processors on a toy mesh that only has 4 quad elements. the code is as following CALL DMPlexCreateFromCellList Check mesh topology. Face, skeleton symmetry CALL DMPlexDistribute (dmserial,0,PETSC_NULL_SF,dmmpi,ierr) . (segmentation fault on rank #1 here) I trace it back to the external library "Chaco" called by PETSc to partition the mesh. The following is the stack For rank #1 #10 main () at /work/03691/yslo/stampede2/linear/rewrite/src/main.F90:18 (at 0x000000000040569d) #9 createmesh (rank=1, nsize=2, dmmpi=...) at /work/03691/yslo/stampede2/linear/rewrite/src/createmesh.F90:107 (at 0x000000000040523b) #8 dmplexdistribute_ (dm=0x320, overlap=0x1, sf=0x7fffffff4480, dmParallel=0x2aaab78a585d, ierr=0x1) at /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/ftn-custom/zplexdistribute.c:15 (at 0x00002aaaac08334e) #7 DMPlexDistribute (dm=0x320, overlap=1, sf=0x7fffffff4480, dmParallel=0x2aaab78a585d) at /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexdistribute.c:1664 (at 0x00002aaaabd12d08) #6 PetscPartitionerPartition (part=0x320, dm=0x1, partSection=0x7fffffff4480, partition=0x2aaab78a585d) at /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexpartition.c:675 (at 0x00002aaaabcfe92c) #5 PetscPartitionerPartition_Chaco (part=0x320, dm=0x1, nparts=-48000, numVertices=-1215670179, start=0x1, adjacency=0x22, partSection=0x0, partition=0x772120) at /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexpartition.c:1279 (at 0x00002aaaabd03dfd) #4 interface.Z (nvtxs=800, start=0x1, adjacency=0x7fffffff4480, vwgts=0x2aaab78a585d, ewgts=0x1, x=0x22, y=0x0, z=0x0, outassignname=0x0, outfilename=0x0, assignment=0x76ff90, architecture=1, ndims_tot=0, mesh_dims=0x7fffffff7038, goal=0x0, global_method=1, local_method=1, rqi_flag=0, vmax=200, ndims=1, eigtol=2.1137067449068142e-314, seed=123636512) at /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/main/interface.c:206 (at 0x00002aaaad7aaf70) #3 submain.Z (graph=0x320, nvtxs=1, nedges=-48000, using_vwgts=-1215670179, using_ewgts=1, igeom=34, coords=0x0, outassignname=0x0, outfilename=0x0, assignment=0x76ff8e, goal=0x7739a0, architecture=1, ndims_tot=0, mesh_dims=0x7fffffff7038, global_method=1, local_method=1, rqi_flag=0, vmax=200, ndims=1, eigtol=2.1137067449068142e-314, seed=123636512) at /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/submain/submain.c:151 (at 0x00002aaaad7ae52e) #2 check_input.Z (graph=0x320, nvtxs=1, nedges=-48000, igeom=-1215670179, coords=0x1, graphname=0x22 , assignment=0x76ff8e, goal=0x7739a0, architecture=1, ndims_tot=0, mesh_dims=0x7fffffff7038, global_method=1, local_method=1, rqi_flag=0, vmax=0x7fffffff47a8, ndims=1, eigtol=2.1137067449068142e-314) at /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/input/check_input.c:56 (at 0x00002aaaad7c6ed1) #1 check_graph (graph=0x320, nvtxs=1, nedges=-48000) at /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/graph/check_graph.c:90 (at 0x00002aaaad8204d3) #0 is_an_edge (vertex=0x320, v2=1, weight2=0x7fffffff4480) at /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/graph/check_graph.c:134 (at 0x00002aaaad8206d3) For rank #0 #18 main () at /work/03691/yslo/stampede2/linear/rewrite/src/main.F90:18 (at 0x000000000040569d) #17 createmesh (rank=0, nsize=2, dmmpi=...) at /work/03691/yslo/stampede2/linear/rewrite/src/createmesh.F90:107 (at 0x000000000040523b) #16 dmplexdistribute_ (dm=0x65f300, overlap=0x0, sf=0x2aaab4c8698c, dmParallel=0xffffffffffffffff, ierr=0x0) at /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/ftn-custom/zplexdistribute.c:15 (at 0x00002aaaac08334e) #15 DMPlexDistribute (dm=0x65f300, overlap=0, sf=0x2aaab4c8698c, dmParallel=0xffffffffffffffff) at /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexdistribute.c:1664 (at 0x00002aaaabd12d08) #14 PetscPartitionerPartition (part=0x65f300, dm=0x0, partSection=0x2aaab4c8698c, partition=0xffffffffffffffff) at /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexpartition.c:675 (at 0x00002aaaabcfe92c) #13 PetscPartitionerPartition_Chaco (part=0x65f300, dm=0x0, nparts=-1261934196, numVertices=-1, start=0x0, adjacency=0x0, partSection=0x7726c0, partition=0x7fffffff71a8) at /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexpartition.c:1314 (at 0x00002aaaabd04029) #12 ISCreateGeneral (comm=6681344, n=0, idx=0x2aaab4c8698c, mode=(PETSC_OWN_POINTER | PETSC_USE_POINTER | unknown: 4294967292), is=0x0) at /home1/apps/intel17/impi17_0/petsc/3.9/src/vec/is/is/impls/general/general.c:671 (at 0x00002aaaab0ea94e) #11 ISGeneralSetIndices (is=0x65f300, n=0, idx=0x2aaab4c8698c, mode=(PETSC_OWN_POINTER | PETSC_USE_POINTER | unknown: 4294967292)) at /home1/apps/intel17/impi17_0/petsc/3.9/src/vec/is/is/impls/general/general.c:698 (at 0x00002aaaab0eaa77) #10 ISGeneralSetIndices_General.Z (is=0x65f300, n=0, idx=0x2aaab4c8698c, mode=(PETSC_OWN_POINTER | PETSC_USE_POINTER | unknown: 4294967292)) at /home1/apps/intel17/impi17_0/petsc/3.9/src/vec/is/is/impls/general/general.c:712 (at 0x00002aaaab0ef567) #9 PetscLayoutSetUp (map=0x65f300) at /home1/apps/intel17/impi17_0/petsc/3.9/src/vec/is/utils/pmap.c:137 (at 0x00002aaaab0c6d52) #8 PetscSplitOwnership (comm=6681344, n=0x0, N=0x2aaab4c8698c) at /home1/apps/intel17/impi17_0/petsc/3.9/src/sys/utils/psplit.c:80 (at 0x00002aaaaaeb7b00) #7 PMPI_Allreduce (sendbuf=0x65f300, recvbuf=0x0, count=-1261934196, datatype=-1, op=0, comm=0) at /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/mpi/coll/allreduce.c:1395 (at 0x00002aaab40966e6) #6 MPIR_Allreduce_intra (sendbuf=0x65f300, recvbuf=0x0, count=-1261934196, datatype=-1, op=0, comm_ptr=0x0, errflag=0x7fffffff4798) at /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/mpi/coll/allreduce.c:339 (at 0x00002aaab409307b) #5 MPIR_Allreduce_shm_generic (sendbuf=, recvbuf=, count=, datatype=, op=, comm_ptr=, errflag=, kind=1476395011) at /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/mpi/coll/allreduce.c:1137 (at 0x00002aaab409307b) #4 I_MPI_COLL_SHM_KNARY_REDUCE (node_comm_ptr=, root=, localbuf=, sendbuf=, recvbuf=, count=, datatype=, op=, errflag=, knomial_factor=) at /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/I_MPI/include/shm_coll_templating.h:1090 (at 0x00002aaab409307b) #3 I_MPI_COLL_SHM_GENERIC_GATHER_REDUCE..1 (node_comm_ptr=0x65f300, root=0, is_reduce=-1261934196, localbuf=0xffffffffffffffff, sendbuf=0x0, recvbuf=0x0, count=1, datatype=1275069445, op=1476395011, errflag=0x7fffffff4798, knomial_factor=4, algo_type=2) at /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/I_MPI/include/shm_coll_templating.h:558 (at 0x00002aaab408f3a9) #2 I_MPI_memcpy (destination=, source=0x0, size=) at /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/I_MPI/include/shm_coll_templating.h:749 (at 0x00002aaab408f3a9) #1 PMPIDI_CH3I_Progress (progress_state=0x65f300, is_blocking=0) at /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/mpid/ch3/channels/nemesis/src/ch3_progress.c:981 (at 0x00002aaab40e85a6) #0 sched_yield () from /lib64/libc.so.6 (at 0x00002aaab7898e47) Any idea why this is happening? the overlap is set to 0, but it is 1 on rank #1 Is there anyway to know the cell and vertex number distributed on each processors? my old code partitions the mesh with Metis, and I always output cell data that shows the rank number on each cell, so I can visualize how mesh is partitioned. It is not necessary, but is there anyway to get it in DMPlex? DMPlexGetCellNumbering might work, but it fails the linking. In fact, many function under developer category fails the linking. Thanks, Yu-Sheng -------------- next part -------------- An HTML attachment was scrubbed... URL: From zakaryah at gmail.com Fri Sep 28 22:13:38 2018 From: zakaryah at gmail.com (zakaryah) Date: Fri, 28 Sep 2018 23:13:38 -0400 Subject: [petsc-users] Fwd: Implementing a homotopy solver In-Reply-To: References: <6377AF62-D548-4276-A2A6-158EE4513593@anl.gov> <00D16613-4B28-4801-96EA-7CE2EF8BF8F7@imperial.ac.uk> <29B68A54-9D29-4881-AD80-98A5E510FD82@anl.gov> Message-ID: I'm working on a homotopy solver which follows a zero curve through solution space. A central aspect of this method is to calculate the vector tangent to the curve, predict the next point, then correct using iteration of, e.g. Newton's method, orthogonal to the tangent vector. Previously, we discussed the possibilities of implementing this within PETSc's SNES. Within my FormJacobian() function, I construct the linear system which defines the tangent vector, solve it, then add the vector to the nullspace of the Jacobian. I think that in principle this can work, but I suspect I'm doing something wrong. Here's a summary of the code within FormJacobian(): - Set values and assemble Jacobian matrix A - this is working fine - Set values and assemble RHS vector b for linear system defining tangent vector n - this is working fine - SNESGetKSP(snes,&my_ksp) - I thought it made sense to use the KSP associated with the SNES, hoping that PCs which use a factorization could be reused when the SNES calls KSPSolve() to calculate the update - KSPSetFromOptions(my_ksp) - not sure this is necessary but one of my problems is setting options for this KSP from the command line and even with this call it doesn't seem to be working properly - MatSetNullSpace(A,NULL) - remove any existing null space from Jacobian - KSPSetOperators(my_ksp,A,P) - P is the other matrix in FormJacobian() - VecSet(n,0) - set initial guess to zero - KSPSolve(my_ksp,b,n) - these solves appear to work, i.e. use the options passed from the command line with -ksp_XXX or -pc_XXX - VecNormalize(n,NULL) - MatNullSpaceCreate(PetscObjectComm((PetscObject)A),PETSC_FALSE,1,&n,&nullsp) - MatSetNullSpace(A,nullsp) - MatNullSpaceDestroy(&nullsp) - return The immediate problem is that the subsequent KSPSolve(), i.e. the one called internally by SNESSolve(), behaves strangely. For example, if I use -pc_type none -ksp_monitor -ksp_monitor_true_residual, then the KSPSolve() that I call within FormJacobian() looks correct - "preconditioned" norm and true norm are identical, and both converge as I expect (i.e. slowly but geometrically). However, the subsequent KSPSolve(), internal to the SNESSolve(), has large differences between the preconditioned norm and the true norm. In addition, the KSP does not converge in the true residual, but I'll have a hard time debugging that without knowing how to properly set the options. I hope someone can help me see what I'm doing wrong. On Sun, Jul 22, 2018 at 9:09 PM zakaryah wrote: > ?Thanks Matt and Barry, > > Matt - if I do the calculation in FormJacobian(), which makes by far the > most sense and is as per your suggestion, do I need to set the operators of > the SNES's KSP back to whatever they were before I set them? The linear > system I want to solve within FormJacobian() involves the Jacobian matrix > itself, and I want to remove the "nullspace" from that same matrix within > FormFunction(). > > Barry - I'm trying to implement a homotopy solver. In short, I have a > system of n nonlinear equations in n variables, F(x), which is hard to > solve because the Jacobian tends to become singular using standard > methods?. I want to add an auxiliary variable, lambda, to create a > homotopy: H(lambda,x) = lambda*F(x) + (1-lambda)G(x), where G is "easy to > solve", and the idea is that the Jacobian of the n+1 variable system will > not become singular along the curve H(lambda,x) = 0. > > The method involves adding an equation to H so that the Jacobian H' is > square. The "submatrix" refers to the n x (n+1) matrix which represents > the Jacobian without the added equation, whereas my FormJacobian() forms > the entire (n+1) x (n+1) matrix H'. I only refer to the submatrix because > it has a nullspace, and I want to find it by solving a linear system > designed for this purpose, H' u = b, where b is not the zero vector. H' > has no nullspace, but I want to remove the projection of u from my SNES > solution vector, as u IS in the nullspace of the submatrix. > > The RHS vector b can be calculated outside the SNES solve. I guess my > FormJacobian() should look like this: > > FormJacobian(SNES snes, Vec x, Mat Amat, Mat Pmat, void *ctx) { > KSP ksp; > MatNullSpace unull; > user_struct *user = (user_struct*)ctx; > > calculate Amat > assemble Amat > > if (Pmat != Amat) { > assemble Amat > } > > SNESGetKSP(snes,&ksp); > KSPSetOperators(ksp,Amat,Pmat); > KSPSolve(ksp,user->b,user->u); > > MatNullSpaceCreate(PetscObjectComm((PetscObject)Amat), PETSC_FALSE, 1, > &(user->u),&unull); > MatSetNullSpace(Amat,unull); > MatNullSpaceDestroy(&unull); > } > > Does this look right? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Sep 28 22:59:41 2018 From: jed at jedbrown.org (Jed Brown) Date: Fri, 28 Sep 2018 21:59:41 -0600 Subject: [petsc-users] MatPtAPNumeric_MPIAIJ_MPIAIJ_scalable In-Reply-To: References: Message-ID: <878t3llzrm.fsf@jedbrown.org> It depends entirely on your matrices. For example, if A is an arrowhead matrix (graph of a star -- one hub and many leaves) then A^2 is dense. If you have particular stencils for A and P, then we could tell you the fill ratio. Fande Kong writes: > Hi All, > > I was wondering how much memory is required to get PtAP done? Do you have > any simple formula to this? So that I can have an estimate. > > > Fande, > > > [132]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [132]PETSC ERROR: Out of memory. This could be due to allocating > [132]PETSC ERROR: too large an object or bleeding by not properly > [132]PETSC ERROR: destroying unneeded objects. > [132]PETSC ERROR: Memory allocated 0 Memory used by process 3249920 > [132]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. > [132]PETSC ERROR: Memory requested 89148704 > [132]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [132]PETSC ERROR: Petsc Release Version 3.9.4, unknown > [132]PETSC ERROR: ../../rattlesnake-opt on a arch-theta-avx512-64-opt named > nid03830 by fdkong Fri Sep 28 22:43:45 2018 > [132]PETSC ERROR: Configure options --LIBS=-lstdc++ > --known-64-bit-blas-indices=0 --known-bits-per-byte=8 > --known-has-attribute-aligned=1 --known-level1-dcache-assoc=8 > --known-level1-dcache-linesize=64 --known-level1-dcache-size=32768 > --known-memcmp-ok=1 --known-mklspblas-supports-zero-based=0 > --known-mpi-c-double-complex=1 --known-mpi-int64_t=1 > --known-mpi-long-double=1 --known-mpi-shared-libraries=0 > --known-sdot-returns-double=0 --known-sizeof-MPI_Comm=4 > --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8 > --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8 > --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8 > --known-sizeof-void-p=8 --known-snrm2-returns-double=0 --with-batch=1 > --with-blaslapack-lib="-mkl > -L/opt/intel/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64" > --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC > --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn > --with-fortranlib-autodetect=0 --with-hdf5=0 --with-memalign=64 > --with-mpiexec=aprun --with-shared-libraries=0 --download-metis=1 > --download-parmetis=1 --download-superlu_dist=1 --download-hypre=1 > --download-ptscotch=1 COPTFLAGS="-O3 -xMIC-AVX512" CXXOPTFLAGS="-O3 > -xMIC-AVX512" FOPTFLAGS="-O3 -xMIC-AVX512" > PETSC_ARCH=arch-theta-avx512-64-opt --with-64-bit-indices=1 > [132]PETSC ERROR: #1 PetscSegBufferCreate() line 64 in > /gpfs/mira-home/fdkong/petsc/src/sys/utils/segbuffer.c > [132]PETSC ERROR: #2 PetscSegBufferCreate() line 64 in > /gpfs/mira-home/fdkong/petsc/src/sys/utils/segbuffer.c > [132]PETSC ERROR: #3 PetscSegBufferExtractInPlace() line 227 in > /gpfs/mira-home/fdkong/petsc/src/sys/utils/segbuffer.c > [132]PETSC ERROR: #4 MatStashScatterBegin_BTS() line 854 in > /gpfs/mira-home/fdkong/petsc/src/mat/utils/matstash.c > [132]PETSC ERROR: #5 MatStashScatterBegin_Private() line 461 in > /gpfs/mira-home/fdkong/petsc/src/mat/utils/matstash.c > [132]PETSC ERROR: #6 MatAssemblyBegin_MPIAIJ() line 683 in > /gpfs/mira-home/fdkong/petsc/src/mat/impls/aij/mpi/mpiaij.c > [132]PETSC ERROR: #7 MatAssemblyBegin() line 5158 in > /gpfs/mira-home/fdkong/petsc/src/mat/interface/matrix.c > [132]PETSC ERROR: #8 MatPtAPNumeric_MPIAIJ_MPIAIJ_scalable() line 262 in > /gpfs/mira-home/fdkong/petsc/src/mat/impls/aij/mpi/mpiptap.c > [132]PETSC ERROR: #9 MatPtAP_MPIAIJ_MPIAIJ() line 172 in > /gpfs/mira-home/fdkong/petsc/src/mat/impls/aij/mpi/mpiptap.c > [132]PETSC ERROR: #10 MatPtAP() line 9182 in > /gpfs/mira-home/fdkong/petsc/src/mat/interface/matrix.c > [132]PETSC ERROR: #11 MatGalerkin() line 10615 in > /gpfs/mira-home/fdkong/petsc/src/mat/interface/matrix.c > [132]PETSC ERROR: #12 PCSetUp_MG() line 730 in > /gpfs/mira-home/fdkong/petsc/src/ksp/pc/impls/mg/mg.c > [132]PETSC ERROR: #13 PCSetUp_HMG() line 336 in > /gpfs/mira-home/fdkong/petsc/src/ksp/pc/impls/hmg/hmg.c > [132]PETSC ERROR: #14 PCSetUp() line 923 in > /gpfs/mira-home/fdkong/petsc/src/ksp/pc/interface/precon.c > [132]PETSC ERROR: #15 KSPSetUp() line 381 in > /gpfs/mira-home/fdkong/petsc/src/ksp/ksp/interface/itfunc.c > [136]PETSC ERROR: --------------------- Error Message > ---------------------------------------------------------- From knepley at gmail.com Sat Sep 29 07:16:44 2018 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 29 Sep 2018 08:16:44 -0400 Subject: [petsc-users] Fwd: Implementing a homotopy solver In-Reply-To: References: <6377AF62-D548-4276-A2A6-158EE4513593@anl.gov> <00D16613-4B28-4801-96EA-7CE2EF8BF8F7@imperial.ac.uk> <29B68A54-9D29-4881-AD80-98A5E510FD82@anl.gov> Message-ID: On Fri, Sep 28, 2018 at 11:13 PM zakaryah wrote: > I'm working on a homotopy solver which follows a zero curve through > solution space. A central aspect of this method is to calculate the vector > tangent to the curve, predict the next point, then correct using iteration > of, e.g. Newton's method, orthogonal to the tangent vector. > > Previously, we discussed the possibilities of implementing this within > PETSc's SNES. Within my FormJacobian() function, I construct the linear > system which defines the tangent vector, solve it, then add the vector to > the nullspace of the Jacobian. I think that in principle this can work, > but I suspect I'm doing something wrong. > > Here's a summary of the code within FormJacobian(): > > - Set values and assemble Jacobian matrix A - this is working fine > - Set values and assemble RHS vector b for linear system defining > tangent vector n - this is working fine > - SNESGetKSP(snes,&my_ksp) - I thought it made sense to use the KSP > associated with the SNES, hoping that PCs which use a factorization could > be reused when the SNES calls KSPSolve() to calculate the update > - KSPSetFromOptions(my_ksp) - not sure this is necessary but one of my > problems is setting options for this KSP from the command line and even > with this call it doesn't seem to be working properly > - MatSetNullSpace(A,NULL) - remove any existing null space from > Jacobian > - KSPSetOperators(my_ksp,A,P) - P is the other matrix in FormJacobian() > - VecSet(n,0) - set initial guess to zero > - KSPSolve(my_ksp,b,n) - these solves appear to work, i.e. use the > options passed from the command line with -ksp_XXX or -pc_XXX > - VecNormalize(n,NULL) > - > MatNullSpaceCreate(PetscObjectComm((PetscObject)A),PETSC_FALSE,1,&n,&nullsp) > - MatSetNullSpace(A,nullsp) > - MatNullSpaceDestroy(&nullsp) > - return > > The immediate problem is that the subsequent KSPSolve(), i.e. the one > called internally by SNESSolve(), behaves strangely. For example, if I use > -pc_type none -ksp_monitor -ksp_monitor_true_residual, then the KSPSolve() > that I call within FormJacobian() looks correct - "preconditioned" norm and > true norm are identical, and both converge as I expect (i.e. slowly but > geometrically). However, the subsequent KSPSolve(), internal to the > SNESSolve(), has large differences between the preconditioned norm and the > true norm. In addition, the KSP does not converge in the true residual, > but I'll have a hard time debugging that without knowing how to properly > set the options. > We need to clear up the usage first. If you want EXACTLY the same solver for both solvers, then reuse the KSP, otherwise do not do it. Does it work then? Thanks, Matt > I hope someone can help me see what I'm doing wrong. > > On Sun, Jul 22, 2018 at 9:09 PM zakaryah wrote: > >> Thanks Matt and Barry, >> >> Matt - if I do the calculation in FormJacobian(), which makes by far the >> most sense and is as per your suggestion, do I need to set the operators of >> the SNES's KSP back to whatever they were before I set them? The linear >> system I want to solve within FormJacobian() involves the Jacobian matrix >> itself, and I want to remove the "nullspace" from that same matrix within >> FormFunction(). >> >> Barry - I'm trying to implement a homotopy solver. In short, I have a >> system of n nonlinear equations in n variables, F(x), which is hard to >> solve because the Jacobian tends to become singular using standard >> methods. I want to add an auxiliary variable, lambda, to create a >> homotopy: H(lambda,x) = lambda*F(x) + (1-lambda)G(x), where G is "easy to >> solve", and the idea is that the Jacobian of the n+1 variable system will >> not become singular along the curve H(lambda,x) = 0. >> >> The method involves adding an equation to H so that the Jacobian H' is >> square. The "submatrix" refers to the n x (n+1) matrix which represents >> the Jacobian without the added equation, whereas my FormJacobian() forms >> the entire (n+1) x (n+1) matrix H'. I only refer to the submatrix because >> it has a nullspace, and I want to find it by solving a linear system >> designed for this purpose, H' u = b, where b is not the zero vector. H' >> has no nullspace, but I want to remove the projection of u from my SNES >> solution vector, as u IS in the nullspace of the submatrix. >> >> The RHS vector b can be calculated outside the SNES solve. I guess my >> FormJacobian() should look like this: >> >> FormJacobian(SNES snes, Vec x, Mat Amat, Mat Pmat, void *ctx) { >> KSP ksp; >> MatNullSpace unull; >> user_struct *user = (user_struct*)ctx; >> >> calculate Amat >> assemble Amat >> >> if (Pmat != Amat) { >> assemble Amat >> } >> >> SNESGetKSP(snes,&ksp); >> KSPSetOperators(ksp,Amat,Pmat); >> KSPSolve(ksp,user->b,user->u); >> >> MatNullSpaceCreate(PetscObjectComm((PetscObject)Amat), PETSC_FALSE, 1, >> &(user->u),&unull); >> MatSetNullSpace(Amat,unull); >> MatNullSpaceDestroy(&unull); >> } >> >> Does this look right? >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From zakaryah at gmail.com Sat Sep 29 08:47:19 2018 From: zakaryah at gmail.com (zakaryah) Date: Sat, 29 Sep 2018 09:47:19 -0400 Subject: [petsc-users] Fwd: Implementing a homotopy solver In-Reply-To: References: <6377AF62-D548-4276-A2A6-158EE4513593@anl.gov> <00D16613-4B28-4801-96EA-7CE2EF8BF8F7@imperial.ac.uk> <29B68A54-9D29-4881-AD80-98A5E510FD82@anl.gov> Message-ID: Hi Matt - thanks for all your help. Let's say I want exactly the same solver for the tangent vector and the SNES update, so I should reuse the KSP. My attempt to do this looks like the summary of FormJacobian() in the previous message: - assemble Jacobian - assemble RHS - get KSP from the SNES passed to FormJacobian() - KSPSetFromOptions() - KSPSetOperators() - KSPSolve() I'm not sure that's the right approach, but it doesn't work - the KSPSolve() in the summary, i.e. the one for the tangent vector, seems to work fine. But the next KSPSolve() looks strange - it seems to use a preconditioner even with -pc_type none, etc. This makes me think I am doing something seriously wrong. On Sat, Sep 29, 2018, 8:16 AM Matthew Knepley wrote: > On Fri, Sep 28, 2018 at 11:13 PM zakaryah wrote: > >> I'm working on a homotopy solver which follows a zero curve through >> solution space. A central aspect of this method is to calculate the vector >> tangent to the curve, predict the next point, then correct using iteration >> of, e.g. Newton's method, orthogonal to the tangent vector. >> >> Previously, we discussed the possibilities of implementing this within >> PETSc's SNES. Within my FormJacobian() function, I construct the linear >> system which defines the tangent vector, solve it, then add the vector to >> the nullspace of the Jacobian. I think that in principle this can work, >> but I suspect I'm doing something wrong. >> >> Here's a summary of the code within FormJacobian(): >> >> - Set values and assemble Jacobian matrix A - this is working fine >> - Set values and assemble RHS vector b for linear system defining >> tangent vector n - this is working fine >> - SNESGetKSP(snes,&my_ksp) - I thought it made sense to use the KSP >> associated with the SNES, hoping that PCs which use a factorization could >> be reused when the SNES calls KSPSolve() to calculate the update >> - KSPSetFromOptions(my_ksp) - not sure this is necessary but one of >> my problems is setting options for this KSP from the command line and even >> with this call it doesn't seem to be working properly >> - MatSetNullSpace(A,NULL) - remove any existing null space from >> Jacobian >> - KSPSetOperators(my_ksp,A,P) - P is the other matrix in >> FormJacobian() >> - VecSet(n,0) - set initial guess to zero >> - KSPSolve(my_ksp,b,n) - these solves appear to work, i.e. use the >> options passed from the command line with -ksp_XXX or -pc_XXX >> - VecNormalize(n,NULL) >> - >> MatNullSpaceCreate(PetscObjectComm((PetscObject)A),PETSC_FALSE,1,&n,&nullsp) >> - MatSetNullSpace(A,nullsp) >> - MatNullSpaceDestroy(&nullsp) >> - return >> >> The immediate problem is that the subsequent KSPSolve(), i.e. the one >> called internally by SNESSolve(), behaves strangely. For example, if I use >> -pc_type none -ksp_monitor -ksp_monitor_true_residual, then the KSPSolve() >> that I call within FormJacobian() looks correct - "preconditioned" norm and >> true norm are identical, and both converge as I expect (i.e. slowly but >> geometrically). However, the subsequent KSPSolve(), internal to the >> SNESSolve(), has large differences between the preconditioned norm and the >> true norm. In addition, the KSP does not converge in the true residual, >> but I'll have a hard time debugging that without knowing how to properly >> set the options. >> > > We need to clear up the usage first. If you want EXACTLY the same solver > for both solvers, then reuse > the KSP, otherwise do not do it. Does it work then? > > Thanks, > > Matt > > >> I hope someone can help me see what I'm doing wrong. >> >> On Sun, Jul 22, 2018 at 9:09 PM zakaryah wrote: >> >>> Thanks Matt and Barry, >>> >>> Matt - if I do the calculation in FormJacobian(), which makes by far the >>> most sense and is as per your suggestion, do I need to set the operators of >>> the SNES's KSP back to whatever they were before I set them? The >>> linear system I want to solve within FormJacobian() involves the Jacobian >>> matrix itself, and I want to remove the "nullspace" from that same matrix >>> within FormFunction(). >>> >>> Barry - I'm trying to implement a homotopy solver. In short, I have a >>> system of n nonlinear equations in n variables, F(x), which is hard to >>> solve because the Jacobian tends to become singular using standard >>> methods. I want to add an auxiliary variable, lambda, to create a >>> homotopy: H(lambda,x) = lambda*F(x) + (1-lambda)G(x), where G is "easy to >>> solve", and the idea is that the Jacobian of the n+1 variable system will >>> not become singular along the curve H(lambda,x) = 0. >>> >>> The method involves adding an equation to H so that the Jacobian H' is >>> square. The "submatrix" refers to the n x (n+1) matrix which represents >>> the Jacobian without the added equation, whereas my FormJacobian() forms >>> the entire (n+1) x (n+1) matrix H'. I only refer to the submatrix because >>> it has a nullspace, and I want to find it by solving a linear system >>> designed for this purpose, H' u = b, where b is not the zero vector. H' >>> has no nullspace, but I want to remove the projection of u from my SNES >>> solution vector, as u IS in the nullspace of the submatrix. >>> >>> The RHS vector b can be calculated outside the SNES solve. I guess my >>> FormJacobian() should look like this: >>> >>> FormJacobian(SNES snes, Vec x, Mat Amat, Mat Pmat, void *ctx) { >>> KSP ksp; >>> MatNullSpace unull; >>> user_struct *user = (user_struct*)ctx; >>> >>> calculate Amat >>> assemble Amat >>> >>> if (Pmat != Amat) { >>> assemble Amat >>> } >>> >>> SNESGetKSP(snes,&ksp); >>> KSPSetOperators(ksp,Amat,Pmat); >>> KSPSolve(ksp,user->b,user->u); >>> >>> MatNullSpaceCreate(PetscObjectComm((PetscObject)Amat), PETSC_FALSE, 1, >>> &(user->u),&unull); >>> MatSetNullSpace(Amat,unull); >>> MatNullSpaceDestroy(&unull); >>> } >>> >>> Does this look right? >>> >>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Sep 29 09:09:24 2018 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 29 Sep 2018 10:09:24 -0400 Subject: [petsc-users] Fwd: Implementing a homotopy solver In-Reply-To: References: <6377AF62-D548-4276-A2A6-158EE4513593@anl.gov> <00D16613-4B28-4801-96EA-7CE2EF8BF8F7@imperial.ac.uk> <29B68A54-9D29-4881-AD80-98A5E510FD82@anl.gov> Message-ID: On Sat, Sep 29, 2018 at 9:47 AM zakaryah wrote: > Hi Matt - thanks for all your help. > > Let's say I want exactly the same solver for the tangent vector and the > SNES update, so I should reuse the KSP. > > My attempt to do this looks like the summary of FormJacobian() in the > previous message: > > - assemble Jacobian > - assemble RHS > - get KSP from the SNES passed to FormJacobian() > - KSPSetFromOptions() > - KSPSetOperators() > - KSPSolve() > > I'm not sure that's the right approach, but it doesn't work - the > KSPSolve() in the summary, i.e. the one for the tangent vector, seems to > work fine. But the next KSPSolve() looks strange - it seems to use a > preconditioner even with -pc_type none, etc. This makes me think I am > doing something seriously wrong. > 1) To get things going, just make the separate. Then we can optimize. 2) Give -ksp_view -ksp_monitor_true_residual -ksp_converged_reason to see what is happening Matt > On Sat, Sep 29, 2018, 8:16 AM Matthew Knepley wrote: > >> On Fri, Sep 28, 2018 at 11:13 PM zakaryah wrote: >> >>> I'm working on a homotopy solver which follows a zero curve through >>> solution space. A central aspect of this method is to calculate the vector >>> tangent to the curve, predict the next point, then correct using iteration >>> of, e.g. Newton's method, orthogonal to the tangent vector. >>> >>> Previously, we discussed the possibilities of implementing this within >>> PETSc's SNES. Within my FormJacobian() function, I construct the linear >>> system which defines the tangent vector, solve it, then add the vector to >>> the nullspace of the Jacobian. I think that in principle this can work, >>> but I suspect I'm doing something wrong. >>> >>> Here's a summary of the code within FormJacobian(): >>> >>> - Set values and assemble Jacobian matrix A - this is working fine >>> - Set values and assemble RHS vector b for linear system defining >>> tangent vector n - this is working fine >>> - SNESGetKSP(snes,&my_ksp) - I thought it made sense to use the KSP >>> associated with the SNES, hoping that PCs which use a factorization could >>> be reused when the SNES calls KSPSolve() to calculate the update >>> - KSPSetFromOptions(my_ksp) - not sure this is necessary but one of >>> my problems is setting options for this KSP from the command line and even >>> with this call it doesn't seem to be working properly >>> - MatSetNullSpace(A,NULL) - remove any existing null space from >>> Jacobian >>> - KSPSetOperators(my_ksp,A,P) - P is the other matrix in >>> FormJacobian() >>> - VecSet(n,0) - set initial guess to zero >>> - KSPSolve(my_ksp,b,n) - these solves appear to work, i.e. use the >>> options passed from the command line with -ksp_XXX or -pc_XXX >>> - VecNormalize(n,NULL) >>> - >>> MatNullSpaceCreate(PetscObjectComm((PetscObject)A),PETSC_FALSE,1,&n,&nullsp) >>> - MatSetNullSpace(A,nullsp) >>> - MatNullSpaceDestroy(&nullsp) >>> - return >>> >>> The immediate problem is that the subsequent KSPSolve(), i.e. the one >>> called internally by SNESSolve(), behaves strangely. For example, if I use >>> -pc_type none -ksp_monitor -ksp_monitor_true_residual, then the KSPSolve() >>> that I call within FormJacobian() looks correct - "preconditioned" norm and >>> true norm are identical, and both converge as I expect (i.e. slowly but >>> geometrically). However, the subsequent KSPSolve(), internal to the >>> SNESSolve(), has large differences between the preconditioned norm and the >>> true norm. In addition, the KSP does not converge in the true residual, >>> but I'll have a hard time debugging that without knowing how to properly >>> set the options. >>> >> >> We need to clear up the usage first. If you want EXACTLY the same solver >> for both solvers, then reuse >> the KSP, otherwise do not do it. Does it work then? >> >> Thanks, >> >> Matt >> >> >>> I hope someone can help me see what I'm doing wrong. >>> >>> On Sun, Jul 22, 2018 at 9:09 PM zakaryah wrote: >>> >>>> Thanks Matt and Barry, >>>> >>>> Matt - if I do the calculation in FormJacobian(), which makes by far >>>> the most sense and is as per your suggestion, do I need to set the >>>> operators of the SNES's KSP back to whatever they were before I set them? The >>>> linear system I want to solve within FormJacobian() involves the Jacobian >>>> matrix itself, and I want to remove the "nullspace" from that same matrix >>>> within FormFunction(). >>>> >>>> Barry - I'm trying to implement a homotopy solver. In short, I have a >>>> system of n nonlinear equations in n variables, F(x), which is hard to >>>> solve because the Jacobian tends to become singular using standard >>>> methods. I want to add an auxiliary variable, lambda, to create a >>>> homotopy: H(lambda,x) = lambda*F(x) + (1-lambda)G(x), where G is "easy to >>>> solve", and the idea is that the Jacobian of the n+1 variable system will >>>> not become singular along the curve H(lambda,x) = 0. >>>> >>>> The method involves adding an equation to H so that the Jacobian H' is >>>> square. The "submatrix" refers to the n x (n+1) matrix which represents >>>> the Jacobian without the added equation, whereas my FormJacobian() forms >>>> the entire (n+1) x (n+1) matrix H'. I only refer to the submatrix because >>>> it has a nullspace, and I want to find it by solving a linear system >>>> designed for this purpose, H' u = b, where b is not the zero vector. H' >>>> has no nullspace, but I want to remove the projection of u from my SNES >>>> solution vector, as u IS in the nullspace of the submatrix. >>>> >>>> The RHS vector b can be calculated outside the SNES solve. I guess my >>>> FormJacobian() should look like this: >>>> >>>> FormJacobian(SNES snes, Vec x, Mat Amat, Mat Pmat, void *ctx) { >>>> KSP ksp; >>>> MatNullSpace unull; >>>> user_struct *user = (user_struct*)ctx; >>>> >>>> calculate Amat >>>> assemble Amat >>>> >>>> if (Pmat != Amat) { >>>> assemble Amat >>>> } >>>> >>>> SNESGetKSP(snes,&ksp); >>>> KSPSetOperators(ksp,Amat,Pmat); >>>> KSPSolve(ksp,user->b,user->u); >>>> >>>> MatNullSpaceCreate(PetscObjectComm((PetscObject)Amat), PETSC_FALSE, >>>> 1, &(user->u),&unull); >>>> MatSetNullSpace(Amat,unull); >>>> MatNullSpaceDestroy(&unull); >>>> } >>>> >>>> Does this look right? >>>> >>>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Sep 29 10:59:49 2018 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 29 Sep 2018 11:59:49 -0400 Subject: [petsc-users] DMPlexDistribute segmentation fault In-Reply-To: References: Message-ID: On Fri, Sep 28, 2018 at 9:57 PM Josh L wrote: > Hi, > > I am implementing DMPlex to handle my mesh in parallel, but I am getting > segmentation fault during DMPlexDistribute. > i am testing with 2 processors on a toy mesh that only has 4 quad elements. > > the code is as following > CALL DMPlexCreateFromCellList > Check mesh topology. Face, skeleton symmetry > CALL DMPlexDistribute (dmserial,0,PETSC_NULL_SF,dmmpi,ierr) . > (segmentation fault on rank #1 here) > Crud, I was missing a check that the overlap is different on different procs. This is now fixed. Second it might be that your Fortran ints are not the same as PetscInt. Try declaring it PetscInt overlap = 0 and then passing 'overlap' instead. Thanks, Matt > I trace it back to the external library "Chaco" called by PETSc to > partition the mesh. > > The following is the stack > > For rank #1 > > #10 main () at /work/03691/yslo/stampede2/linear/rewrite/src/main.F90:18 > (at 0x000000000040569d) > #9 createmesh (rank=1, nsize=2, dmmpi=...) at > /work/03691/yslo/stampede2/linear/rewrite/src/createmesh.F90:107 (at > 0x000000000040523b) > #8 dmplexdistribute_ (dm=0x320, overlap=0x1, sf=0x7fffffff4480, > dmParallel=0x2aaab78a585d, ierr=0x1) at > /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/ftn-custom/zplexdistribute.c:15 > (at 0x00002aaaac08334e) > #7 DMPlexDistribute (dm=0x320, overlap=1, sf=0x7fffffff4480, > dmParallel=0x2aaab78a585d) at > /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexdistribute.c:1664 > (at 0x00002aaaabd12d08) > #6 PetscPartitionerPartition (part=0x320, dm=0x1, > partSection=0x7fffffff4480, partition=0x2aaab78a585d) at > /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexpartition.c:675 > (at 0x00002aaaabcfe92c) > #5 PetscPartitionerPartition_Chaco (part=0x320, dm=0x1, nparts=-48000, > numVertices=-1215670179, start=0x1, adjacency=0x22, partSection=0x0, > partition=0x772120) at > /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexpartition.c:1279 > (at 0x00002aaaabd03dfd) > #4 interface.Z (nvtxs=800, start=0x1, adjacency=0x7fffffff4480, > vwgts=0x2aaab78a585d, ewgts=0x1, x=0x22, y=0x0, z=0x0, outassignname=0x0, > outfilename=0x0, assignment=0x76ff90, architecture=1, ndims_tot=0, > mesh_dims=0x7fffffff7038, goal=0x0, global_method=1, local_method=1, > rqi_flag=0, vmax=200, ndims=1, eigtol=2.1137067449068142e-314, > seed=123636512) at > /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/main/interface.c:206 > (at 0x00002aaaad7aaf70) > #3 submain.Z (graph=0x320, nvtxs=1, nedges=-48000, > using_vwgts=-1215670179, using_ewgts=1, igeom=34, coords=0x0, > outassignname=0x0, outfilename=0x0, assignment=0x76ff8e, goal=0x7739a0, > architecture=1, ndims_tot=0, mesh_dims=0x7fffffff7038, global_method=1, > local_method=1, rqi_flag=0, vmax=200, ndims=1, > eigtol=2.1137067449068142e-314, seed=123636512) at > /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/submain/submain.c:151 > (at 0x00002aaaad7ae52e) > #2 check_input.Z (graph=0x320, nvtxs=1, nedges=-48000, igeom=-1215670179, > coords=0x1, graphname=0x22 , > assignment=0x76ff8e, goal=0x7739a0, architecture=1, ndims_tot=0, > mesh_dims=0x7fffffff7038, global_method=1, local_method=1, rqi_flag=0, > vmax=0x7fffffff47a8, ndims=1, eigtol=2.1137067449068142e-314) at > /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/input/check_input.c:56 > (at 0x00002aaaad7c6ed1) > #1 check_graph (graph=0x320, nvtxs=1, nedges=-48000) at > /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/graph/check_graph.c:90 > (at 0x00002aaaad8204d3) > #0 is_an_edge (vertex=0x320, v2=1, weight2=0x7fffffff4480) at > /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/graph/check_graph.c:134 > (at 0x00002aaaad8206d3) > > For rank #0 > > #18 main () at /work/03691/yslo/stampede2/linear/rewrite/src/main.F90:18 > (at 0x000000000040569d) > #17 createmesh (rank=0, nsize=2, dmmpi=...) at > /work/03691/yslo/stampede2/linear/rewrite/src/createmesh.F90:107 (at > 0x000000000040523b) > #16 dmplexdistribute_ (dm=0x65f300, overlap=0x0, sf=0x2aaab4c8698c, > dmParallel=0xffffffffffffffff, ierr=0x0) at > /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/ftn-custom/zplexdistribute.c:15 > (at 0x00002aaaac08334e) > #15 DMPlexDistribute (dm=0x65f300, overlap=0, sf=0x2aaab4c8698c, > dmParallel=0xffffffffffffffff) at > /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexdistribute.c:1664 > (at 0x00002aaaabd12d08) > #14 PetscPartitionerPartition (part=0x65f300, dm=0x0, > partSection=0x2aaab4c8698c, partition=0xffffffffffffffff) at > /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexpartition.c:675 > (at 0x00002aaaabcfe92c) > #13 PetscPartitionerPartition_Chaco (part=0x65f300, dm=0x0, > nparts=-1261934196, numVertices=-1, start=0x0, adjacency=0x0, > partSection=0x7726c0, partition=0x7fffffff71a8) at > /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexpartition.c:1314 > (at 0x00002aaaabd04029) > #12 ISCreateGeneral (comm=6681344, n=0, idx=0x2aaab4c8698c, > mode=(PETSC_OWN_POINTER | PETSC_USE_POINTER | unknown: 4294967292), is=0x0) > at > /home1/apps/intel17/impi17_0/petsc/3.9/src/vec/is/is/impls/general/general.c:671 > (at 0x00002aaaab0ea94e) > #11 ISGeneralSetIndices (is=0x65f300, n=0, idx=0x2aaab4c8698c, > mode=(PETSC_OWN_POINTER | PETSC_USE_POINTER | unknown: 4294967292)) at > /home1/apps/intel17/impi17_0/petsc/3.9/src/vec/is/is/impls/general/general.c:698 > (at 0x00002aaaab0eaa77) > #10 ISGeneralSetIndices_General.Z (is=0x65f300, n=0, idx=0x2aaab4c8698c, > mode=(PETSC_OWN_POINTER | PETSC_USE_POINTER | unknown: 4294967292)) at > /home1/apps/intel17/impi17_0/petsc/3.9/src/vec/is/is/impls/general/general.c:712 > (at 0x00002aaaab0ef567) > #9 PetscLayoutSetUp (map=0x65f300) at > /home1/apps/intel17/impi17_0/petsc/3.9/src/vec/is/utils/pmap.c:137 (at > 0x00002aaaab0c6d52) > #8 PetscSplitOwnership (comm=6681344, n=0x0, N=0x2aaab4c8698c) at > /home1/apps/intel17/impi17_0/petsc/3.9/src/sys/utils/psplit.c:80 (at > 0x00002aaaaaeb7b00) > #7 PMPI_Allreduce (sendbuf=0x65f300, recvbuf=0x0, count=-1261934196, > datatype=-1, op=0, comm=0) at > /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/mpi/coll/allreduce.c:1395 > (at 0x00002aaab40966e6) > #6 MPIR_Allreduce_intra (sendbuf=0x65f300, recvbuf=0x0, count=-1261934196, > datatype=-1, op=0, comm_ptr=0x0, errflag=0x7fffffff4798) at > /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/mpi/coll/allreduce.c:339 > (at 0x00002aaab409307b) > #5 MPIR_Allreduce_shm_generic (sendbuf=, recvbuf= out>, count=, datatype=, op=, > comm_ptr=, errflag=, kind=1476395011) at > /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/mpi/coll/allreduce.c:1137 > (at 0x00002aaab409307b) > #4 I_MPI_COLL_SHM_KNARY_REDUCE (node_comm_ptr=, > root=, localbuf=, sendbuf=, > recvbuf=, count=, datatype=, > op=, errflag=, knomial_factor= out>) at > /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/I_MPI/include/shm_coll_templating.h:1090 > (at 0x00002aaab409307b) > #3 I_MPI_COLL_SHM_GENERIC_GATHER_REDUCE..1 (node_comm_ptr=0x65f300, > root=0, is_reduce=-1261934196, localbuf=0xffffffffffffffff, sendbuf=0x0, > recvbuf=0x0, count=1, datatype=1275069445, op=1476395011, > errflag=0x7fffffff4798, knomial_factor=4, algo_type=2) at > /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/I_MPI/include/shm_coll_templating.h:558 > (at 0x00002aaab408f3a9) > #2 I_MPI_memcpy (destination=, source=0x0, size= out>) at > /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/I_MPI/include/shm_coll_templating.h:749 > (at 0x00002aaab408f3a9) > #1 PMPIDI_CH3I_Progress (progress_state=0x65f300, is_blocking=0) at > /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/mpid/ch3/channels/nemesis/src/ch3_progress.c:981 > (at 0x00002aaab40e85a6) > #0 sched_yield () from /lib64/libc.so.6 (at 0x00002aaab7898e47) > > > Any idea why this is happening? the overlap is set to 0, but it is 1 on > rank #1 > > > Is there anyway to know the cell and vertex number distributed on each > processors? > my old code partitions the mesh with Metis, and I always output cell data > that shows the rank number on each cell, so I can visualize how mesh is > partitioned. > It is not necessary, but is there anyway to get it in DMPlex? > DMPlexGetCellNumbering might work, but it fails the linking. In fact, many > function under developer category fails the linking. > > > Thanks, > Yu-Sheng > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Sat Sep 29 19:12:55 2018 From: dave.mayhem23 at gmail.com (Dave May) Date: Sun, 30 Sep 2018 02:12:55 +0200 Subject: [petsc-users] Fwd: Implementing a homotopy solver In-Reply-To: References: <6377AF62-D548-4276-A2A6-158EE4513593@anl.gov> <00D16613-4B28-4801-96EA-7CE2EF8BF8F7@imperial.ac.uk> <29B68A54-9D29-4881-AD80-98A5E510FD82@anl.gov> Message-ID: On Sat, 29 Sep 2018 at 16:09, Matthew Knepley wrote: > On Sat, Sep 29, 2018 at 9:47 AM zakaryah wrote: > >> Hi Matt - thanks for all your help. >> >> Let's say I want exactly the same solver for the tangent vector and the >> SNES update, so I should reuse the KSP. >> > If you want to do this, there is no need or reason to call KSPSetFromOptions() inside your Jacobian evaluator - just call SNESSetFromOptions once on the outer object. >> My attempt to do this looks like the summary of FormJacobian() in the >> previous message: >> >> - assemble Jacobian >> - assemble RHS >> - get KSP from the SNES passed to FormJacobian() >> - KSPSetFromOptions() >> - KSPSetOperators() >> - KSPSolve() >> >> I'm not sure that's the right approach, but it doesn't work - the >> KSPSolve() in the summary, i.e. the one for the tangent vector, seems to >> work fine. But the next KSPSolve() looks strange - it seems to use a >> preconditioner even with -pc_type none, etc. This makes me think I am >> doing something seriously wrong. >> > > 1) To get things going, just make the separate. Then we can optimize. > > 2) Give -ksp_view -ksp_monitor_true_residual -ksp_converged_reason to see > what is happening > > Matt > > >> On Sat, Sep 29, 2018, 8:16 AM Matthew Knepley wrote: >> >>> On Fri, Sep 28, 2018 at 11:13 PM zakaryah wrote: >>> >>>> I'm working on a homotopy solver which follows a zero curve through >>>> solution space. A central aspect of this method is to calculate the vector >>>> tangent to the curve, predict the next point, then correct using iteration >>>> of, e.g. Newton's method, orthogonal to the tangent vector. >>>> >>>> Previously, we discussed the possibilities of implementing this within >>>> PETSc's SNES. Within my FormJacobian() function, I construct the linear >>>> system which defines the tangent vector, solve it, then add the vector to >>>> the nullspace of the Jacobian. I think that in principle this can work, >>>> but I suspect I'm doing something wrong. >>>> >>>> Here's a summary of the code within FormJacobian(): >>>> >>>> - Set values and assemble Jacobian matrix A - this is working fine >>>> - Set values and assemble RHS vector b for linear system defining >>>> tangent vector n - this is working fine >>>> - SNESGetKSP(snes,&my_ksp) - I thought it made sense to use the KSP >>>> associated with the SNES, hoping that PCs which use a factorization could >>>> be reused when the SNES calls KSPSolve() to calculate the update >>>> - KSPSetFromOptions(my_ksp) - not sure this is necessary but one of >>>> my problems is setting options for this KSP from the command line and even >>>> with this call it doesn't seem to be working properly >>>> - MatSetNullSpace(A,NULL) - remove any existing null space from >>>> Jacobian >>>> - KSPSetOperators(my_ksp,A,P) - P is the other matrix in >>>> FormJacobian() >>>> - VecSet(n,0) - set initial guess to zero >>>> - KSPSolve(my_ksp,b,n) - these solves appear to work, i.e. use the >>>> options passed from the command line with -ksp_XXX or -pc_XXX >>>> - VecNormalize(n,NULL) >>>> - >>>> MatNullSpaceCreate(PetscObjectComm((PetscObject)A),PETSC_FALSE,1,&n,&nullsp) >>>> - MatSetNullSpace(A,nullsp) >>>> - MatNullSpaceDestroy(&nullsp) >>>> - return >>>> >>>> The immediate problem is that the subsequent KSPSolve(), i.e. the one >>>> called internally by SNESSolve(), behaves strangely. For example, if I use >>>> -pc_type none -ksp_monitor -ksp_monitor_true_residual, then the KSPSolve() >>>> that I call within FormJacobian() looks correct - "preconditioned" norm and >>>> true norm are identical, and both converge as I expect (i.e. slowly but >>>> geometrically). However, the subsequent KSPSolve(), internal to the >>>> SNESSolve(), has large differences between the preconditioned norm and the >>>> true norm. In addition, the KSP does not converge in the true residual, >>>> but I'll have a hard time debugging that without knowing how to properly >>>> set the options. >>>> >>> >>> We need to clear up the usage first. If you want EXACTLY the same solver >>> for both solvers, then reuse >>> the KSP, otherwise do not do it. Does it work then? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> I hope someone can help me see what I'm doing wrong. >>>> >>>> On Sun, Jul 22, 2018 at 9:09 PM zakaryah wrote: >>>> >>>>> Thanks Matt and Barry, >>>>> >>>>> Matt - if I do the calculation in FormJacobian(), which makes by far >>>>> the most sense and is as per your suggestion, do I need to set the >>>>> operators of the SNES's KSP back to whatever they were before I set them? The >>>>> linear system I want to solve within FormJacobian() involves the Jacobian >>>>> matrix itself, and I want to remove the "nullspace" from that same matrix >>>>> within FormFunction(). >>>>> >>>>> Barry - I'm trying to implement a homotopy solver. In short, I have a >>>>> system of n nonlinear equations in n variables, F(x), which is hard to >>>>> solve because the Jacobian tends to become singular using standard >>>>> methods. I want to add an auxiliary variable, lambda, to create a >>>>> homotopy: H(lambda,x) = lambda*F(x) + (1-lambda)G(x), where G is "easy to >>>>> solve", and the idea is that the Jacobian of the n+1 variable system will >>>>> not become singular along the curve H(lambda,x) = 0. >>>>> >>>>> The method involves adding an equation to H so that the Jacobian H' is >>>>> square. The "submatrix" refers to the n x (n+1) matrix which represents >>>>> the Jacobian without the added equation, whereas my FormJacobian() forms >>>>> the entire (n+1) x (n+1) matrix H'. I only refer to the submatrix because >>>>> it has a nullspace, and I want to find it by solving a linear system >>>>> designed for this purpose, H' u = b, where b is not the zero vector. H' >>>>> has no nullspace, but I want to remove the projection of u from my SNES >>>>> solution vector, as u IS in the nullspace of the submatrix. >>>>> >>>>> The RHS vector b can be calculated outside the SNES solve. I guess my >>>>> FormJacobian() should look like this: >>>>> >>>>> FormJacobian(SNES snes, Vec x, Mat Amat, Mat Pmat, void *ctx) { >>>>> KSP ksp; >>>>> MatNullSpace unull; >>>>> user_struct *user = (user_struct*)ctx; >>>>> >>>>> calculate Amat >>>>> assemble Amat >>>>> >>>>> if (Pmat != Amat) { >>>>> assemble Amat >>>>> } >>>>> >>>>> SNESGetKSP(snes,&ksp); >>>>> KSPSetOperators(ksp,Amat,Pmat); >>>>> KSPSolve(ksp,user->b,user->u); >>>>> >>>>> MatNullSpaceCreate(PetscObjectComm((PetscObject)Amat), PETSC_FALSE, >>>>> 1, &(user->u),&unull); >>>>> MatSetNullSpace(Amat,unull); >>>>> MatNullSpaceDestroy(&unull); >>>>> } >>>>> >>>>> Does this look right? >>>>> >>>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ysjosh.lo at gmail.com Sat Sep 29 19:47:01 2018 From: ysjosh.lo at gmail.com (Josh L) Date: Sat, 29 Sep 2018 19:47:01 -0500 Subject: [petsc-users] DMPlexDistribute segmentation fault In-Reply-To: References: Message-ID: It is actually my mistake. Although every rank should call CreateFromCellList, only 1 proc needs to read in the mesh, and other just create empty mesh. It works fine now. Thanks, Josh Matthew Knepley ? 2018?9?29? ?? ??11:00??? > On Fri, Sep 28, 2018 at 9:57 PM Josh L wrote: > >> Hi, >> >> I am implementing DMPlex to handle my mesh in parallel, but I am getting >> segmentation fault during DMPlexDistribute. >> i am testing with 2 processors on a toy mesh that only has 4 quad >> elements. >> >> the code is as following >> CALL DMPlexCreateFromCellList >> Check mesh topology. Face, skeleton symmetry >> CALL DMPlexDistribute (dmserial,0,PETSC_NULL_SF,dmmpi,ierr) . >> (segmentation fault on rank #1 here) >> > > Crud, I was missing a check that the overlap is different on different > procs. This is now fixed. > > Second it might be that your Fortran ints are not the same as PetscInt. > Try declaring it > > PetscInt overlap = 0 > > and then passing 'overlap' instead. > > Thanks, > > Matt > > >> I trace it back to the external library "Chaco" called by PETSc to >> partition the mesh. >> >> The following is the stack >> >> For rank #1 >> >> #10 main () at /work/03691/yslo/stampede2/linear/rewrite/src/main.F90:18 >> (at 0x000000000040569d) >> #9 createmesh (rank=1, nsize=2, dmmpi=...) at >> /work/03691/yslo/stampede2/linear/rewrite/src/createmesh.F90:107 (at >> 0x000000000040523b) >> #8 dmplexdistribute_ (dm=0x320, overlap=0x1, sf=0x7fffffff4480, >> dmParallel=0x2aaab78a585d, ierr=0x1) at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/ftn-custom/zplexdistribute.c:15 >> (at 0x00002aaaac08334e) >> #7 DMPlexDistribute (dm=0x320, overlap=1, sf=0x7fffffff4480, >> dmParallel=0x2aaab78a585d) at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexdistribute.c:1664 >> (at 0x00002aaaabd12d08) >> #6 PetscPartitionerPartition (part=0x320, dm=0x1, >> partSection=0x7fffffff4480, partition=0x2aaab78a585d) at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexpartition.c:675 >> (at 0x00002aaaabcfe92c) >> #5 PetscPartitionerPartition_Chaco (part=0x320, dm=0x1, nparts=-48000, >> numVertices=-1215670179, start=0x1, adjacency=0x22, partSection=0x0, >> partition=0x772120) at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexpartition.c:1279 >> (at 0x00002aaaabd03dfd) >> #4 interface.Z (nvtxs=800, start=0x1, adjacency=0x7fffffff4480, >> vwgts=0x2aaab78a585d, ewgts=0x1, x=0x22, y=0x0, z=0x0, outassignname=0x0, >> outfilename=0x0, assignment=0x76ff90, architecture=1, ndims_tot=0, >> mesh_dims=0x7fffffff7038, goal=0x0, global_method=1, local_method=1, >> rqi_flag=0, vmax=200, ndims=1, eigtol=2.1137067449068142e-314, >> seed=123636512) at >> /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/main/interface.c:206 >> (at 0x00002aaaad7aaf70) >> #3 submain.Z (graph=0x320, nvtxs=1, nedges=-48000, >> using_vwgts=-1215670179, using_ewgts=1, igeom=34, coords=0x0, >> outassignname=0x0, outfilename=0x0, assignment=0x76ff8e, goal=0x7739a0, >> architecture=1, ndims_tot=0, mesh_dims=0x7fffffff7038, global_method=1, >> local_method=1, rqi_flag=0, vmax=200, ndims=1, >> eigtol=2.1137067449068142e-314, seed=123636512) at >> /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/submain/submain.c:151 >> (at 0x00002aaaad7ae52e) >> #2 check_input.Z (graph=0x320, nvtxs=1, nedges=-48000, igeom=-1215670179, >> coords=0x1, graphname=0x22 , >> assignment=0x76ff8e, goal=0x7739a0, architecture=1, ndims_tot=0, >> mesh_dims=0x7fffffff7038, global_method=1, local_method=1, rqi_flag=0, >> vmax=0x7fffffff47a8, ndims=1, eigtol=2.1137067449068142e-314) at >> /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/input/check_input.c:56 >> (at 0x00002aaaad7c6ed1) >> #1 check_graph (graph=0x320, nvtxs=1, nedges=-48000) at >> /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/graph/check_graph.c:90 >> (at 0x00002aaaad8204d3) >> #0 is_an_edge (vertex=0x320, v2=1, weight2=0x7fffffff4480) at >> /tmp/petsc-build/externalpackages/skylake/skylake/Chaco-2.2-p2/code/graph/check_graph.c:134 >> (at 0x00002aaaad8206d3) >> >> For rank #0 >> >> #18 main () at /work/03691/yslo/stampede2/linear/rewrite/src/main.F90:18 >> (at 0x000000000040569d) >> #17 createmesh (rank=0, nsize=2, dmmpi=...) at >> /work/03691/yslo/stampede2/linear/rewrite/src/createmesh.F90:107 (at >> 0x000000000040523b) >> #16 dmplexdistribute_ (dm=0x65f300, overlap=0x0, sf=0x2aaab4c8698c, >> dmParallel=0xffffffffffffffff, ierr=0x0) at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/ftn-custom/zplexdistribute.c:15 >> (at 0x00002aaaac08334e) >> #15 DMPlexDistribute (dm=0x65f300, overlap=0, sf=0x2aaab4c8698c, >> dmParallel=0xffffffffffffffff) at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexdistribute.c:1664 >> (at 0x00002aaaabd12d08) >> #14 PetscPartitionerPartition (part=0x65f300, dm=0x0, >> partSection=0x2aaab4c8698c, partition=0xffffffffffffffff) at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexpartition.c:675 >> (at 0x00002aaaabcfe92c) >> #13 PetscPartitionerPartition_Chaco (part=0x65f300, dm=0x0, >> nparts=-1261934196, numVertices=-1, start=0x0, adjacency=0x0, >> partSection=0x7726c0, partition=0x7fffffff71a8) at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/dm/impls/plex/plexpartition.c:1314 >> (at 0x00002aaaabd04029) >> #12 ISCreateGeneral (comm=6681344, n=0, idx=0x2aaab4c8698c, >> mode=(PETSC_OWN_POINTER | PETSC_USE_POINTER | unknown: 4294967292), is=0x0) >> at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/vec/is/is/impls/general/general.c:671 >> (at 0x00002aaaab0ea94e) >> #11 ISGeneralSetIndices (is=0x65f300, n=0, idx=0x2aaab4c8698c, >> mode=(PETSC_OWN_POINTER | PETSC_USE_POINTER | unknown: 4294967292)) at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/vec/is/is/impls/general/general.c:698 >> (at 0x00002aaaab0eaa77) >> #10 ISGeneralSetIndices_General.Z (is=0x65f300, n=0, idx=0x2aaab4c8698c, >> mode=(PETSC_OWN_POINTER | PETSC_USE_POINTER | unknown: 4294967292)) at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/vec/is/is/impls/general/general.c:712 >> (at 0x00002aaaab0ef567) >> #9 PetscLayoutSetUp (map=0x65f300) at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/vec/is/utils/pmap.c:137 (at >> 0x00002aaaab0c6d52) >> #8 PetscSplitOwnership (comm=6681344, n=0x0, N=0x2aaab4c8698c) at >> /home1/apps/intel17/impi17_0/petsc/3.9/src/sys/utils/psplit.c:80 (at >> 0x00002aaaaaeb7b00) >> #7 PMPI_Allreduce (sendbuf=0x65f300, recvbuf=0x0, count=-1261934196, >> datatype=-1, op=0, comm=0) at >> /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/mpi/coll/allreduce.c:1395 >> (at 0x00002aaab40966e6) >> #6 MPIR_Allreduce_intra (sendbuf=0x65f300, recvbuf=0x0, >> count=-1261934196, datatype=-1, op=0, comm_ptr=0x0, errflag=0x7fffffff4798) >> at >> /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/mpi/coll/allreduce.c:339 >> (at 0x00002aaab409307b) >> #5 MPIR_Allreduce_shm_generic (sendbuf=, >> recvbuf=, count=, datatype=, >> op=, comm_ptr=, errflag=, >> kind=1476395011) at >> /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/mpi/coll/allreduce.c:1137 >> (at 0x00002aaab409307b) >> #4 I_MPI_COLL_SHM_KNARY_REDUCE (node_comm_ptr=, >> root=, localbuf=, sendbuf=, >> recvbuf=, count=, datatype=, >> op=, errflag=, knomial_factor=> out>) at >> /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/I_MPI/include/shm_coll_templating.h:1090 >> (at 0x00002aaab409307b) >> #3 I_MPI_COLL_SHM_GENERIC_GATHER_REDUCE..1 (node_comm_ptr=0x65f300, >> root=0, is_reduce=-1261934196, localbuf=0xffffffffffffffff, sendbuf=0x0, >> recvbuf=0x0, count=1, datatype=1275069445, op=1476395011, >> errflag=0x7fffffff4798, knomial_factor=4, algo_type=2) at >> /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/I_MPI/include/shm_coll_templating.h:558 >> (at 0x00002aaab408f3a9) >> #2 I_MPI_memcpy (destination=, source=0x0, size=> out>) at >> /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/I_MPI/include/shm_coll_templating.h:749 >> (at 0x00002aaab408f3a9) >> #1 PMPIDI_CH3I_Progress (progress_state=0x65f300, is_blocking=0) at >> /tmp/mpi.xtmpdir.7b663e0dc22b2304e487307e376dc132.14974_32e/mpi.32e.ww14.20170405/dev/x86_64/release_mt/../../src/mpid/ch3/channels/nemesis/src/ch3_progress.c:981 >> (at 0x00002aaab40e85a6) >> #0 sched_yield () from /lib64/libc.so.6 (at 0x00002aaab7898e47) >> >> >> Any idea why this is happening? the overlap is set to 0, but it is 1 on >> rank #1 >> >> >> Is there anyway to know the cell and vertex number distributed on each >> processors? >> my old code partitions the mesh with Metis, and I always output cell data >> that shows the rank number on each cell, so I can visualize how mesh is >> partitioned. >> It is not necessary, but is there anyway to get it in DMPlex? >> DMPlexGetCellNumbering might work, but it fails the linking. In fact, >> many function under developer category fails the linking. >> >> >> Thanks, >> Yu-Sheng >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zakaryah at gmail.com Sun Sep 30 17:05:49 2018 From: zakaryah at gmail.com (zakaryah) Date: Sun, 30 Sep 2018 18:05:49 -0400 Subject: [petsc-users] Fwd: Implementing a homotopy solver In-Reply-To: References: <6377AF62-D548-4276-A2A6-158EE4513593@anl.gov> <00D16613-4B28-4801-96EA-7CE2EF8BF8F7@imperial.ac.uk> <29B68A54-9D29-4881-AD80-98A5E510FD82@anl.gov> Message-ID: OK, thanks. I'm using a composite DM, DM_packer. To make a separate KSP, I did the following in the FormJacobian() routine, after assembling the Jacobian matrix A and the RHS for the tangent vector, b: - KSPCreate(PETSC_COMM_WORLD,&ksp) - KSPSetDM(ksp,DM_packer) - KSPSetDMActive(ksp,PETSC_FALSE) - because I want to set the operators - KSPSetOperators(ksp,A,P) - VecSet(n,0) - set initial guess to zero - KSPSolve(ksp,b,n) - this solve works correctly - VecNormalize(n,NULL) - MatNullSpaceCreate(PetscObjectComm((PetscObject)A),PETSC_FALSE,1,&n,&nullsp) - MatSetNullSpace(A,nullsp) Then, with -snes_type newtonls -pc_type none -ksp_monitor -ksp_monitor_true_residual -ksp_view, the output for the KSPSolve described above, i.e. for the tangent vector, looks correct: 0 KSP preconditioned resid norm 1.000000000000e+03 true resid norm 1.000000000000e+03 ||r(i)||/||b|| 1.000000000000e+00 ... 185 KSP preconditioned resid norm 9.900713131874e-03 true resid norm 9.900713131904e-03 ||r(i)||/||b|| 9.900713131904e-06 Linear solve converged due to CONVERGED_RTOL iterations 185 KSP Object: 1 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=78247, cols=78247 total: nonzeros=6063481, allocated nonzeros=6063481 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 26083 nodes, limit used is 5 But the next KSP, i.e. the one in the SNES, doesn't converge in the true residual: 0 KSP preconditioned resid norm 2.045599092896e-04 true resid norm 2.803828296212e-04 ||r(i)||/||b|| 1.000000000000e+00 191 KSP preconditioned resid norm 2.009941278534e-09 true resid norm 1.636010142734e-04 ||r(i)||/||b|| 5.834915586465e-01 KSP Object: 1 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: none linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=78247, cols=78247 total: nonzeros=6063481, allocated nonzeros=6063481 total number of mallocs used during MatSetValues calls =0 has attached null space using I-node routines: found 26083 nodes, limit used is 5 My guess about what's going on is that the tangent vector n isn't really in the nullspace of A. Rather, it's in the nullspace of the m x (m+1) submatrix of A. So, An=c e_{m+1}, where c is an arbitrary constant and e_{m+1} is the m+1 th basis vector. The nonlinear function also has an added row, F_{m+1}, which is set to zero in the FormFunction() routine. I don't care about the value of F_{m+1}, but I suppose that if it's included in the true residual, and NOT in the "preconditioned" residual, even with pc_type none, then it will be tricky to diagnose the performance. Should I be using my own routine to evaluate the residual, so that F_{m+1} is not included? -------------- next part -------------- An HTML attachment was scrubbed... URL: