[petsc-users] Need help improving working PETSC code

[Barry Smith]

   Please provide a bit more detail about the operator. Is it a pressure solver for CFD? Cell centered? Does the matrix has a null space of the constant functions? Is it the same linear system for each "time-step?" in the CFD solver or different?

   How many iterations is hypre BoomerAMG typically taking? 

   This is an extremely tight tolerance, why do you set it so small? KSPSetTolerances(ksp, 1e-13,  1e-13,  PETSC_DEFAULT,  PETSC_DEFAULT) ;

   Send the output from a run with -log_summary so we can see where the time is being spent.

   Barry


   
> On Jul 8, 2015, at 11:11 AM, Ganesh Vijayakumar <ganesh.iitm at gmail.com> wrote:
> 
> Hello,
> 
>  First of all..thanks to the PETSC developers and everyone else contributing supporting material on the web. 
> 
>  I need to solve a system of equations Ax =b, where A is symmetric, sparse, unstructured and in parallel as a part of a finite volume solver for CFD. Decomposition is already done. I initialize the matrix as MPIAIJ even though it's symmetric as that's the only thing I could get to work.
> 
> MatCreate(PETSC_COMM_WORLD,&A);                                                                                                  
> MatSetType(A,MATMPIAIJ);
> MatSetSizes(A,nCellsCurProc,nCellsCurProc,nTotalCells,nTotalCells);                                                           
> MatMPIAIJSetPreallocation(A,10,PETSC_NULL,5,PETSC_NULL);                         
> MatSetUp(A) ;
> // Assemble matrix using MatSetValue.. assemble full matrix even though it's symmetric.
> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY) ;
> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY) ;
> //Create solver
> KSPCreate(PETSC_COMM_WORLD,&ksp);
> KSPSetType(ksp,KSPFGMRES;)                                                                                                          
> KSPGetPC(ksp, &pc);
> PCFactorSetShiftType(pc, MAT_SHIFT_POSITIVE_DEFINITE);
> 
> //PCSetType(pc, PCML); //Trilinos - Multilevel
> 
> //PCSetType(pc, PCBJACOBI) ; //Incomplete LU
> //PetscOptionsSetValue("-sub_pc_type", "ilu") ;
> KSPSetFromOptions(ksp);                                                                                                            
> 
> // PCSetType(pc, PCASM); //Additive Schwarz Methods
> // PCASMSetOverlap(pc,overlap);
> 
> PCSetType(pc, PCHYPRE); //HYPRE
> PCHYPRESetType(pc, "boomeramg");
> 
> //PCHYPRESetType(pc, "parasails");                    
> 
> KSPSetPC(ksp, pc);
> KSPSetTolerances(ksp, 1e-13,  1e-13,  PETSC_DEFAULT,  PETSC_DEFAULT) ;
> KSPSetInitialGuessNonzero(ksp, PETSC_TRUE);
> KSPSetOperators(ksp, A, A);
> KSPMonitorDefault(ksp, 1, 1, pVoid) ;
> 
> KSPSolve(ksp,b,x);
> VecGetArray(x, &getResultArray) ;
> KSPGetResidualNorm(ksp, &endNorm) ;
> KSPGetIterationNumber(ksp, &nIterations);
> 
> KSPDestroy(&ksp);
> 
> 
>  I know that this works and gives me the correct result. However, boomerAMG is proving to be too slow.. atleast the way I use it at 256 processors with around 12 million unknowns. I need your advice on the preconditioner. Am I using it right? Is there anything else I could I be doing better?
> 
> ganesh