[From nobody Wed Aug 23 15:40:50 2017 From: "Chetan Jhurani" <chetan.jhurani@gmail.com> To: "'For users of the development version of PETSc'" <petsc-dev@mcs.anl.gov> Subject: discrepancy with bcgs, mpi, cusp, and no preconditioning Date: Thu, 10 May 2012 12:45:30 -0700 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_00A0_01CE2199.D4B7EEE0" X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Ac0u5XTijDQD/nasToO8PCfAcmou/Q== Content-Language: en-us This is a multi-part message in MIME format. ------=_NextPart_000_00A0_01CE2199.D4B7EEE0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi wizards of "->ops->" indirections, I'm trying to use bcgs without preconditioning, for now, and the iterations using "-vec_type cusp -mat_type mpiaijcusp" don't match serial or non-GPU options. I've attached the test program and the 4 outputs (serial/parallel + CPU/GPU). All this is with petsc-dev downloaded now and real scalars. Only the parallel GPU results are different starting from third residual norm seen in results.txt. The other three match one another. Am I doing something wrong? fbcgs (bcgs with -ksp_bcgs_flexible) works fine with all the serial/parallel or CPU/GPU options I've tried. Let me know if you need the matrix, rhs, and initial guess binary files that are read in by the test program. Thanks, Chetan ------=_NextPart_000_00A0_01CE2199.D4B7EEE0 Content-Type: text/plain; name="test_ksp.cpp" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="test_ksp.cpp" #include <petscmat.h> #include <petscvec.h> #include <petscksp.h> #include <vector> #include <fstream> #include <iostream> PetscErrorCode read_vec(Vec& out, const char* file) { PetscErrorCode ierr = 0; ierr = VecCreate(PETSC_COMM_WORLD, &out); CHKERRQ(ierr); ierr = VecSetFromOptions(out); CHKERRQ(ierr); PetscViewer reader; ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD, file, FILE_MODE_READ, &reader); CHKERRQ(ierr); ierr = VecLoad(out, reader); CHKERRQ(ierr); ierr = PetscViewerDestroy(&reader); CHKERRQ(ierr); return ierr; } PetscErrorCode read_mat(Mat& out, const char* file) { PetscErrorCode ierr = 0; ierr = MatCreate(PETSC_COMM_WORLD, &out); CHKERRQ(ierr); ierr = MatSetFromOptions(out); CHKERRQ(ierr); PetscViewer reader; ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD, file, FILE_MODE_READ, &reader); CHKERRQ(ierr); ierr = MatLoad(out, reader); CHKERRQ(ierr); ierr = PetscViewerDestroy(&reader); CHKERRQ(ierr); return ierr; } int main(int argc, char* argv[]) { PetscErrorCode ierr = 0; ierr = PetscInitialize(&argc, &argv, PETSC_NULL, PETSC_NULL); CHKERRQ(ierr); Mat A_mat; ierr = read_mat(A_mat, "a.petsc_bin"); CHKERRQ(ierr); Vec b_vec, x0_vec; ierr = read_vec(b_vec, "b.petsc_bin"); CHKERRQ(ierr); ierr = read_vec(x0_vec, "x0.petsc_bin"); CHKERRQ(ierr); KSP ksp; ierr = KSPCreate(PETSC_COMM_WORLD, &ksp); CHKERRQ(ierr); PC pc; ierr = KSPGetPC(ksp, &pc); CHKERRQ(ierr); ierr = PCSetType(pc, PCNONE); CHKERRQ(ierr); PetscReal rtol, abstol, dtol; PetscInt max_iter; ierr = KSPGetTolerances(ksp, &rtol, &abstol, &dtol, &max_iter); CHKERRQ(ierr); rtol = 1E-8; abstol = 0; max_iter = 10; ierr = KSPSetTolerances(ksp, rtol, abstol, dtol, max_iter); CHKERRQ(ierr); ierr = KSPSetOperators(ksp, A_mat, A_mat, SAME_PRECONDITIONER); CHKERRQ(ierr); ierr = KSPSetInitialGuessNonzero(ksp, PETSC_TRUE); CHKERRQ(ierr); ierr = KSPSetFromOptions(ksp); CHKERRQ(ierr); ierr = KSPSolve(ksp, b_vec, x0_vec); CHKERRQ(ierr); ierr = KSPDestroy(&ksp); CHKERRQ(ierr); ierr = MatDestroy(&A_mat); CHKERRQ(ierr); ierr = VecDestroy(&b_vec); CHKERRQ(ierr); ierr = VecDestroy(&x0_vec); CHKERRQ(ierr); ierr = PetscFinalize(); CHKERRQ(ierr); return ierr; } ------=_NextPart_000_00A0_01CE2199.D4B7EEE0 Content-Type: text/plain; name="result.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="result.txt" [jhurani@mg1 krylov]$ bash -x run.sh + ksp=bcgs + mpiexec -n 1 ./test_ksp.exe -ksp_monitor -ksp_type bcgs 0 KSP Residual norm 1.952299312119e+02 1 KSP Residual norm 4.924902191069e+01 2 KSP Residual norm 1.393838666448e+01 3 KSP Residual norm 4.765851663701e+00 4 KSP Residual norm 1.859652065579e+00 5 KSP Residual norm 8.498696828719e-01 6 KSP Residual norm 2.567018670294e+00 7 KSP Residual norm 9.628917893930e-02 8 KSP Residual norm 3.959095364957e-02 9 KSP Residual norm 6.603905987516e-02 10 KSP Residual norm 6.548400558250e-03 + mpiexec -n 4 ./test_ksp.exe -ksp_monitor -ksp_type bcgs 0 KSP Residual norm 1.952299312119e+02 1 KSP Residual norm 4.924902191069e+01 2 KSP Residual norm 1.393838666448e+01 3 KSP Residual norm 4.765851663701e+00 4 KSP Residual norm 1.859652065579e+00 5 KSP Residual norm 8.498696828719e-01 6 KSP Residual norm 2.567018670294e+00 7 KSP Residual norm 9.628917893929e-02 8 KSP Residual norm 3.959095364957e-02 9 KSP Residual norm 6.603905987518e-02 10 KSP Residual norm 6.548400558251e-03 + mpiexec -n 1 ./test_ksp.exe -ksp_monitor -ksp_type bcgs -vec_type cusp -mat_type aijcusp 0 KSP Residual norm 1.952299312119e+02 1 KSP Residual norm 4.924902191069e+01 2 KSP Residual norm 1.393838666448e+01 3 KSP Residual norm 4.765851663701e+00 4 KSP Residual norm 1.859652065579e+00 5 KSP Residual norm 8.498696828719e-01 6 KSP Residual norm 2.567018670294e+00 7 KSP Residual norm 9.628917893930e-02 8 KSP Residual norm 3.959095364957e-02 9 KSP Residual norm 6.603905987515e-02 10 KSP Residual norm 6.548400558250e-03 + mpiexec -n 4 ./test_ksp.exe -ksp_monitor -ksp_type bcgs -vec_type cusp -mat_type mpiaijcusp 0 KSP Residual norm 1.952299312119e+02 1 KSP Residual norm 4.924902191069e+01 2 KSP Residual norm 3.681298932599e+01 3 KSP Residual norm 2.799054681516e+01 4 KSP Residual norm 1.599021555910e+01 5 KSP Residual norm 1.009936483902e+01 6 KSP Residual norm 6.214971385795e+00 7 KSP Residual norm 3.412355095382e+00 8 KSP Residual norm 2.049245379370e+00 9 KSP Residual norm 1.254036741543e+00 10 KSP Residual norm 7.845146110644e-01 ------=_NextPart_000_00A0_01CE2199.D4B7EEE0-- ]