From afrah.nacib at gmail.com Fri Sep 1 04:48:38 2017 From: afrah.nacib at gmail.com (Afrah Najib) Date: Fri, 1 Sep 2017 12:48:38 +0300 Subject: [petsc-users] PETSc in PDSLin Message-ID: I want to use use PETSc in a hybrid solver(direct- iterative) called PDSLin 2.0.0.[http://portal.nersc.gov/project/sparse/pdslin/] the version of PETSc used is petsc3.7.6. When running the examples, I got the following errors: :~/pdslin_2.0.0/examples$ make all /usr/share/mpich-install/bin/mpicc -I/usr/share/mpich-install/include -I/home/afrah/pdslin_2.0.0/include -I/home/afrah/SuperLU_DIST_5.1.2/SRC -I/usr/include -I/usr/local/include -I/home/afrah/PETSc-library2/petsc-3.7.6/source/include -I/home/afrah/PETSc-library2/petsc-3.7.6-complex/source/include -I/home/afrah/scotch-6.0.4/include -c dtest.c -o dtest.o -g -O0 -fopenmp -std=c99 -DWITH_PETSC dtest.c: In function ?main?: dtest.c:53:20: warning: implicit declaration of function ?pdslin_print_input? [-Wimplicit-function-declaration] if ( !proc_id ) pdslin_print_input(&input); ^ dtest.c:127:39: warning: implicit declaration of function ?pdslin_print_stat? [-Wimplicit-function-declaration] if( input.verbose == PDSLin_VALL ) pdslin_print_stat( &sta ^ /usr/share/mpich-install/bin/mpicc -fopenmp dtest.o -L/usr/share/mpich-install/lib/libmpi.a /home/afrah/pdslin_2.0.0/lib/libpdslin.a -lm -llapack -lblas -Wl,-rpath,/home/afrah/SuperLU_DIST_5.1.2/lib -L/home/afrah/SuperLU_DIST_5.1.2/lib -lsuperlu_dist -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lmetis -Wl,-rpath,/usr/share/parmetis/lib -L/usr/share/parmetis/lib -lparmetis -L/home/afrah/PETSc-library2/ petsc-3.7.6/source/lib/libpetsc.so -o dtest -g -O0 -fopenmp -std=c99 -DWITH_PETSC /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function `PetscMPITypeSizeComm': /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:328: undefined reference to `PetscError' /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:329: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function `dcomp_schur': /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:92: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:92: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:121: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:121: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:129: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ home/afrah/pdslin_2.0.0/src/dpdslin_core.c:129: more undefined references to `petsc_gather_ct' follow /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function `dcomp_sol': /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:486: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:486: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:495: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:495: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:496: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ home/afrah/pdslin_2.0.0/src/dpdslin_core.c:496: more undefined references to `petsc_allreduce_ct' follow /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function `dcomp_sol': /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:614: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:614: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:625: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:625: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:625: undefined reference to `petsc_send_len' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:659: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:659: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:793: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:793: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:793: undefined reference to `petsc_send_len' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:795: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:795: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:795: undefined reference to `petsc_send_len' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:803: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:803: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:812: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:812: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:862: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:862: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:891: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ home/afrah/pdslin_2.0.0/src/dpdslin_core.c:891: more undefined references to `petsc_allreduce_ct' follow /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function `dallgather_ssol': /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1094: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1094: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function `dsparsify_schur': /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1331: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1331: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1357: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1357: more undefined references to `petsc_gather_ct' follow /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function `dsparsify_schur': /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1789: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1789: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1861: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1861: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function `PetscMPITypeSizeComm': /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:328: undefined reference to `PetscError' /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:329: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function `dpetsc_init': /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:60: undefined reference to `PetscInitialized' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:62: undefined reference to `PetscInitialize' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:63: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function `dpetsc_clean': /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:71: undefined reference to `PetscFinalize' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:72: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function `dsolve_schur_petsc': /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:164: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:164: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:164: undefined reference to `petsc_send_len' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:229: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:229: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:229: undefined reference to `petsc_send_len' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:251: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:251: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:251: undefined reference to `petsc_send_len' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:292: undefined reference to `VecCreate' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:292: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:293: undefined reference to `VecSetSizes' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:293: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:294: undefined reference to `VecSetFromOptions' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:294: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:295: undefined reference to `VecDuplicate' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:295: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:296: undefined reference to `VecDuplicate' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:296: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:305: undefined reference to `MatCreate' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:305: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:306: undefined reference to `MatSetSizes' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:306: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:308: undefined reference to `MatSetFromOptions' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:308: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:311: undefined reference to `MatCreateShell' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:312: undefined reference to `MatShellSetOperation' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:325: undefined reference to `MatCreate' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:325: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:326: undefined reference to `MatSetSizes' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:326: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:328: undefined reference to `MatSetFromOptions' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:328: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:346: undefined reference to `MatMPIAIJSetPreallocation' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:347: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:348: undefined reference to `MatSeqAIJSetPreallocation' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:349: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:359: undefined reference to `MatMPIAIJSetPreallocation' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:360: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:361: undefined reference to `MatSeqAIJSetPreallocation' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:362: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:377: undefined reference to `MatSetValues' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:378: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:383: undefined reference to `MatAssemblyBegin' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:383: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:384: undefined reference to `MatAssemblyEnd' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:384: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:386: undefined reference to `MatAssemblyBegin' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:386: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:387: undefined reference to `MatAssemblyEnd' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:387: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:396: undefined reference to `VecSetValues' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:397: undefined reference to `VecAssemblyBegin' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:397: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:398: undefined reference to `VecAssemblyEnd' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:398: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:409: undefined reference to `KSPCreate' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:410: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:421: undefined reference to `KSPSetOperators' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:427: undefined reference to `KSPSetOperators' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:430: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:439: undefined reference to `KSPSetFromOptions' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:439: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:440: undefined reference to `KSPSetTolerances' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:441: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:450: undefined reference to `KSPSetType' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:450: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:451: undefined reference to `KSPGMRESSetRestart' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:451: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:463: undefined reference to `KSPSetType' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:463: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:468: undefined reference to `KSPGMRESSetRestart' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:468: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:470: undefined reference to ` KSPGMRESModifiedGramSchmidtOrthogonalization' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:470: undefined reference to `KSPGMRESSetOrthogonalization' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:471: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:473: undefined reference to ` KSPGMRESClassicalGramSchmidtOrthogonalization' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:473: undefined reference to `KSPGMRESSetOrthogonalization' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:474: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:481: undefined reference to `KSPSetType' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:481: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:483: undefined reference to `KSPSetPCSide' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:483: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:488: undefined reference to `KSPSetType' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:488: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:495: undefined reference to `KSPGetPC' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:495: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:502: undefined reference to `PCSetType' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:502: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:503: undefined reference to `PCFactorSetLevels' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:503: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:510: undefined reference to `PCSetType' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:510: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:520: undefined reference to `PCFactorSetDropTolerance' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:522: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:527: undefined reference to `PCSetType' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:527: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:528: undefined reference to `PCASMSetOverlap' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:528: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:535: undefined reference to `KSPSetUp' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:535: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:536: undefined reference to `PCASMGetSubKSP' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:536: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:538: undefined reference to `KSPGetPC' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:538: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:539: undefined reference to `KSPSetType' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:539: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:540: undefined reference to `PCSetType' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:540: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:544: undefined reference to `PCSetType' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:572: undefined reference to `PCShellSetApply' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:573: undefined reference to `PCShellSetContext' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:589: undefined reference to `KSPSetInitialGuessNonzero' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:590: undefined reference to `VecSetValues' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:592: undefined reference to `VecAssemblyBegin' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:592: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:593: undefined reference to `VecAssemblyEnd' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:593: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:595: undefined reference to `KSPSetInitialGuessNonzero' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:601: undefined reference to `KSPSolve' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:602: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:613: undefined reference to `KSPGetConvergedReason' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:617: undefined reference to `KSPGetIterationNumber' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:628: undefined reference to `KSPGetResidualNorm' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:635: undefined reference to `PETSC_VIEWER_STDOUT_' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:635: undefined reference to `KSPView' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:636: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:664: undefined reference to `VecGetValues' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:677: undefined reference to `VecDestroy' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:677: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:678: undefined reference to `VecDestroy' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:678: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:679: undefined reference to `VecDestroy' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:679: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:680: undefined reference to `MatDestroy' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:680: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:681: undefined reference to `KSPDestroy' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:681: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:707: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:707: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:708: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:708: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function `dprecond_slu': /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:781: undefined reference to `PCShellGetContext' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:847: undefined reference to `VecGetValues' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:848: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:858: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:858: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:876: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:876: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:881: undefined reference to `VecSetValues' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:883: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:884: undefined reference to `VecAssemblyBegin' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:884: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:885: undefined reference to `VecAssemblyEnd' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:885: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function `dmatop': /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1025: undefined reference to `MatShellGetContext' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1064: undefined reference to `VecGetValues' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1066: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1093: undefined reference to `VecSetValues' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1095: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1096: undefined reference to `VecAssemblyBegin' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1096: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1097: undefined reference to `VecAssemblyEnd' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1097: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function `dmat_schur': /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1161: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1161: undefined reference to `petsc_gather_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1379: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1379: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1379: undefined reference to `petsc_send_len' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1399: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1399: undefined reference to `petsc_allreduce_ct' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function `dpetsc_iluksolver': /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1486: undefined reference to `PETSC_COMM_WORLD' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1486: undefined reference to `VecCreate' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1486: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1487: undefined reference to `VecSetSizes' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1487: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1488: undefined reference to `VecSetFromOptions' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1488: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1489: undefined reference to `VecDuplicate' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1489: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1490: undefined reference to `VecDuplicate' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1490: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1498: undefined reference to `PETSC_COMM_WORLD' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1498: undefined reference to `MatCreate' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1498: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1499: undefined reference to `MatSetSizes' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1499: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1501: undefined reference to `MatSetFromOptions' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1501: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1515: undefined reference to `MatSetValues' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1520: undefined reference to `MatAssemblyBegin' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1520: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1521: undefined reference to `MatAssemblyEnd' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1521: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1529: undefined reference to `VecSetValues' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1530: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1531: undefined reference to `VecAssemblyBegin' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1531: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1532: undefined reference to `VecAssemblyEnd' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1532: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1538: undefined reference to `PETSC_COMM_WORLD' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1538: undefined reference to `KSPCreate' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1539: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1548: undefined reference to `KSPSetOperators' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1550: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1557: undefined reference to `KSPSetType' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1557: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1558: undefined reference to `KSPSetTolerances' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1559: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1561: undefined reference to `KSPSetResidualHistory' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1567: undefined reference to `KSPGetPC' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1567: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1568: undefined reference to `PCSetType' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1568: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1572: undefined reference to `PCFactorSetLevels' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1572: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1574: undefined reference to `KSPSetFromOptions' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1574: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1580: undefined reference to `KSPSolve' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1581: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1586: undefined reference to `KSPGetConvergedReason' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1593: undefined reference to `KSPGetIterationNumber' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1594: undefined reference to `KSPGetResidualNorm' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1610: undefined reference to `VecAssemblyBegin' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1610: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1611: undefined reference to `VecAssemblyEnd' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1611: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1615: undefined reference to `VecGetValues' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1622: undefined reference to `PETSC_COMM_WORLD' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1622: undefined reference to `PETSC_VIEWER_STDOUT_' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1622: undefined reference to `KSPView' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1623: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1628: undefined reference to `VecDestroy' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1628: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1629: undefined reference to `VecDestroy' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1629: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1630: undefined reference to `VecDestroy' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1630: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1631: undefined reference to `MatDestroy' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1631: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1632: undefined reference to `KSPDestroy' /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1632: undefined reference to `PetscError' /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_fgmres.o): In function `PetscMPITypeSizeComm': /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:328: undefined reference to `PetscError' /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:329: undefined reference to `PetscError' -------------- next part -------------- An HTML attachment was scrubbed... URL: From jakub.kruzik at vsb.cz Fri Sep 1 06:40:04 2017 From: jakub.kruzik at vsb.cz (Jakub Kruzik) Date: Fri, 1 Sep 2017 13:40:04 +0200 Subject: [petsc-users] Fwd: direct solvers on KNL In-Reply-To: References: Message-ID: <74eb23c0-d5f8-bb22-6a60-547b80c2640b@vsb.cz> Hi, I am looking at a single node performance of MUMPS and SuperLU on KNL 7230 (on Theta). I am using KSP example ex2 (http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html) with m X n = 2880 x 2880. KNL runs in cache and quad modes. Times in seconds for 24 cores: mumps: ? 279 superlu: ? 326 cg: ??????????? 116 Times in seconds for 64 cores: mumps: ? 316 superlu: ? 410 cg : ?????????? 49 The performance for 24 cores is OK - both direct solvers are roughly 3.5 times slower than 2x E5-2680v3. (According to people from Intel, the single core performance of KNL is about 3-4 times lower than that of E5-2680v3). However, strong scalability is really bad. I am using cray-petsc/3.7.6.0 module. I tried my own PETSc compilation with MKL and MUMPS/SuperLU installed by PETSc configure but the results are similar. Please find attached Theta submission script and logs for KNL and Haswells. Why the performance of direct solvers on a full node is so bad? Best, Jakub -------------- next part -------------- A non-text attachment was scrubbed... Name: batch.sub Type: text/x-microdvd Size: 2517 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: knl.log Type: text/x-log Size: 64418 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: haswell.log Type: text/x-log Size: 53537 bytes Desc: not available URL: From knepley at gmail.com Fri Sep 1 08:15:51 2017 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 1 Sep 2017 09:15:51 -0400 Subject: [petsc-users] Fwd: direct solvers on KNL In-Reply-To: <74eb23c0-d5f8-bb22-6a60-547b80c2640b@vsb.cz> References: <74eb23c0-d5f8-bb22-6a60-547b80c2640b@vsb.cz> Message-ID: On Fri, Sep 1, 2017 at 7:40 AM, Jakub Kruzik wrote: > Hi, > > I am looking at a single node performance of MUMPS and SuperLU on KNL 7230 > (on Theta). I am using KSP example ex2 (http://www.mcs.anl.gov/petsc/ > petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html) with m X n = > 2880 x 2880. KNL runs in cache and quad modes. > > Times in seconds for 24 cores: > mumps: 279 > superlu: 326 > cg: 116 > > Times in seconds for 64 cores: > mumps: 316 > superlu: 410 > cg : 49 > > The performance for 24 cores is OK - both direct solvers are roughly 3.5 > times slower than 2x E5-2680v3. (According to people from Intel, the single > core performance of KNL is about 3-4 times lower than that of E5-2680v3). > However, strong scalability is really bad. > > I am using cray-petsc/3.7.6.0 module. I tried my own PETSc compilation > with MKL and MUMPS/SuperLU installed by PETSc configure but the results are > similar. > > Please find attached Theta submission script and logs for KNL and Haswells. > > Why the performance of direct solvers on a full node is so bad? > Admittedly it was for different computations, but we saw strong scaling degradation after 32 cores of KNL in https://arxiv.org/abs/1705.09907, and we also saw strong scaling tail off as the problem size got this small in https://arxiv.org/abs/1705.03625 Thanks, Matt > Best, > Jakub > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Sep 1 08:19:14 2017 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 1 Sep 2017 09:19:14 -0400 Subject: [petsc-users] PETSc in PDSLin In-Reply-To: References: Message-ID: On Fri, Sep 1, 2017 at 5:48 AM, Afrah Najib wrote: > I want to use use PETSc in a hybrid solver(direct- iterative) called > PDSLin 2.0.0.[http://portal.nersc.gov/project/sparse/pdslin/] > > the version of PETSc used is petsc3.7.6. When running the examples, I got > the following errors: > > > > :~/pdslin_2.0.0/examples$ make all > /usr/share/mpich-install/bin/mpicc -I/usr/share/mpich-install/include > -I/home/afrah/pdslin_2.0.0/include -I/home/afrah/SuperLU_DIST_5.1.2/SRC > -I/usr/include -I/usr/local/include -I/home/afrah/PETSc-library2/petsc-3.7.6/source/include > -I/home/afrah/PETSc-library2/petsc-3.7.6-complex/source/include > -I/home/afrah/scotch-6.0.4/include -c dtest.c -o dtest.o -g -O0 -fopenmp > -std=c99 -DWITH_PETSC > dtest.c: In function ?main?: > dtest.c:53:20: warning: implicit declaration of function > ?pdslin_print_input? [-Wimplicit-function-declaration] > if ( !proc_id ) pdslin_print_input(&input); > ^ > dtest.c:127:39: warning: implicit declaration of function > ?pdslin_print_stat? [-Wimplicit-function-declaration] > if( input.verbose == PDSLin_VALL ) pdslin_print_stat( &sta > Note that below, you do not have -lpetsc. I would recommend using the PETSc makefiles, but you can also run 'make getlinklibs' to the correct link line. Thanks, Matt > ^ > /usr/share/mpich-install/bin/mpicc -fopenmp dtest.o > -L/usr/share/mpich-install/lib/libmpi.a /home/afrah/pdslin_2.0.0/lib/libpdslin.a > -lm -llapack -lblas -Wl,-rpath,/home/afrah/SuperLU_DIST_5.1.2/lib > -L/home/afrah/SuperLU_DIST_5.1.2/lib -lsuperlu_dist > -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lmetis > -Wl,-rpath,/usr/share/parmetis/lib -L/usr/share/parmetis/lib -lparmetis > -L/home/afrah/PETSc-library2/petsc-3.7.6/source/lib/libpetsc.so -o dtest > -g -O0 -fopenmp -std=c99 -DWITH_PETSC > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > `PetscMPITypeSizeComm': > /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:328: > undefined reference to `PetscError' > /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:329: > undefined reference to `PetscError' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > `dcomp_schur': > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:92: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:92: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:121: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:121: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:129: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ho > me/afrah/pdslin_2.0.0/src/dpdslin_core.c:129: more undefined references > to `petsc_gather_ct' follow > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > `dcomp_sol': > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:486: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:486: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:495: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:495: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:496: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ho > me/afrah/pdslin_2.0.0/src/dpdslin_core.c:496: more undefined references > to `petsc_allreduce_ct' follow > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > `dcomp_sol': > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:614: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:614: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:625: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:625: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:625: undefined reference to > `petsc_send_len' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:659: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:659: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:793: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:793: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:793: undefined reference to > `petsc_send_len' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:795: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:795: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:795: undefined reference to > `petsc_send_len' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:803: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:803: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:812: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:812: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:862: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:862: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:891: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ho > me/afrah/pdslin_2.0.0/src/dpdslin_core.c:891: more undefined references > to `petsc_allreduce_ct' follow > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > `dallgather_ssol': > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1094: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1094: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > `dsparsify_schur': > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1331: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1331: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1357: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ho > me/afrah/pdslin_2.0.0/src/dpdslin_core.c:1357: more undefined references > to `petsc_gather_ct' follow > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > `dsparsify_schur': > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1789: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1789: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1861: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1861: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > `PetscMPITypeSizeComm': > /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:328: > undefined reference to `PetscError' > /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:329: > undefined reference to `PetscError' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > `dpetsc_init': > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:60: undefined reference to > `PetscInitialized' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:62: undefined reference to > `PetscInitialize' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:63: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > `dpetsc_clean': > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:71: undefined reference to > `PetscFinalize' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:72: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > `dsolve_schur_petsc': > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:164: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:164: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:164: undefined reference to > `petsc_send_len' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:229: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:229: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:229: undefined reference to > `petsc_send_len' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:251: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:251: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:251: undefined reference to > `petsc_send_len' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:292: undefined reference to > `VecCreate' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:292: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:293: undefined reference to > `VecSetSizes' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:293: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:294: undefined reference to > `VecSetFromOptions' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:294: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:295: undefined reference to > `VecDuplicate' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:295: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:296: undefined reference to > `VecDuplicate' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:296: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:305: undefined reference to > `MatCreate' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:305: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:306: undefined reference to > `MatSetSizes' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:306: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:308: undefined reference to > `MatSetFromOptions' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:308: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:311: undefined reference to > `MatCreateShell' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:312: undefined reference to > `MatShellSetOperation' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:325: undefined reference to > `MatCreate' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:325: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:326: undefined reference to > `MatSetSizes' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:326: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:328: undefined reference to > `MatSetFromOptions' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:328: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:346: undefined reference to > `MatMPIAIJSetPreallocation' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:347: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:348: undefined reference to > `MatSeqAIJSetPreallocation' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:349: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:359: undefined reference to > `MatMPIAIJSetPreallocation' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:360: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:361: undefined reference to > `MatSeqAIJSetPreallocation' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:362: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:377: undefined reference to > `MatSetValues' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:378: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:383: undefined reference to > `MatAssemblyBegin' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:383: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:384: undefined reference to > `MatAssemblyEnd' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:384: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:386: undefined reference to > `MatAssemblyBegin' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:386: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:387: undefined reference to > `MatAssemblyEnd' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:387: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:396: undefined reference to > `VecSetValues' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:397: undefined reference to > `VecAssemblyBegin' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:397: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:398: undefined reference to > `VecAssemblyEnd' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:398: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:409: undefined reference to > `KSPCreate' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:410: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:421: undefined reference to > `KSPSetOperators' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:427: undefined reference to > `KSPSetOperators' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:430: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:439: undefined reference to > `KSPSetFromOptions' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:439: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:440: undefined reference to > `KSPSetTolerances' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:441: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:450: undefined reference to > `KSPSetType' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:450: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:451: undefined reference to > `KSPGMRESSetRestart' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:451: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:463: undefined reference to > `KSPSetType' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:463: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:468: undefined reference to > `KSPGMRESSetRestart' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:468: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:470: undefined reference to > `KSPGMRESModifiedGramSchmidtOrthogonalization' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:470: undefined reference to > `KSPGMRESSetOrthogonalization' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:471: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:473: undefined reference to > `KSPGMRESClassicalGramSchmidtOrthogonalization' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:473: undefined reference to > `KSPGMRESSetOrthogonalization' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:474: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:481: undefined reference to > `KSPSetType' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:481: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:483: undefined reference to > `KSPSetPCSide' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:483: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:488: undefined reference to > `KSPSetType' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:488: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:495: undefined reference to > `KSPGetPC' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:495: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:502: undefined reference to > `PCSetType' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:502: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:503: undefined reference to > `PCFactorSetLevels' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:503: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:510: undefined reference to > `PCSetType' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:510: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:520: undefined reference to > `PCFactorSetDropTolerance' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:522: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:527: undefined reference to > `PCSetType' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:527: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:528: undefined reference to > `PCASMSetOverlap' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:528: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:535: undefined reference to > `KSPSetUp' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:535: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:536: undefined reference to > `PCASMGetSubKSP' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:536: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:538: undefined reference to > `KSPGetPC' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:538: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:539: undefined reference to > `KSPSetType' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:539: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:540: undefined reference to > `PCSetType' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:540: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:544: undefined reference to > `PCSetType' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:572: undefined reference to > `PCShellSetApply' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:573: undefined reference to > `PCShellSetContext' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:589: undefined reference to > `KSPSetInitialGuessNonzero' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:590: undefined reference to > `VecSetValues' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:592: undefined reference to > `VecAssemblyBegin' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:592: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:593: undefined reference to > `VecAssemblyEnd' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:593: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:595: undefined reference to > `KSPSetInitialGuessNonzero' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:601: undefined reference to > `KSPSolve' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:602: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:613: undefined reference to > `KSPGetConvergedReason' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:617: undefined reference to > `KSPGetIterationNumber' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:628: undefined reference to > `KSPGetResidualNorm' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:635: undefined reference to > `PETSC_VIEWER_STDOUT_' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:635: undefined reference to > `KSPView' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:636: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:664: undefined reference to > `VecGetValues' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:677: undefined reference to > `VecDestroy' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:677: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:678: undefined reference to > `VecDestroy' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:678: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:679: undefined reference to > `VecDestroy' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:679: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:680: undefined reference to > `MatDestroy' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:680: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:681: undefined reference to > `KSPDestroy' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:681: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:707: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:707: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:708: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:708: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > `dprecond_slu': > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:781: undefined reference to > `PCShellGetContext' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:847: undefined reference to > `VecGetValues' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:848: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:858: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:858: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:876: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:876: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:881: undefined reference to > `VecSetValues' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:883: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:884: undefined reference to > `VecAssemblyBegin' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:884: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:885: undefined reference to > `VecAssemblyEnd' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:885: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > `dmatop': > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1025: undefined reference to > `MatShellGetContext' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1064: undefined reference to > `VecGetValues' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1066: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1093: undefined reference to > `VecSetValues' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1095: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1096: undefined reference to > `VecAssemblyBegin' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1096: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1097: undefined reference to > `VecAssemblyEnd' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1097: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > `dmat_schur': > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1161: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1161: undefined reference to > `petsc_gather_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1379: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1379: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1379: undefined reference to > `petsc_send_len' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1399: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1399: undefined reference to > `petsc_allreduce_ct' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > `dpetsc_iluksolver': > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1486: undefined reference to > `PETSC_COMM_WORLD' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1486: undefined reference to > `VecCreate' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1486: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1487: undefined reference to > `VecSetSizes' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1487: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1488: undefined reference to > `VecSetFromOptions' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1488: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1489: undefined reference to > `VecDuplicate' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1489: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1490: undefined reference to > `VecDuplicate' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1490: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1498: undefined reference to > `PETSC_COMM_WORLD' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1498: undefined reference to > `MatCreate' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1498: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1499: undefined reference to > `MatSetSizes' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1499: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1501: undefined reference to > `MatSetFromOptions' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1501: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1515: undefined reference to > `MatSetValues' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1520: undefined reference to > `MatAssemblyBegin' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1520: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1521: undefined reference to > `MatAssemblyEnd' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1521: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1529: undefined reference to > `VecSetValues' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1530: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1531: undefined reference to > `VecAssemblyBegin' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1531: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1532: undefined reference to > `VecAssemblyEnd' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1532: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1538: undefined reference to > `PETSC_COMM_WORLD' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1538: undefined reference to > `KSPCreate' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1539: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1548: undefined reference to > `KSPSetOperators' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1550: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1557: undefined reference to > `KSPSetType' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1557: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1558: undefined reference to > `KSPSetTolerances' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1559: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1561: undefined reference to > `KSPSetResidualHistory' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1567: undefined reference to > `KSPGetPC' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1567: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1568: undefined reference to > `PCSetType' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1568: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1572: undefined reference to > `PCFactorSetLevels' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1572: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1574: undefined reference to > `KSPSetFromOptions' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1574: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1580: undefined reference to > `KSPSolve' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1581: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1586: undefined reference to > `KSPGetConvergedReason' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1593: undefined reference to > `KSPGetIterationNumber' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1594: undefined reference to > `KSPGetResidualNorm' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1610: undefined reference to > `VecAssemblyBegin' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1610: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1611: undefined reference to > `VecAssemblyEnd' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1611: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1615: undefined reference to > `VecGetValues' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1622: undefined reference to > `PETSC_COMM_WORLD' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1622: undefined reference to > `PETSC_VIEWER_STDOUT_' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1622: undefined reference to > `KSPView' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1623: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1628: undefined reference to > `VecDestroy' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1628: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1629: undefined reference to > `VecDestroy' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1629: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1630: undefined reference to > `VecDestroy' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1630: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1631: undefined reference to > `MatDestroy' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1631: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1632: undefined reference to > `KSPDestroy' > /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1632: undefined reference to > `PetscError' > /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_fgmres.o): In function > `PetscMPITypeSizeComm': > /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:328: > undefined reference to `PetscError' > /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:329: > undefined reference to `PetscError' > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From afrah.nacib at gmail.com Fri Sep 1 09:11:43 2017 From: afrah.nacib at gmail.com (Afrah Najib) Date: Fri, 1 Sep 2017 17:11:43 +0300 Subject: [petsc-users] PETSc in PDSLin In-Reply-To: References: Message-ID: I am compiling PDSLin examples. I installed PETSc with the examples with no errors. When I use -Wl,-rpath option , I got this error: :~/pdslin_2.0.0/examples$ make all /usr/share/mpich-install/bin/mpicc -I/usr/share/mpich-install/include -I/home/afrah/pdslin_2.0.0/include -I/home/afrah/SuperLU_DIST_5.1.2/SRC -I/usr/include -I/usr/local/include -I/home/afrah/PETSc-library2/petsc-3.7.6/source/include -I/home/afrah/PETSc-library2/petsc-3.7.6-complex/source/include -I/home/afrah/scotch-6.0.4/include -c dtest.c -o dtest.o -g -O0 -fopenmp -std=c99 -DWITH_PETSC dtest.c: In function ?main?: dtest.c:53:20: warning: implicit declaration of function ?pdslin_print_input? [-Wimplicit-function-declaration] if ( !proc_id ) pdslin_print_input(&input); ^ dtest.c:127:39: warning: implicit declaration of function ?pdslin_print_stat? [-Wimplicit-function-declaration] if( input.verbose == PDSLin_VALL ) pdslin_print_stat( &stat, matrix.pdslin_comm ); ^ /usr/share/mpich-install/bin/mpicc -fopenmp dtest.o -L/usr/share/mpich-install/lib/libmpi.a /home/afrah/pdslin_2.0.0/lib/libpdslin.a -lm -llapack -lblas -Wl,-rpath,/home/afrah/SuperLU_DIST_5.1.2/lib -L/home/afrah/SuperLU_DIST_5.1.2/lib -lsuperlu_dist -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lmetis -Wl,-rpath,/usr/share/parmetis/lib -L/usr/share/parmetis/lib -lparmetis -Wl,-rpath,/home/afrah/PETSc-library2/petsc-3.7.6/source/lib -L/home/afrah/PETSc-library2/petsc-3.7.6/source/lib -llibpetsc -o dtest -g -O0 -fopenmp -std=c99 -DWITH_PETSC /usr/bin/ld: cannot find -llibpetsc collect2: error: ld returned 1 exit status makefile:10: recipe for target 'dtest' failed make: *** [dtest] Error 1 On 1 September 2017 at 16:19, Matthew Knepley wrote: > On Fri, Sep 1, 2017 at 5:48 AM, Afrah Najib wrote: > >> I want to use use PETSc in a hybrid solver(direct- iterative) called >> PDSLin 2.0.0.[http://portal.nersc.gov/project/sparse/pdslin/] >> >> the version of PETSc used is petsc3.7.6. When running the examples, I got >> the following errors: >> >> >> >> :~/pdslin_2.0.0/examples$ make all >> /usr/share/mpich-install/bin/mpicc -I/usr/share/mpich-install/include >> -I/home/afrah/pdslin_2.0.0/include -I/home/afrah/SuperLU_DIST_5.1.2/SRC >> -I/usr/include -I/usr/local/include -I/home/afrah/PETSc-library2/petsc-3.7.6/source/include >> -I/home/afrah/PETSc-library2/petsc-3.7.6-complex/source/include >> -I/home/afrah/scotch-6.0.4/include -c dtest.c -o dtest.o -g -O0 -fopenmp >> -std=c99 -DWITH_PETSC >> dtest.c: In function ?main?: >> dtest.c:53:20: warning: implicit declaration of function >> ?pdslin_print_input? [-Wimplicit-function-declaration] >> if ( !proc_id ) pdslin_print_input(&input); >> ^ >> dtest.c:127:39: warning: implicit declaration of function >> ?pdslin_print_stat? [-Wimplicit-function-declaration] >> if( input.verbose == PDSLin_VALL ) pdslin_print_stat( &sta >> > > Note that below, you do not have -lpetsc. I would recommend using the > PETSc makefiles, but you can also run 'make getlinklibs' to the correct > link line. > > Thanks, > > Matt > > >> ^ >> /usr/share/mpich-install/bin/mpicc -fopenmp dtest.o >> -L/usr/share/mpich-install/lib/libmpi.a /home/afrah/pdslin_2.0.0/lib/libpdslin.a >> -lm -llapack -lblas -Wl,-rpath,/home/afrah/SuperLU_DIST_5.1.2/lib >> -L/home/afrah/SuperLU_DIST_5.1.2/lib -lsuperlu_dist >> -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lmetis >> -Wl,-rpath,/usr/share/parmetis/lib -L/usr/share/parmetis/lib -lparmetis >> -L/home/afrah/PETSc-library2/petsc-3.7.6/source/lib/libpetsc.so -o >> dtest -g -O0 -fopenmp -std=c99 -DWITH_PETSC >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function >> `PetscMPITypeSizeComm': >> /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:328: >> undefined reference to `PetscError' >> /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:329: >> undefined reference to `PetscError' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function >> `dcomp_schur': >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:92: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:92: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:121: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:121: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:129: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ho >> me/afrah/pdslin_2.0.0/src/dpdslin_core.c:129: more undefined references >> to `petsc_gather_ct' follow >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function >> `dcomp_sol': >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:486: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:486: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:495: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:495: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:496: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ho >> me/afrah/pdslin_2.0.0/src/dpdslin_core.c:496: more undefined references >> to `petsc_allreduce_ct' follow >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function >> `dcomp_sol': >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:614: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:614: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:625: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:625: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:625: undefined reference to >> `petsc_send_len' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:659: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:659: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:793: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:793: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:793: undefined reference to >> `petsc_send_len' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:795: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:795: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:795: undefined reference to >> `petsc_send_len' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:803: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:803: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:812: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:812: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:862: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:862: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:891: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ho >> me/afrah/pdslin_2.0.0/src/dpdslin_core.c:891: more undefined references >> to `petsc_allreduce_ct' follow >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function >> `dallgather_ssol': >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1094: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1094: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function >> `dsparsify_schur': >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1331: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1331: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1357: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ho >> me/afrah/pdslin_2.0.0/src/dpdslin_core.c:1357: more undefined references >> to `petsc_gather_ct' follow >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function >> `dsparsify_schur': >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1789: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1789: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1861: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1861: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function >> `PetscMPITypeSizeComm': >> /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:328: >> undefined reference to `PetscError' >> /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:329: >> undefined reference to `PetscError' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function >> `dpetsc_init': >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:60: undefined reference to >> `PetscInitialized' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:62: undefined reference to >> `PetscInitialize' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:63: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function >> `dpetsc_clean': >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:71: undefined reference to >> `PetscFinalize' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:72: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function >> `dsolve_schur_petsc': >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:164: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:164: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:164: undefined reference to >> `petsc_send_len' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:229: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:229: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:229: undefined reference to >> `petsc_send_len' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:251: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:251: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:251: undefined reference to >> `petsc_send_len' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:292: undefined reference to >> `VecCreate' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:292: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:293: undefined reference to >> `VecSetSizes' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:293: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:294: undefined reference to >> `VecSetFromOptions' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:294: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:295: undefined reference to >> `VecDuplicate' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:295: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:296: undefined reference to >> `VecDuplicate' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:296: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:305: undefined reference to >> `MatCreate' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:305: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:306: undefined reference to >> `MatSetSizes' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:306: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:308: undefined reference to >> `MatSetFromOptions' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:308: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:311: undefined reference to >> `MatCreateShell' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:312: undefined reference to >> `MatShellSetOperation' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:325: undefined reference to >> `MatCreate' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:325: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:326: undefined reference to >> `MatSetSizes' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:326: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:328: undefined reference to >> `MatSetFromOptions' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:328: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:346: undefined reference to >> `MatMPIAIJSetPreallocation' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:347: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:348: undefined reference to >> `MatSeqAIJSetPreallocation' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:349: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:359: undefined reference to >> `MatMPIAIJSetPreallocation' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:360: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:361: undefined reference to >> `MatSeqAIJSetPreallocation' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:362: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:377: undefined reference to >> `MatSetValues' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:378: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:383: undefined reference to >> `MatAssemblyBegin' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:383: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:384: undefined reference to >> `MatAssemblyEnd' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:384: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:386: undefined reference to >> `MatAssemblyBegin' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:386: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:387: undefined reference to >> `MatAssemblyEnd' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:387: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:396: undefined reference to >> `VecSetValues' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:397: undefined reference to >> `VecAssemblyBegin' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:397: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:398: undefined reference to >> `VecAssemblyEnd' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:398: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:409: undefined reference to >> `KSPCreate' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:410: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:421: undefined reference to >> `KSPSetOperators' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:427: undefined reference to >> `KSPSetOperators' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:430: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:439: undefined reference to >> `KSPSetFromOptions' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:439: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:440: undefined reference to >> `KSPSetTolerances' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:441: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:450: undefined reference to >> `KSPSetType' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:450: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:451: undefined reference to >> `KSPGMRESSetRestart' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:451: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:463: undefined reference to >> `KSPSetType' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:463: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:468: undefined reference to >> `KSPGMRESSetRestart' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:468: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:470: undefined reference to >> `KSPGMRESModifiedGramSchmidtOrthogonalization' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:470: undefined reference to >> `KSPGMRESSetOrthogonalization' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:471: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:473: undefined reference to >> `KSPGMRESClassicalGramSchmidtOrthogonalization' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:473: undefined reference to >> `KSPGMRESSetOrthogonalization' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:474: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:481: undefined reference to >> `KSPSetType' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:481: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:483: undefined reference to >> `KSPSetPCSide' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:483: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:488: undefined reference to >> `KSPSetType' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:488: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:495: undefined reference to >> `KSPGetPC' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:495: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:502: undefined reference to >> `PCSetType' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:502: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:503: undefined reference to >> `PCFactorSetLevels' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:503: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:510: undefined reference to >> `PCSetType' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:510: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:520: undefined reference to >> `PCFactorSetDropTolerance' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:522: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:527: undefined reference to >> `PCSetType' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:527: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:528: undefined reference to >> `PCASMSetOverlap' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:528: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:535: undefined reference to >> `KSPSetUp' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:535: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:536: undefined reference to >> `PCASMGetSubKSP' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:536: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:538: undefined reference to >> `KSPGetPC' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:538: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:539: undefined reference to >> `KSPSetType' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:539: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:540: undefined reference to >> `PCSetType' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:540: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:544: undefined reference to >> `PCSetType' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:572: undefined reference to >> `PCShellSetApply' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:573: undefined reference to >> `PCShellSetContext' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:589: undefined reference to >> `KSPSetInitialGuessNonzero' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:590: undefined reference to >> `VecSetValues' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:592: undefined reference to >> `VecAssemblyBegin' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:592: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:593: undefined reference to >> `VecAssemblyEnd' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:593: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:595: undefined reference to >> `KSPSetInitialGuessNonzero' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:601: undefined reference to >> `KSPSolve' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:602: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:613: undefined reference to >> `KSPGetConvergedReason' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:617: undefined reference to >> `KSPGetIterationNumber' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:628: undefined reference to >> `KSPGetResidualNorm' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:635: undefined reference to >> `PETSC_VIEWER_STDOUT_' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:635: undefined reference to >> `KSPView' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:636: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:664: undefined reference to >> `VecGetValues' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:677: undefined reference to >> `VecDestroy' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:677: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:678: undefined reference to >> `VecDestroy' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:678: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:679: undefined reference to >> `VecDestroy' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:679: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:680: undefined reference to >> `MatDestroy' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:680: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:681: undefined reference to >> `KSPDestroy' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:681: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:707: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:707: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:708: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:708: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function >> `dprecond_slu': >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:781: undefined reference to >> `PCShellGetContext' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:847: undefined reference to >> `VecGetValues' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:848: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:858: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:858: undefined reference to >> `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:876: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:876: undefined reference to >> `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:881: undefined reference to >> `VecSetValues' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:883: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:884: undefined reference to >> `VecAssemblyBegin' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:884: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:885: undefined reference to >> `VecAssemblyEnd' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:885: undefined reference to >> `PetscError' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function >> `dmatop': >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1025: undefined reference >> to `MatShellGetContext' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1064: undefined reference >> to `VecGetValues' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1066: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1093: undefined reference >> to `VecSetValues' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1095: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1096: undefined reference >> to `VecAssemblyBegin' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1096: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1097: undefined reference >> to `VecAssemblyEnd' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1097: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function >> `dmat_schur': >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1161: undefined reference >> to `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1161: undefined reference >> to `petsc_gather_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1379: undefined reference >> to `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1379: undefined reference >> to `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1379: undefined reference >> to `petsc_send_len' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1399: undefined reference >> to `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1399: undefined reference >> to `petsc_allreduce_ct' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function >> `dpetsc_iluksolver': >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1486: undefined reference >> to `PETSC_COMM_WORLD' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1486: undefined reference >> to `VecCreate' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1486: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1487: undefined reference >> to `VecSetSizes' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1487: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1488: undefined reference >> to `VecSetFromOptions' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1488: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1489: undefined reference >> to `VecDuplicate' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1489: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1490: undefined reference >> to `VecDuplicate' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1490: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1498: undefined reference >> to `PETSC_COMM_WORLD' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1498: undefined reference >> to `MatCreate' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1498: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1499: undefined reference >> to `MatSetSizes' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1499: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1501: undefined reference >> to `MatSetFromOptions' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1501: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1515: undefined reference >> to `MatSetValues' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1520: undefined reference >> to `MatAssemblyBegin' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1520: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1521: undefined reference >> to `MatAssemblyEnd' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1521: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1529: undefined reference >> to `VecSetValues' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1530: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1531: undefined reference >> to `VecAssemblyBegin' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1531: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1532: undefined reference >> to `VecAssemblyEnd' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1532: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1538: undefined reference >> to `PETSC_COMM_WORLD' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1538: undefined reference >> to `KSPCreate' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1539: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1548: undefined reference >> to `KSPSetOperators' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1550: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1557: undefined reference >> to `KSPSetType' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1557: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1558: undefined reference >> to `KSPSetTolerances' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1559: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1561: undefined reference >> to `KSPSetResidualHistory' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1567: undefined reference >> to `KSPGetPC' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1567: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1568: undefined reference >> to `PCSetType' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1568: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1572: undefined reference >> to `PCFactorSetLevels' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1572: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1574: undefined reference >> to `KSPSetFromOptions' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1574: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1580: undefined reference >> to `KSPSolve' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1581: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1586: undefined reference >> to `KSPGetConvergedReason' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1593: undefined reference >> to `KSPGetIterationNumber' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1594: undefined reference >> to `KSPGetResidualNorm' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1610: undefined reference >> to `VecAssemblyBegin' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1610: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1611: undefined reference >> to `VecAssemblyEnd' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1611: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1615: undefined reference >> to `VecGetValues' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1622: undefined reference >> to `PETSC_COMM_WORLD' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1622: undefined reference >> to `PETSC_VIEWER_STDOUT_' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1622: undefined reference >> to `KSPView' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1623: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1628: undefined reference >> to `VecDestroy' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1628: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1629: undefined reference >> to `VecDestroy' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1629: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1630: undefined reference >> to `VecDestroy' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1630: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1631: undefined reference >> to `MatDestroy' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1631: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1632: undefined reference >> to `KSPDestroy' >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1632: undefined reference >> to `PetscError' >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_fgmres.o): In function >> `PetscMPITypeSizeComm': >> /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:328: >> undefined reference to `PetscError' >> /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:329: >> undefined reference to `PetscError' >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Sep 1 09:22:29 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 1 Sep 2017 09:22:29 -0500 Subject: [petsc-users] PETSc in PDSLin In-Reply-To: References: Message-ID: > /usr/bin/ld: cannot find -llibpetsc It should be -lpetsc and not -llibpetsc As Matt indicated - use 'make getlinklibs' to get the corect link command [and library list] - and use it in your makefile to avoid these link errors. Satish On Fri, 1 Sep 2017, Afrah Najib wrote: > I am compiling PDSLin examples. I installed PETSc with the examples with no > errors. When I use -Wl,-rpath option , I got this error: > > :~/pdslin_2.0.0/examples$ make all > /usr/share/mpich-install/bin/mpicc -I/usr/share/mpich-install/include > -I/home/afrah/pdslin_2.0.0/include -I/home/afrah/SuperLU_DIST_5.1.2/SRC > -I/usr/include -I/usr/local/include > -I/home/afrah/PETSc-library2/petsc-3.7.6/source/include > -I/home/afrah/PETSc-library2/petsc-3.7.6-complex/source/include > -I/home/afrah/scotch-6.0.4/include -c dtest.c -o dtest.o -g -O0 -fopenmp > -std=c99 -DWITH_PETSC > dtest.c: In function ?main?: > dtest.c:53:20: warning: implicit declaration of function > ?pdslin_print_input? [-Wimplicit-function-declaration] > if ( !proc_id ) pdslin_print_input(&input); > ^ > dtest.c:127:39: warning: implicit declaration of function > ?pdslin_print_stat? [-Wimplicit-function-declaration] > if( input.verbose == PDSLin_VALL ) pdslin_print_stat( &stat, > matrix.pdslin_comm ); > ^ > /usr/share/mpich-install/bin/mpicc -fopenmp dtest.o > -L/usr/share/mpich-install/lib/libmpi.a > /home/afrah/pdslin_2.0.0/lib/libpdslin.a -lm -llapack -lblas > -Wl,-rpath,/home/afrah/SuperLU_DIST_5.1.2/lib > -L/home/afrah/SuperLU_DIST_5.1.2/lib -lsuperlu_dist > -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lmetis > -Wl,-rpath,/usr/share/parmetis/lib -L/usr/share/parmetis/lib -lparmetis > -Wl,-rpath,/home/afrah/PETSc-library2/petsc-3.7.6/source/lib > -L/home/afrah/PETSc-library2/petsc-3.7.6/source/lib -llibpetsc -o dtest -g > -O0 -fopenmp -std=c99 -DWITH_PETSC > /usr/bin/ld: cannot find -llibpetsc > collect2: error: ld returned 1 exit status > makefile:10: recipe for target 'dtest' failed > make: *** [dtest] Error 1 > > > On 1 September 2017 at 16:19, Matthew Knepley wrote: > > > On Fri, Sep 1, 2017 at 5:48 AM, Afrah Najib wrote: > > > >> I want to use use PETSc in a hybrid solver(direct- iterative) called > >> PDSLin 2.0.0.[http://portal.nersc.gov/project/sparse/pdslin/] > >> > >> the version of PETSc used is petsc3.7.6. When running the examples, I got > >> the following errors: > >> > >> > >> > >> :~/pdslin_2.0.0/examples$ make all > >> /usr/share/mpich-install/bin/mpicc -I/usr/share/mpich-install/include > >> -I/home/afrah/pdslin_2.0.0/include -I/home/afrah/SuperLU_DIST_5.1.2/SRC > >> -I/usr/include -I/usr/local/include -I/home/afrah/PETSc-library2/petsc-3.7.6/source/include > >> -I/home/afrah/PETSc-library2/petsc-3.7.6-complex/source/include > >> -I/home/afrah/scotch-6.0.4/include -c dtest.c -o dtest.o -g -O0 -fopenmp > >> -std=c99 -DWITH_PETSC > >> dtest.c: In function ?main?: > >> dtest.c:53:20: warning: implicit declaration of function > >> ?pdslin_print_input? [-Wimplicit-function-declaration] > >> if ( !proc_id ) pdslin_print_input(&input); > >> ^ > >> dtest.c:127:39: warning: implicit declaration of function > >> ?pdslin_print_stat? [-Wimplicit-function-declaration] > >> if( input.verbose == PDSLin_VALL ) pdslin_print_stat( &sta > >> > > > > Note that below, you do not have -lpetsc. I would recommend using the > > PETSc makefiles, but you can also run 'make getlinklibs' to the correct > > link line. > > > > Thanks, > > > > Matt > > > > > >> ^ > >> /usr/share/mpich-install/bin/mpicc -fopenmp dtest.o > >> -L/usr/share/mpich-install/lib/libmpi.a /home/afrah/pdslin_2.0.0/lib/libpdslin.a > >> -lm -llapack -lblas -Wl,-rpath,/home/afrah/SuperLU_DIST_5.1.2/lib > >> -L/home/afrah/SuperLU_DIST_5.1.2/lib -lsuperlu_dist > >> -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lmetis > >> -Wl,-rpath,/usr/share/parmetis/lib -L/usr/share/parmetis/lib -lparmetis > >> -L/home/afrah/PETSc-library2/petsc-3.7.6/source/lib/libpetsc.so -o > >> dtest -g -O0 -fopenmp -std=c99 -DWITH_PETSC > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > >> `PetscMPITypeSizeComm': > >> /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:328: > >> undefined reference to `PetscError' > >> /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:329: > >> undefined reference to `PetscError' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > >> `dcomp_schur': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:92: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:92: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:121: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:121: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:129: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ho > >> me/afrah/pdslin_2.0.0/src/dpdslin_core.c:129: more undefined references > >> to `petsc_gather_ct' follow > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > >> `dcomp_sol': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:486: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:486: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:495: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:495: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:496: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ho > >> me/afrah/pdslin_2.0.0/src/dpdslin_core.c:496: more undefined references > >> to `petsc_allreduce_ct' follow > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > >> `dcomp_sol': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:614: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:614: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:625: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:625: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:625: undefined reference to > >> `petsc_send_len' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:659: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:659: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:793: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:793: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:793: undefined reference to > >> `petsc_send_len' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:795: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:795: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:795: undefined reference to > >> `petsc_send_len' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:803: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:803: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:812: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:812: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:862: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:862: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:891: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ho > >> me/afrah/pdslin_2.0.0/src/dpdslin_core.c:891: more undefined references > >> to `petsc_allreduce_ct' follow > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > >> `dallgather_ssol': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1094: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1094: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > >> `dsparsify_schur': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1331: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1331: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1357: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o):/ho > >> me/afrah/pdslin_2.0.0/src/dpdslin_core.c:1357: more undefined references > >> to `petsc_gather_ct' follow > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_core.o): In function > >> `dsparsify_schur': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1789: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1789: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1861: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_core.c:1861: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > >> `PetscMPITypeSizeComm': > >> /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:328: > >> undefined reference to `PetscError' > >> /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:329: > >> undefined reference to `PetscError' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > >> `dpetsc_init': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:60: undefined reference to > >> `PetscInitialized' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:62: undefined reference to > >> `PetscInitialize' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:63: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > >> `dpetsc_clean': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:71: undefined reference to > >> `PetscFinalize' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:72: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > >> `dsolve_schur_petsc': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:164: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:164: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:164: undefined reference to > >> `petsc_send_len' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:229: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:229: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:229: undefined reference to > >> `petsc_send_len' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:251: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:251: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:251: undefined reference to > >> `petsc_send_len' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:292: undefined reference to > >> `VecCreate' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:292: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:293: undefined reference to > >> `VecSetSizes' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:293: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:294: undefined reference to > >> `VecSetFromOptions' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:294: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:295: undefined reference to > >> `VecDuplicate' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:295: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:296: undefined reference to > >> `VecDuplicate' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:296: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:305: undefined reference to > >> `MatCreate' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:305: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:306: undefined reference to > >> `MatSetSizes' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:306: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:308: undefined reference to > >> `MatSetFromOptions' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:308: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:311: undefined reference to > >> `MatCreateShell' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:312: undefined reference to > >> `MatShellSetOperation' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:325: undefined reference to > >> `MatCreate' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:325: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:326: undefined reference to > >> `MatSetSizes' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:326: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:328: undefined reference to > >> `MatSetFromOptions' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:328: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:346: undefined reference to > >> `MatMPIAIJSetPreallocation' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:347: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:348: undefined reference to > >> `MatSeqAIJSetPreallocation' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:349: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:359: undefined reference to > >> `MatMPIAIJSetPreallocation' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:360: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:361: undefined reference to > >> `MatSeqAIJSetPreallocation' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:362: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:377: undefined reference to > >> `MatSetValues' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:378: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:383: undefined reference to > >> `MatAssemblyBegin' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:383: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:384: undefined reference to > >> `MatAssemblyEnd' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:384: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:386: undefined reference to > >> `MatAssemblyBegin' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:386: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:387: undefined reference to > >> `MatAssemblyEnd' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:387: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:396: undefined reference to > >> `VecSetValues' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:397: undefined reference to > >> `VecAssemblyBegin' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:397: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:398: undefined reference to > >> `VecAssemblyEnd' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:398: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:409: undefined reference to > >> `KSPCreate' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:410: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:421: undefined reference to > >> `KSPSetOperators' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:427: undefined reference to > >> `KSPSetOperators' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:430: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:439: undefined reference to > >> `KSPSetFromOptions' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:439: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:440: undefined reference to > >> `KSPSetTolerances' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:441: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:450: undefined reference to > >> `KSPSetType' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:450: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:451: undefined reference to > >> `KSPGMRESSetRestart' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:451: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:463: undefined reference to > >> `KSPSetType' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:463: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:468: undefined reference to > >> `KSPGMRESSetRestart' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:468: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:470: undefined reference to > >> `KSPGMRESModifiedGramSchmidtOrthogonalization' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:470: undefined reference to > >> `KSPGMRESSetOrthogonalization' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:471: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:473: undefined reference to > >> `KSPGMRESClassicalGramSchmidtOrthogonalization' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:473: undefined reference to > >> `KSPGMRESSetOrthogonalization' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:474: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:481: undefined reference to > >> `KSPSetType' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:481: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:483: undefined reference to > >> `KSPSetPCSide' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:483: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:488: undefined reference to > >> `KSPSetType' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:488: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:495: undefined reference to > >> `KSPGetPC' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:495: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:502: undefined reference to > >> `PCSetType' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:502: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:503: undefined reference to > >> `PCFactorSetLevels' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:503: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:510: undefined reference to > >> `PCSetType' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:510: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:520: undefined reference to > >> `PCFactorSetDropTolerance' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:522: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:527: undefined reference to > >> `PCSetType' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:527: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:528: undefined reference to > >> `PCASMSetOverlap' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:528: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:535: undefined reference to > >> `KSPSetUp' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:535: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:536: undefined reference to > >> `PCASMGetSubKSP' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:536: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:538: undefined reference to > >> `KSPGetPC' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:538: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:539: undefined reference to > >> `KSPSetType' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:539: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:540: undefined reference to > >> `PCSetType' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:540: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:544: undefined reference to > >> `PCSetType' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:572: undefined reference to > >> `PCShellSetApply' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:573: undefined reference to > >> `PCShellSetContext' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:589: undefined reference to > >> `KSPSetInitialGuessNonzero' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:590: undefined reference to > >> `VecSetValues' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:592: undefined reference to > >> `VecAssemblyBegin' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:592: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:593: undefined reference to > >> `VecAssemblyEnd' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:593: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:595: undefined reference to > >> `KSPSetInitialGuessNonzero' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:601: undefined reference to > >> `KSPSolve' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:602: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:613: undefined reference to > >> `KSPGetConvergedReason' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:617: undefined reference to > >> `KSPGetIterationNumber' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:628: undefined reference to > >> `KSPGetResidualNorm' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:635: undefined reference to > >> `PETSC_VIEWER_STDOUT_' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:635: undefined reference to > >> `KSPView' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:636: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:664: undefined reference to > >> `VecGetValues' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:677: undefined reference to > >> `VecDestroy' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:677: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:678: undefined reference to > >> `VecDestroy' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:678: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:679: undefined reference to > >> `VecDestroy' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:679: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:680: undefined reference to > >> `MatDestroy' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:680: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:681: undefined reference to > >> `KSPDestroy' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:681: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:707: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:707: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:708: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:708: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > >> `dprecond_slu': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:781: undefined reference to > >> `PCShellGetContext' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:847: undefined reference to > >> `VecGetValues' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:848: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:858: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:858: undefined reference to > >> `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:876: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:876: undefined reference to > >> `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:881: undefined reference to > >> `VecSetValues' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:883: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:884: undefined reference to > >> `VecAssemblyBegin' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:884: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:885: undefined reference to > >> `VecAssemblyEnd' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:885: undefined reference to > >> `PetscError' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > >> `dmatop': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1025: undefined reference > >> to `MatShellGetContext' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1064: undefined reference > >> to `VecGetValues' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1066: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1093: undefined reference > >> to `VecSetValues' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1095: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1096: undefined reference > >> to `VecAssemblyBegin' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1096: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1097: undefined reference > >> to `VecAssemblyEnd' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1097: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > >> `dmat_schur': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1161: undefined reference > >> to `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1161: undefined reference > >> to `petsc_gather_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1379: undefined reference > >> to `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1379: undefined reference > >> to `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1379: undefined reference > >> to `petsc_send_len' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1399: undefined reference > >> to `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1399: undefined reference > >> to `petsc_allreduce_ct' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_petsc.o): In function > >> `dpetsc_iluksolver': > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1486: undefined reference > >> to `PETSC_COMM_WORLD' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1486: undefined reference > >> to `VecCreate' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1486: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1487: undefined reference > >> to `VecSetSizes' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1487: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1488: undefined reference > >> to `VecSetFromOptions' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1488: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1489: undefined reference > >> to `VecDuplicate' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1489: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1490: undefined reference > >> to `VecDuplicate' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1490: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1498: undefined reference > >> to `PETSC_COMM_WORLD' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1498: undefined reference > >> to `MatCreate' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1498: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1499: undefined reference > >> to `MatSetSizes' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1499: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1501: undefined reference > >> to `MatSetFromOptions' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1501: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1515: undefined reference > >> to `MatSetValues' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1520: undefined reference > >> to `MatAssemblyBegin' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1520: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1521: undefined reference > >> to `MatAssemblyEnd' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1521: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1529: undefined reference > >> to `VecSetValues' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1530: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1531: undefined reference > >> to `VecAssemblyBegin' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1531: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1532: undefined reference > >> to `VecAssemblyEnd' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1532: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1538: undefined reference > >> to `PETSC_COMM_WORLD' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1538: undefined reference > >> to `KSPCreate' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1539: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1548: undefined reference > >> to `KSPSetOperators' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1550: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1557: undefined reference > >> to `KSPSetType' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1557: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1558: undefined reference > >> to `KSPSetTolerances' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1559: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1561: undefined reference > >> to `KSPSetResidualHistory' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1567: undefined reference > >> to `KSPGetPC' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1567: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1568: undefined reference > >> to `PCSetType' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1568: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1572: undefined reference > >> to `PCFactorSetLevels' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1572: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1574: undefined reference > >> to `KSPSetFromOptions' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1574: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1580: undefined reference > >> to `KSPSolve' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1581: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1586: undefined reference > >> to `KSPGetConvergedReason' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1593: undefined reference > >> to `KSPGetIterationNumber' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1594: undefined reference > >> to `KSPGetResidualNorm' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1610: undefined reference > >> to `VecAssemblyBegin' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1610: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1611: undefined reference > >> to `VecAssemblyEnd' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1611: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1615: undefined reference > >> to `VecGetValues' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1622: undefined reference > >> to `PETSC_COMM_WORLD' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1622: undefined reference > >> to `PETSC_VIEWER_STDOUT_' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1622: undefined reference > >> to `KSPView' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1623: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1628: undefined reference > >> to `VecDestroy' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1628: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1629: undefined reference > >> to `VecDestroy' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1629: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1630: undefined reference > >> to `VecDestroy' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1630: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1631: undefined reference > >> to `MatDestroy' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1631: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1632: undefined reference > >> to `KSPDestroy' > >> /home/afrah/pdslin_2.0.0/src/dpdslin_petsc.c:1632: undefined reference > >> to `PetscError' > >> /home/afrah/pdslin_2.0.0/lib/libpdslin.a(dpdslin_fgmres.o): In function > >> `PetscMPITypeSizeComm': > >> /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:328: > >> undefined reference to `PetscError' > >> /home/afrah/PETSc-library2/petsc-3.7.6/source/include/petsclog.h:329: > >> undefined reference to `PetscError' > >> > >> > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which their > > experiments lead. > > -- Norbert Wiener > > > > http://www.caam.rice.edu/~mk51/ > > > From bsmith at mcs.anl.gov Fri Sep 1 09:57:48 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 1 Sep 2017 09:57:48 -0500 Subject: [petsc-users] Fwd: direct solvers on KNL In-Reply-To: <74eb23c0-d5f8-bb22-6a60-547b80c2640b@vsb.cz> References: <74eb23c0-d5f8-bb22-6a60-547b80c2640b@vsb.cz> Message-ID: <51B1A3DB-BB20-48CA-8FB0-4A40BAAB615F@mcs.anl.gov> Since MUMPS and SuperLU don't have much profiling, you really need to use a profiling system, probably Intel's VTune is the way to go, to understand the performance on that machine. Barry > On Sep 1, 2017, at 6:40 AM, Jakub Kruzik wrote: > > Hi, > > I am looking at a single node performance of MUMPS and SuperLU on KNL 7230 (on Theta). I am using KSP example ex2 (http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html) with m X n = 2880 x 2880. KNL runs in cache and quad modes. > > Times in seconds for 24 cores: > mumps: 279 > superlu: 326 > cg: 116 > > Times in seconds for 64 cores: > mumps: 316 > superlu: 410 > cg : 49 > > The performance for 24 cores is OK - both direct solvers are roughly 3.5 times slower than 2x E5-2680v3. (According to people from Intel, the single core performance of KNL is about 3-4 times lower than that of E5-2680v3). However, strong scalability is really bad. > > I am using cray-petsc/3.7.6.0 module. I tried my own PETSc compilation with MKL and MUMPS/SuperLU installed by PETSc configure but the results are similar. > > Please find attached Theta submission script and logs for KNL and Haswells. > > Why the performance of direct solvers on a full node is so bad? > > Best, > Jakub > From jed at jedbrown.org Fri Sep 1 11:36:29 2017 From: jed at jedbrown.org (Jed Brown) Date: Fri, 01 Sep 2017 10:36:29 -0600 Subject: [petsc-users] Fwd: direct solvers on KNL In-Reply-To: References: <74eb23c0-d5f8-bb22-6a60-547b80c2640b@vsb.cz> Message-ID: <87ziaecr5e.fsf@jedbrown.org> Matthew Knepley writes: > Admittedly it was for different computations, but we saw strong scaling > degradation after 32 cores of KNL > in https://arxiv.org/abs/1705.09907, What is "heterogeneous" about KNL? Also, how is it an "accelerator" (a term created by NVIDIA marketing to replace "coprocessor"). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From zakaryah at gmail.com Fri Sep 1 16:31:23 2017 From: zakaryah at gmail.com (zakaryah .) Date: Fri, 1 Sep 2017 17:31:23 -0400 Subject: [petsc-users] A number of questions about DMDA with SNES and Quasi-Newton methods In-Reply-To: References: <020DAED9-AAAD-4741-976B-BB2EF6C79D3F@mcs.anl.gov> <5BF530F6-8D79-47E6-B2C5-B0702FF2AA59@mcs.anl.gov> <7EE84495-4346-4BFB-B9AD-BCAA3C722461@mcs.anl.gov> <08AA1273-062D-4FDD-8709-40AA1FE8AA92@mcs.anl.gov> <06932E72-E8F3-4EA8-889F-A570AD660E32@mcs.anl.gov> <98D45588-381F-44FB-9D20-65D1638E357F@mcs.anl.gov> Message-ID: You were right - I don't think the Jacobian has errors but there was a mistake in the function evaluation. Now when I run with -snes_type newtonls -snes_monitor -ksp_monitor -ksp_monitor_true_residual -ksp_converged_reason -snes_converged_reason -pc_type lu -snes_linesearch_monitor the SNES converges fairly quickly and there are no problems with the line search. I also ran with -snes_type test -snes_test_display, at various initial guesses, and in all cases the hand-coded Jacobian agreed well with the finite difference Jacobian. Unfortunately, I need to solve the system on a grid which is too large for LU decomp. When I try running the same grid with the same options except without -pc_type lu, everything converges fine. However, when I increase the grid size, even by a factor 2 in each dimension, the KSP no longer converges with the default options (neither the preconditioned residual nor the true residual decrease from about iteration 6000 onward). Should I move on to looking into the linear solver? To check if the linear system is singular, I ran the problem on the small grid (KSP converges here but not for 2^3-fold larger grid) with -pc_type svd -pc_svd_monitor and got the following output: 0 SNES Function norm 1.259563818203e+03 SVD: condition number 9.125223106399e+02, 0 of 4590 singular values are (nearly) zero SVD: smallest singular values: 6.205544702362e+00 8.361112027794e+00 9.327934960932e+00 1.250439191766e+01 1.373952564946e+01 SVD: largest singular values : 5.497346539524e+03 5.517949695999e+03 5.539227197966e+03 5.596380525169e+03 5.662697990578e+03 0 KSP Residual norm 7.340750993927e+00 0 KSP preconditioned resid norm 7.340750993927e+00 true resid norm 1.259563818203e+03 ||r(i)||/||b|| 1.000000000000e+00 1 KSP Residual norm 8.452960708813e-13 1 KSP preconditioned resid norm 8.452960708813e-13 true resid norm 2.092403440031e-10 ||r(i)||/||b|| 1.661212722842e-13 Linear solve converged due to CONVERGED_RTOL iterations 1 Line search: Using full step: fnorm 1.259563818203e+03 gnorm 2.909228222342e+02 1 SNES Function norm 2.909228222342e+02 SVD: condition number 8.837760047818e+02, 0 of 4590 singular values are (nearly) zero SVD: smallest singular values: 6.853997087392e+00 8.414157285275e+00 9.547538541816e+00 1.307676522991e+01 1.424251871761e+01 SVD: largest singular values : 5.674169364836e+03 5.711489755811e+03 5.754496402805e+03 5.905177592591e+03 6.057398162681e+03 0 KSP Residual norm 6.537083346419e-01 0 KSP preconditioned resid norm 6.537083346419e-01 true resid norm 2.909228222342e+02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP Residual norm 4.766960746477e-14 1 KSP preconditioned resid norm 4.766960746477e-14 true resid norm 2.206377081276e-11 ||r(i)||/||b|| 7.584063238254e-14 Linear solve converged due to CONVERGED_RTOL iterations 1 Line search: Using full step: fnorm 2.909228222342e+02 gnorm 8.519091421012e+00 2 SNES Function norm 8.519091421012e+00 SVD: condition number 8.866866174487e+02, 0 of 4590 singular values are (nearly) zero SVD: smallest singular values: 6.754153695550e+00 8.386828076859e+00 9.514891596031e+00 1.302462235577e+01 1.424263827525e+01 SVD: largest singular values : 5.671563211881e+03 5.685626194828e+03 5.739548236997e+03 5.833938566429e+03 5.988817694036e+03 0 KSP Residual norm 3.040350335176e-02 0 KSP preconditioned resid norm 3.040350335176e-02 true resid norm 8.519091421012e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP Residual norm 3.285565040836e-15 1 KSP preconditioned resid norm 3.285565040836e-15 true resid norm 9.005553787171e-13 ||r(i)||/||b|| 1.057102611314e-13 Linear solve converged due to CONVERGED_RTOL iterations 1 Line search: Using full step: fnorm 8.519091421012e+00 gnorm 3.086711801483e-02 3 SNES Function norm 3.086711801483e-02 SVD: condition number 8.863731497353e+02, 0 of 4590 singular values are (nearly) zero SVD: smallest singular values: 6.750850248606e+00 8.386167701356e+00 9.513896745592e+00 1.302218359138e+01 1.424165997875e+01 SVD: largest singular values : 5.669769588361e+03 5.683785644290e+03 5.736938066065e+03 5.828997334185e+03 5.983772398249e+03 0 KSP Residual norm 3.152844718957e-05 0 KSP preconditioned resid norm 3.152844718957e-05 true resid norm 3.086711801483e-02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP Residual norm 3.048057745094e-18 1 KSP preconditioned resid norm 3.048057745094e-18 true resid norm 9.890968076924e-16 ||r(i)||/||b|| 3.204370447598e-14 Linear solve converged due to CONVERGED_RTOL iterations 1 Line search: Using full step: fnorm 3.086711801483e-02 gnorm 2.527921600901e-07 4 SNES Function norm 2.527921600901e-07 Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 4 I suppose it is possible that the linear system becomes more singular as the grid size increases but I think it will take forever to run SVD on it. I have not tried other PC or KSP methods. Any advice? On Wed, Aug 30, 2017 at 12:37 PM, Matthew Knepley wrote: > On Wed, Aug 30, 2017 at 12:21 AM, zakaryah . wrote: > >> ?OK, this is very helpful (and not totally clear from the "Why is >> Newton's method not converging" FAQ at this address: >> https://scicomp.stackexchange.com/questions/30/why-is-newton >> s-method-not-converging. Here's the output, using the intermediate grid >> and the options -snes_type newtonls -snes_monitor -ksp_monitor >> -ksp_monitor_true_residual -ksp_converged_reason -snes_converged_reason >> -pc_type lu -snes_linesearch_monitor:? >> >> 0 SNES Function norm 1.370293318432e+04 >> >> 0 KSP Residual norm 7.457516441709e+02 >> >> 0 KSP preconditioned resid norm 7.457516441709e+02 true resid norm >> 1.370293318432e+04 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 1.244742959480e-10 >> >> 1 KSP preconditioned resid norm 1.244742959480e-10 true resid norm >> 9.866304569403e-09 ||r(i)||/||b|| 7.200140609815e-13 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 6.541326653047e+05 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 8.963755251044e+04 lambda=5.0000000000000003e-02 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.788913078169e+04 lambda=2.5000000000000001e-02 >> >> Line search: Cubically determined step, current gnorm >> 1.340565818376e+04 lambda=1.2500000000000001e-02 >> > Your Jacobian is wrong. You solved the Newton linear system, which should > be guaranteed to give a residual reduction, but > you get nothing. Thus we conclude that your Jacobian is wrong, giving a > wrong descent direction. I would recommend simplifying > the problem until you can check the Jacobian entries by hand. > > Thanks, > > Matt > >> 1 SNES Function norm 1.340565818376e+04 >> >> 0 KSP Residual norm 3.837938728414e+02 >> >> 0 KSP preconditioned resid norm 3.837938728414e+02 true resid norm >> 1.340565818376e+04 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 3.935624425879e-11 >> >> 1 KSP preconditioned resid norm 3.935624425879e-11 true resid norm >> 4.012917478400e-09 ||r(i)||/||b|| 2.993450544086e-13 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 7.752009107979e+05 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 9.942192482364e+04 lambda=5.0000000000000003e-02 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.724069009956e+04 lambda=2.5000000000000001e-02 >> >> Line search: Cubically determined step, current gnorm >> 1.272263847462e+04 lambda=1.2500000000000001e-02 >> >> 2 SNES Function norm 1.272263847462e+04 >> >> 0 KSP Residual norm 8.610741245938e+01 >> >> 0 KSP preconditioned resid norm 8.610741245938e+01 true resid norm >> 1.272263847462e+04 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 3.953708398506e-12 >> >> 1 KSP preconditioned resid norm 3.953708398506e-12 true resid norm >> 1.259105691297e-09 ||r(i)||/||b|| 9.896576828850e-14 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 1.126216236986e+04 >> >> Line search: Quadratically determined step, >> lambda=1.0000000000000001e-01 >> >> 3 SNES Function norm 1.126216236986e+04 >> >> 0 KSP Residual norm 6.100581225849e+01 >> >> 0 KSP preconditioned resid norm 6.100581225849e+01 true resid norm >> 1.126216236986e+04 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 3.714526144776e-12 >> >> 1 KSP preconditioned resid norm 3.714526144776e-12 true resid norm >> 1.150524854441e-09 ||r(i)||/||b|| 1.021584325156e-13 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 9.992994468502e+03 >> >> Line search: Quadratically determined step, >> lambda=1.0000000000000001e-01 >> >> 4 SNES Function norm 9.992994468502e+03 >> >> 0 KSP Residual norm 5.452156705070e+01 >> >> 0 KSP preconditioned resid norm 5.452156705070e+01 true resid norm >> 9.992994468502e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 2.345517221383e-12 >> >> 1 KSP preconditioned resid norm 2.345517221383e-12 true resid norm >> 6.645776037775e-10 ||r(i)||/||b|| 6.650435020976e-14 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 8.954956134809e+03 >> >> Line search: Quadratically determined step, >> lambda=1.0000000000000001e-01 >> >> 5 SNES Function norm 8.954956134809e+03 >> >> 0 KSP Residual norm 1.869342714623e+02 >> >> 0 KSP preconditioned resid norm 1.869342714623e+02 true resid norm >> 8.954956134809e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 6.952075013553e-12 >> >> 1 KSP preconditioned resid norm 6.952075013553e-12 true resid norm >> 2.332836306210e-09 ||r(i)||/||b|| 2.605078429298e-13 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 9.366608314830e+04 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.702858832608e+04 lambda=5.0000000000000003e-02 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 9.141708166107e+03 lambda=2.5000000000000001e-02 >> >> Line search: Cubically determined step, current gnorm >> 8.847002105227e+03 lambda=1.2500000000000001e-02 >> >> 6 SNES Function norm 8.847002105227e+03 >> >> 0 KSP Residual norm 4.136774315749e+01 >> >> 0 KSP preconditioned resid norm 4.136774315749e+01 true resid norm >> 8.847002105227e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 2.340140336645e-12 >> >> 1 KSP preconditioned resid norm 2.340140336645e-12 true resid norm >> 6.952836046439e-10 ||r(i)||/||b|| 7.858974106416e-14 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 7.928904942575e+03 >> >> Line search: Quadratically determined step, >> lambda=1.0000000000000001e-01 >> >> 7 SNES Function norm 7.928904942575e+03 >> >> 0 KSP Residual norm 9.421715655897e+01 >> >> 0 KSP preconditioned resid norm 9.421715655897e+01 true resid norm >> 7.928904942575e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 5.662350656880e-12 >> >> 1 KSP preconditioned resid norm 5.662350656880e-12 true resid norm >> 1.323447274435e-09 ||r(i)||/||b|| 1.669142566370e-13 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 4.679455846837e+04 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.024301325332e+04 lambda=5.0000000000000003e-02 >> >> Line search: Cubically determined step, current gnorm >> 7.882357228991e+03 lambda=2.5000000000000001e-02 >> >> 8 SNES Function norm 7.882357228991e+03 >> >> 0 KSP Residual norm 3.101373992455e+01 >> >> 0 KSP preconditioned resid norm 3.101373992455e+01 true resid norm >> 7.882357228991e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 1.176793992597e-12 >> >> 1 KSP preconditioned resid norm 1.176793992597e-12 true resid norm >> 4.948698048884e-10 ||r(i)||/||b|| 6.278195602050e-14 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: Using full step: fnorm 7.882357228991e+03 gnorm >> 5.263287196819e+03 >> >> 9 SNES Function norm 5.263287196819e+03 >> >> 0 KSP Residual norm 8.541947236232e+00 >> >> 0 KSP preconditioned resid norm 8.541947236232e+00 true resid norm >> 5.263287196819e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 1.741221887649e-12 >> >> 1 KSP preconditioned resid norm 1.741221887649e-12 true resid norm >> 9.735248249306e-10 ||r(i)||/||b|| 1.849651726242e-13 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 4.836381737060e+03 >> >> Line search: Quadratically determined step, >> lambda=1.0000000000000001e-01 >> >> 10 SNES Function norm 4.836381737060e+03 >> >> 0 KSP Residual norm 1.926518638579e+01 >> >> 0 KSP preconditioned resid norm 1.926518638579e+01 true resid norm >> 4.836381737060e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 1.278653333162e-11 >> >> 1 KSP preconditioned resid norm 1.278653333162e-11 true resid norm >> 1.211659620475e-08 ||r(i)||/||b|| 2.505301868938e-12 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 4.550187794983e+03 >> >> Line search: Quadratically determined step, >> lambda=1.0000000000000001e-01 >> >> 11 SNES Function norm 4.550187794983e+03 >> >> 0 KSP Residual norm 5.492305435155e+00 >> >> 0 KSP preconditioned resid norm 5.492305435155e+00 true resid norm >> 4.550187794983e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 3.941029797735e-13 >> >> 1 KSP preconditioned resid norm 3.941029797735e-13 true resid norm >> 2.123615113926e-10 ||r(i)||/||b|| 4.667093336823e-14 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 3.210238905481e+03 >> >> Line search: Quadratically determined step, >> lambda=4.4544417493894700e-01 >> >> 12 SNES Function norm 3.210238905481e+03 >> >> 0 KSP Residual norm 7.578202966222e+00 >> >> 0 KSP preconditioned resid norm 7.578202966222e+00 true resid norm >> 3.210238905481e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 3.800509410151e-12 >> >> 1 KSP preconditioned resid norm 3.800509410151e-12 true resid norm >> 3.408323024026e-09 ||r(i)||/||b|| 1.061703855812e-12 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 2.924559961092e+03 >> >> Line search: Quadratically determined step, >> lambda=1.0000000000000001e-01 >> >> 13 SNES Function norm 2.924559961092e+03 >> >> 0 KSP Residual norm 5.076980628354e+00 >> >> 0 KSP preconditioned resid norm 5.076980628354e+00 true resid norm >> 2.924559961092e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 1.468522311180e-13 >> >> 1 KSP preconditioned resid norm 1.468522311180e-13 true resid norm >> 9.059064389128e-11 ||r(i)||/||b|| 3.097582032733e-14 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 2.389633730237e+03 >> >> Line search: Quadratically determined step, >> lambda=2.1257217805235396e-01 >> >> 14 SNES Function norm 2.389633730237e+03 >> >> 0 KSP Residual norm 5.166598232781e+00 >> >> 0 KSP preconditioned resid norm 5.166598232781e+00 true resid norm >> 2.389633730237e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 2.249406847764e-13 >> >> 1 KSP preconditioned resid norm 2.249406847764e-13 true resid norm >> 4.780078098549e-11 ||r(i)||/||b|| 2.000339231098e-14 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 2.093485770827e+03 >> >> Line search: Quadratically determined step, >> lambda=1.5993239844479040e-01 >> >> 15 SNES Function norm 2.093485770827e+03 >> >> 0 KSP Residual norm 8.445727235216e+00 >> >> 0 KSP preconditioned resid norm 8.445727235216e+00 true resid norm >> 2.093485770827e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 1.917214128329e-13 >> >> 1 KSP preconditioned resid norm 1.917214128329e-13 true resid norm >> 1.577089911021e-10 ||r(i)||/||b|| 7.533320421842e-14 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 1.967949189728e+03 >> >> Line search: Quadratically determined step, >> lambda=1.0000000000000001e-01 >> >> 16 SNES Function norm 1.967949189728e+03 >> >> 0 KSP Residual norm 2.548962277036e+01 >> >> 0 KSP preconditioned resid norm 2.548962277036e+01 true resid norm >> 1.967949189728e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 4.393171339150e-12 >> >> 1 KSP preconditioned resid norm 4.393171339150e-12 true resid norm >> 6.268541493766e-10 ||r(i)||/||b|| 3.185316738097e-13 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 3.445905389608e+03 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.129006719528e+03 lambda=5.0000000000000003e-02 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.969320033677e+03 lambda=1.8287150237426619e-02 >> >> Line search: Cubically determined step, current gnorm >> 1.963119931876e+03 lambda=8.7403113229804347e-03 >> >> 17 SNES Function norm 1.963119931876e+03 >> >> 0 KSP Residual norm 1.189834379636e+02 >> >> 0 KSP preconditioned resid norm 1.189834379636e+02 true resid norm >> 1.963119931876e+03 ||r(i)||/||b|| 1.000000000000e+00 >> >> 1 KSP Residual norm 3.213937715681e-11 >> >> 1 KSP preconditioned resid norm 3.213937715681e-11 true resid norm >> 8.137767291258e-08 ||r(i)||/||b|| 4.145323553147e-11 >> >> Linear solve converged due to CONVERGED_RTOL iterations 1 >> >> Line search: gnorm after quadratic fit 1.329932469429e+05 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.079971894561e+04 lambda=5.0000000000000003e-02 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 4.672632539568e+03 lambda=2.5000000000000001e-02 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 2.337320012333e+03 lambda=1.2500000000000001e-02 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.989655494060e+03 lambda=3.7908486903933652e-03 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.964061592872e+03 lambda=4.3979444116724767e-04 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963285343925e+03 lambda=9.8760011176842895e-05 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963156689078e+03 lambda=2.3385231527184855e-05 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963128672876e+03 lambda=5.6482781216089357e-06 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963122042807e+03 lambda=1.3692247365910180e-06 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963120443658e+03 lambda=3.3226543340386796e-07 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963120056069e+03 lambda=8.0648305027594934e-08 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963119962021e+03 lambda=1.9576315045380916e-08 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963119939193e+03 lambda=4.7519586695347587e-09 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963119933652e+03 lambda=1.1534950726729967e-09 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963119932307e+03 lambda=2.8000026447574534e-10 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963119931981e+03 lambda=6.7967445495582766e-11 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963119931902e+03 lambda=1.6498100190994443e-11 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963119931882e+03 lambda=4.0046243381676681e-12 >> >> Line search: Cubic step no good, shrinking lambda, current gnorm >> 1.963119931878e+03 lambda=9.7209812173761137e-13 >> >> Line search: unable to find good step length! After 19 tries >> >> Line search: fnorm=1.9631199318761367e+03, >> gnorm=1.9631199318776339e+03, ynorm=1.1898343796305618e+02, >> minlambda=9.9999999999999998e-13, lambda=9.7209812173761137e-13, initial >> slope=-3.8538398668951751e+06 >> >> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 17 >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Sep 2 08:31:38 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 2 Sep 2017 08:31:38 -0500 Subject: [petsc-users] A number of questions about DMDA with SNES and Quasi-Newton methods In-Reply-To: References: <020DAED9-AAAD-4741-976B-BB2EF6C79D3F@mcs.anl.gov> <5BF530F6-8D79-47E6-B2C5-B0702FF2AA59@mcs.anl.gov> <7EE84495-4346-4BFB-B9AD-BCAA3C722461@mcs.anl.gov> <08AA1273-062D-4FDD-8709-40AA1FE8AA92@mcs.anl.gov> <06932E72-E8F3-4EA8-889F-A570AD660E32@mcs.anl.gov> <98D45588-381F-44FB-9D20-65D1638E357F@mcs.anl.gov> Message-ID: <876F0ACF-62C0-4DC9-B203-2893B25E3938@mcs.anl.gov> Have you tried -pc_type hypre and then -pc_type gamg ? Where does your problem come from? What physical models and what type of discretization? Barry > On Sep 1, 2017, at 4:31 PM, zakaryah . wrote: > > You were right - I don't think the Jacobian has errors but there was a mistake in the function evaluation. Now when I run with -snes_type newtonls -snes_monitor -ksp_monitor -ksp_monitor_true_residual -ksp_converged_reason -snes_converged_reason -pc_type lu -snes_linesearch_monitor the SNES converges fairly quickly and there are no problems with the line search. I also ran with -snes_type test -snes_test_display, at various initial guesses, and in all cases the hand-coded Jacobian agreed well with the finite difference Jacobian. > > Unfortunately, I need to solve the system on a grid which is too large for LU decomp. When I try running the same grid with the same options except without -pc_type lu, everything converges fine. However, when I increase the grid size, even by a factor 2 in each dimension, the KSP no longer converges with the default options (neither the preconditioned residual nor the true residual decrease from about iteration 6000 onward). Should I move on to looking into the linear solver? > > To check if the linear system is singular, I ran the problem on the small grid (KSP converges here but not for 2^3-fold larger grid) with -pc_type svd -pc_svd_monitor and got the following output: > > 0 SNES Function norm 1.259563818203e+03 > > SVD: condition number 9.125223106399e+02, 0 of 4590 singular values are (nearly) zero > > SVD: smallest singular values: 6.205544702362e+00 8.361112027794e+00 9.327934960932e+00 1.250439191766e+01 1.373952564946e+01 > > SVD: largest singular values : 5.497346539524e+03 5.517949695999e+03 5.539227197966e+03 5.596380525169e+03 5.662697990578e+03 > > 0 KSP Residual norm 7.340750993927e+00 > > 0 KSP preconditioned resid norm 7.340750993927e+00 true resid norm 1.259563818203e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 8.452960708813e-13 > > 1 KSP preconditioned resid norm 8.452960708813e-13 true resid norm 2.092403440031e-10 ||r(i)||/||b|| 1.661212722842e-13 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: Using full step: fnorm 1.259563818203e+03 gnorm 2.909228222342e+02 > > 1 SNES Function norm 2.909228222342e+02 > > SVD: condition number 8.837760047818e+02, 0 of 4590 singular values are (nearly) zero > > SVD: smallest singular values: 6.853997087392e+00 8.414157285275e+00 9.547538541816e+00 1.307676522991e+01 1.424251871761e+01 > > SVD: largest singular values : 5.674169364836e+03 5.711489755811e+03 5.754496402805e+03 5.905177592591e+03 6.057398162681e+03 > > 0 KSP Residual norm 6.537083346419e-01 > > 0 KSP preconditioned resid norm 6.537083346419e-01 true resid norm 2.909228222342e+02 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 4.766960746477e-14 > > 1 KSP preconditioned resid norm 4.766960746477e-14 true resid norm 2.206377081276e-11 ||r(i)||/||b|| 7.584063238254e-14 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: Using full step: fnorm 2.909228222342e+02 gnorm 8.519091421012e+00 > > 2 SNES Function norm 8.519091421012e+00 > > SVD: condition number 8.866866174487e+02, 0 of 4590 singular values are (nearly) zero > > SVD: smallest singular values: 6.754153695550e+00 8.386828076859e+00 9.514891596031e+00 1.302462235577e+01 1.424263827525e+01 > > SVD: largest singular values : 5.671563211881e+03 5.685626194828e+03 5.739548236997e+03 5.833938566429e+03 5.988817694036e+03 > > 0 KSP Residual norm 3.040350335176e-02 > > 0 KSP preconditioned resid norm 3.040350335176e-02 true resid norm 8.519091421012e+00 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 3.285565040836e-15 > > 1 KSP preconditioned resid norm 3.285565040836e-15 true resid norm 9.005553787171e-13 ||r(i)||/||b|| 1.057102611314e-13 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: Using full step: fnorm 8.519091421012e+00 gnorm 3.086711801483e-02 > > 3 SNES Function norm 3.086711801483e-02 > > SVD: condition number 8.863731497353e+02, 0 of 4590 singular values are (nearly) zero > > SVD: smallest singular values: 6.750850248606e+00 8.386167701356e+00 9.513896745592e+00 1.302218359138e+01 1.424165997875e+01 > > SVD: largest singular values : 5.669769588361e+03 5.683785644290e+03 5.736938066065e+03 5.828997334185e+03 5.983772398249e+03 > > 0 KSP Residual norm 3.152844718957e-05 > > 0 KSP preconditioned resid norm 3.152844718957e-05 true resid norm 3.086711801483e-02 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 3.048057745094e-18 > > 1 KSP preconditioned resid norm 3.048057745094e-18 true resid norm 9.890968076924e-16 ||r(i)||/||b|| 3.204370447598e-14 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: Using full step: fnorm 3.086711801483e-02 gnorm 2.527921600901e-07 > > 4 SNES Function norm 2.527921600901e-07 > > Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 4 > > > > I suppose it is possible that the linear system becomes more singular as the grid size increases but I think it will take forever to run SVD on it. I have not tried other PC or KSP methods. Any advice? > > > On Wed, Aug 30, 2017 at 12:37 PM, Matthew Knepley wrote: > On Wed, Aug 30, 2017 at 12:21 AM, zakaryah . wrote: > ?OK, this is very helpful (and not totally clear from the "Why is Newton's method not converging" FAQ at this address: https://scicomp.stackexchange.com/questions/30/why-is-newtons-method-not-converging. Here's the output, using the intermediate grid and the options -snes_type newtonls -snes_monitor -ksp_monitor -ksp_monitor_true_residual -ksp_converged_reason -snes_converged_reason -pc_type lu -snes_linesearch_monitor:? > > 0 SNES Function norm 1.370293318432e+04 > > 0 KSP Residual norm 7.457516441709e+02 > > 0 KSP preconditioned resid norm 7.457516441709e+02 true resid norm 1.370293318432e+04 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 1.244742959480e-10 > > 1 KSP preconditioned resid norm 1.244742959480e-10 true resid norm 9.866304569403e-09 ||r(i)||/||b|| 7.200140609815e-13 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 6.541326653047e+05 > > Line search: Cubic step no good, shrinking lambda, current gnorm 8.963755251044e+04 lambda=5.0000000000000003e-02 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.788913078169e+04 lambda=2.5000000000000001e-02 > > Line search: Cubically determined step, current gnorm 1.340565818376e+04 lambda=1.2500000000000001e-02 > > Your Jacobian is wrong. You solved the Newton linear system, which should be guaranteed to give a residual reduction, but > you get nothing. Thus we conclude that your Jacobian is wrong, giving a wrong descent direction. I would recommend simplifying > the problem until you can check the Jacobian entries by hand. > > Thanks, > > Matt > 1 SNES Function norm 1.340565818376e+04 > > 0 KSP Residual norm 3.837938728414e+02 > > 0 KSP preconditioned resid norm 3.837938728414e+02 true resid norm 1.340565818376e+04 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 3.935624425879e-11 > > 1 KSP preconditioned resid norm 3.935624425879e-11 true resid norm 4.012917478400e-09 ||r(i)||/||b|| 2.993450544086e-13 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 7.752009107979e+05 > > Line search: Cubic step no good, shrinking lambda, current gnorm 9.942192482364e+04 lambda=5.0000000000000003e-02 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.724069009956e+04 lambda=2.5000000000000001e-02 > > Line search: Cubically determined step, current gnorm 1.272263847462e+04 lambda=1.2500000000000001e-02 > > 2 SNES Function norm 1.272263847462e+04 > > 0 KSP Residual norm 8.610741245938e+01 > > 0 KSP preconditioned resid norm 8.610741245938e+01 true resid norm 1.272263847462e+04 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 3.953708398506e-12 > > 1 KSP preconditioned resid norm 3.953708398506e-12 true resid norm 1.259105691297e-09 ||r(i)||/||b|| 9.896576828850e-14 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 1.126216236986e+04 > > Line search: Quadratically determined step, lambda=1.0000000000000001e-01 > > 3 SNES Function norm 1.126216236986e+04 > > 0 KSP Residual norm 6.100581225849e+01 > > 0 KSP preconditioned resid norm 6.100581225849e+01 true resid norm 1.126216236986e+04 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 3.714526144776e-12 > > 1 KSP preconditioned resid norm 3.714526144776e-12 true resid norm 1.150524854441e-09 ||r(i)||/||b|| 1.021584325156e-13 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 9.992994468502e+03 > > Line search: Quadratically determined step, lambda=1.0000000000000001e-01 > > 4 SNES Function norm 9.992994468502e+03 > > 0 KSP Residual norm 5.452156705070e+01 > > 0 KSP preconditioned resid norm 5.452156705070e+01 true resid norm 9.992994468502e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 2.345517221383e-12 > > 1 KSP preconditioned resid norm 2.345517221383e-12 true resid norm 6.645776037775e-10 ||r(i)||/||b|| 6.650435020976e-14 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 8.954956134809e+03 > > Line search: Quadratically determined step, lambda=1.0000000000000001e-01 > > 5 SNES Function norm 8.954956134809e+03 > > 0 KSP Residual norm 1.869342714623e+02 > > 0 KSP preconditioned resid norm 1.869342714623e+02 true resid norm 8.954956134809e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 6.952075013553e-12 > > 1 KSP preconditioned resid norm 6.952075013553e-12 true resid norm 2.332836306210e-09 ||r(i)||/||b|| 2.605078429298e-13 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 9.366608314830e+04 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.702858832608e+04 lambda=5.0000000000000003e-02 > > Line search: Cubic step no good, shrinking lambda, current gnorm 9.141708166107e+03 lambda=2.5000000000000001e-02 > > Line search: Cubically determined step, current gnorm 8.847002105227e+03 lambda=1.2500000000000001e-02 > > 6 SNES Function norm 8.847002105227e+03 > > 0 KSP Residual norm 4.136774315749e+01 > > 0 KSP preconditioned resid norm 4.136774315749e+01 true resid norm 8.847002105227e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 2.340140336645e-12 > > 1 KSP preconditioned resid norm 2.340140336645e-12 true resid norm 6.952836046439e-10 ||r(i)||/||b|| 7.858974106416e-14 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 7.928904942575e+03 > > Line search: Quadratically determined step, lambda=1.0000000000000001e-01 > > 7 SNES Function norm 7.928904942575e+03 > > 0 KSP Residual norm 9.421715655897e+01 > > 0 KSP preconditioned resid norm 9.421715655897e+01 true resid norm 7.928904942575e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 5.662350656880e-12 > > 1 KSP preconditioned resid norm 5.662350656880e-12 true resid norm 1.323447274435e-09 ||r(i)||/||b|| 1.669142566370e-13 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 4.679455846837e+04 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.024301325332e+04 lambda=5.0000000000000003e-02 > > Line search: Cubically determined step, current gnorm 7.882357228991e+03 lambda=2.5000000000000001e-02 > > 8 SNES Function norm 7.882357228991e+03 > > 0 KSP Residual norm 3.101373992455e+01 > > 0 KSP preconditioned resid norm 3.101373992455e+01 true resid norm 7.882357228991e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 1.176793992597e-12 > > 1 KSP preconditioned resid norm 1.176793992597e-12 true resid norm 4.948698048884e-10 ||r(i)||/||b|| 6.278195602050e-14 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: Using full step: fnorm 7.882357228991e+03 gnorm 5.263287196819e+03 > > 9 SNES Function norm 5.263287196819e+03 > > 0 KSP Residual norm 8.541947236232e+00 > > 0 KSP preconditioned resid norm 8.541947236232e+00 true resid norm 5.263287196819e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 1.741221887649e-12 > > 1 KSP preconditioned resid norm 1.741221887649e-12 true resid norm 9.735248249306e-10 ||r(i)||/||b|| 1.849651726242e-13 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 4.836381737060e+03 > > Line search: Quadratically determined step, lambda=1.0000000000000001e-01 > > 10 SNES Function norm 4.836381737060e+03 > > 0 KSP Residual norm 1.926518638579e+01 > > 0 KSP preconditioned resid norm 1.926518638579e+01 true resid norm 4.836381737060e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 1.278653333162e-11 > > 1 KSP preconditioned resid norm 1.278653333162e-11 true resid norm 1.211659620475e-08 ||r(i)||/||b|| 2.505301868938e-12 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 4.550187794983e+03 > > Line search: Quadratically determined step, lambda=1.0000000000000001e-01 > > 11 SNES Function norm 4.550187794983e+03 > > 0 KSP Residual norm 5.492305435155e+00 > > 0 KSP preconditioned resid norm 5.492305435155e+00 true resid norm 4.550187794983e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 3.941029797735e-13 > > 1 KSP preconditioned resid norm 3.941029797735e-13 true resid norm 2.123615113926e-10 ||r(i)||/||b|| 4.667093336823e-14 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 3.210238905481e+03 > > Line search: Quadratically determined step, lambda=4.4544417493894700e-01 > > 12 SNES Function norm 3.210238905481e+03 > > 0 KSP Residual norm 7.578202966222e+00 > > 0 KSP preconditioned resid norm 7.578202966222e+00 true resid norm 3.210238905481e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 3.800509410151e-12 > > 1 KSP preconditioned resid norm 3.800509410151e-12 true resid norm 3.408323024026e-09 ||r(i)||/||b|| 1.061703855812e-12 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 2.924559961092e+03 > > Line search: Quadratically determined step, lambda=1.0000000000000001e-01 > > 13 SNES Function norm 2.924559961092e+03 > > 0 KSP Residual norm 5.076980628354e+00 > > 0 KSP preconditioned resid norm 5.076980628354e+00 true resid norm 2.924559961092e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 1.468522311180e-13 > > 1 KSP preconditioned resid norm 1.468522311180e-13 true resid norm 9.059064389128e-11 ||r(i)||/||b|| 3.097582032733e-14 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 2.389633730237e+03 > > Line search: Quadratically determined step, lambda=2.1257217805235396e-01 > > 14 SNES Function norm 2.389633730237e+03 > > 0 KSP Residual norm 5.166598232781e+00 > > 0 KSP preconditioned resid norm 5.166598232781e+00 true resid norm 2.389633730237e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 2.249406847764e-13 > > 1 KSP preconditioned resid norm 2.249406847764e-13 true resid norm 4.780078098549e-11 ||r(i)||/||b|| 2.000339231098e-14 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 2.093485770827e+03 > > Line search: Quadratically determined step, lambda=1.5993239844479040e-01 > > 15 SNES Function norm 2.093485770827e+03 > > 0 KSP Residual norm 8.445727235216e+00 > > 0 KSP preconditioned resid norm 8.445727235216e+00 true resid norm 2.093485770827e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 1.917214128329e-13 > > 1 KSP preconditioned resid norm 1.917214128329e-13 true resid norm 1.577089911021e-10 ||r(i)||/||b|| 7.533320421842e-14 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 1.967949189728e+03 > > Line search: Quadratically determined step, lambda=1.0000000000000001e-01 > > 16 SNES Function norm 1.967949189728e+03 > > 0 KSP Residual norm 2.548962277036e+01 > > 0 KSP preconditioned resid norm 2.548962277036e+01 true resid norm 1.967949189728e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 4.393171339150e-12 > > 1 KSP preconditioned resid norm 4.393171339150e-12 true resid norm 6.268541493766e-10 ||r(i)||/||b|| 3.185316738097e-13 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 3.445905389608e+03 > > Line search: Cubic step no good, shrinking lambda, current gnorm 2.129006719528e+03 lambda=5.0000000000000003e-02 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.969320033677e+03 lambda=1.8287150237426619e-02 > > Line search: Cubically determined step, current gnorm 1.963119931876e+03 lambda=8.7403113229804347e-03 > > 17 SNES Function norm 1.963119931876e+03 > > 0 KSP Residual norm 1.189834379636e+02 > > 0 KSP preconditioned resid norm 1.189834379636e+02 true resid norm 1.963119931876e+03 ||r(i)||/||b|| 1.000000000000e+00 > > 1 KSP Residual norm 3.213937715681e-11 > > 1 KSP preconditioned resid norm 3.213937715681e-11 true resid norm 8.137767291258e-08 ||r(i)||/||b|| 4.145323553147e-11 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > Line search: gnorm after quadratic fit 1.329932469429e+05 > > Line search: Cubic step no good, shrinking lambda, current gnorm 2.079971894561e+04 lambda=5.0000000000000003e-02 > > Line search: Cubic step no good, shrinking lambda, current gnorm 4.672632539568e+03 lambda=2.5000000000000001e-02 > > Line search: Cubic step no good, shrinking lambda, current gnorm 2.337320012333e+03 lambda=1.2500000000000001e-02 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.989655494060e+03 lambda=3.7908486903933652e-03 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.964061592872e+03 lambda=4.3979444116724767e-04 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963285343925e+03 lambda=9.8760011176842895e-05 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963156689078e+03 lambda=2.3385231527184855e-05 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963128672876e+03 lambda=5.6482781216089357e-06 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963122042807e+03 lambda=1.3692247365910180e-06 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963120443658e+03 lambda=3.3226543340386796e-07 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963120056069e+03 lambda=8.0648305027594934e-08 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963119962021e+03 lambda=1.9576315045380916e-08 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963119939193e+03 lambda=4.7519586695347587e-09 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963119933652e+03 lambda=1.1534950726729967e-09 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963119932307e+03 lambda=2.8000026447574534e-10 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963119931981e+03 lambda=6.7967445495582766e-11 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963119931902e+03 lambda=1.6498100190994443e-11 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963119931882e+03 lambda=4.0046243381676681e-12 > > Line search: Cubic step no good, shrinking lambda, current gnorm 1.963119931878e+03 lambda=9.7209812173761137e-13 > > Line search: unable to find good step length! After 19 tries > > Line search: fnorm=1.9631199318761367e+03, gnorm=1.9631199318776339e+03, ynorm=1.1898343796305618e+02, minlambda=9.9999999999999998e-13, lambda=9.7209812173761137e-13, initial slope=-3.8538398668951751e+06 > > Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 17 > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > From afrah.nacib at gmail.com Sat Sep 2 18:59:56 2017 From: afrah.nacib at gmail.com (Afrah Najib) Date: Sun, 3 Sep 2017 02:59:56 +0300 Subject: [petsc-users] Tracing PETSc codes Message-ID: Hi, How can I trace PETSc codes to better understand what's going on? I installed code::blocks [http://www.codeblocks.org/downloads] but I know that just pressing build/ run with the default options will not help. Can I trace the codes of PETSc ? Which environment you recommend? Thanks a lot -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Sep 2 19:09:23 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 2 Sep 2017 19:09:23 -0500 Subject: [petsc-users] Tracing PETSc codes In-Reply-To: References: Message-ID: <4D17754E-5501-4CD9-B630-59347AE14F76@mcs.anl.gov> What do you mean by trace? The PETSc -log_view option gives a high level view of where the time is being spent, vector operations, preconditioners etc. This is usually all you need to understand where the time is being spent and what might need to be optimized. Barry > On Sep 2, 2017, at 6:59 PM, Afrah Najib wrote: > > Hi, > How can I trace PETSc codes to better understand what's going on? > > I installed code::blocks [http://www.codeblocks.org/downloads] but I know that just pressing build/ run with the default options will not help. > > Can I trace the codes of PETSc ? Which environment you recommend? > > Thanks a lot From bsmith at mcs.anl.gov Sat Sep 2 19:20:25 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 2 Sep 2017 19:20:25 -0500 Subject: [petsc-users] Tracing PETSc codes In-Reply-To: References: <4D17754E-5501-4CD9-B630-59347AE14F76@mcs.anl.gov> Message-ID: <3045CBE0-AE88-42AA-BBBD-266FDBDEF028@mcs.anl.gov> Ahh, you should be able to do this with any debugger like gdb, lldb, or the Intel compiler. Each has separate commands for "watching" or "tracing" variables when they change values. You can use the PETSc command line option -start_in_debugger to run in parallel. Or if you have access to Totalview, it is a parallel graphical debugger. Barry > On Sep 2, 2017, at 7:17 PM, Afrah Najib wrote: > > I mean debugging the code by adding break points etc, so that I can trace how variables are changing at each line of code > > On 3 September 2017 at 03:09, Barry Smith wrote: > > What do you mean by trace? The PETSc -log_view option gives a high level view of where the time is being spent, vector operations, preconditioners etc. This is usually all you need to understand where the time is being spent and what might need to be optimized. > > Barry > > > On Sep 2, 2017, at 6:59 PM, Afrah Najib wrote: > > > > Hi, > > How can I trace PETSc codes to better understand what's going on? > > > > I installed code::blocks [http://www.codeblocks.org/downloads] but I know that just pressing build/ run with the default options will not help. > > > > Can I trace the codes of PETSc ? Which environment you recommend? > > > > Thanks a lot > > From ling.zou at inl.gov Sun Sep 3 12:02:44 2017 From: ling.zou at inl.gov (Zou, Ling) Date: Sun, 3 Sep 2017 11:02:44 -0600 Subject: [petsc-users] PETSc with mesh refinement Message-ID: Hi All, I wonder if there is a good example on how PETSc should work with mesh refinement. I have done some online search/PETSc-user email search, but did not find a good one. Here is my case. Currently, assuming NO mesh refinement (so I know exactly the problem size, mesh connectivity, non-zero pattern of matrices, etc.) In a transient simulation, my PETSc code looks like: ======================================================================== SetupVectors(); // unknown vec, old unknown, etc. SetupResFunc(); // residual function, basically pointing to the discretized PDE SetupMatrices(); // allocate memory for preconditioning matrix based on non-zero pattern (so it is mesh dependent) // I also use finite differencing to get preconditioning matrix, so there is another MatFDColoring object associated with the matrix for (int step = 0; step < max_step; step++) { SNESSolve(); } ======================================================================== This setup works perfectly for my problems. (I understand it is low efficient to use FD to get the preconditioning matrix, but it is not a priority to me at this moment). However, now I would like to do some mesh refinement, so the previous setup would not work, because the problem size, non-zero pattern of the matrix and MatFDColoring change with mesh size (and connectivity). I suppose my code should be changed like this: ======================================================================== SetupVectors(); SetupResFunc(); SetupMatrices(); for (int step = 0; step < max_step; step++) { MeshRefinement(); // I will manage my mesh AdjustVectors(); // resize vector AdjustMatrices(); // resize matrix, re-allocate memory, adjust non-zero pattern, etc. SNESSolve(); } ======================================================================== The issue is that I understand one should not re-allocate memory to both PETSc vector and matrix dynamically during simulation, which has been discussed here: https://scicomp.stackexchange.com/questions/373/is-it-possible-to-dynamically-resize-a-sparse-matrix-in-the-petsc-library So, the final questions are: 1) Any good example on PETSc working with mesh refinement? 2) Any suggestion on my proposed code structure? Best, Ling -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Sep 3 13:44:51 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 3 Sep 2017 14:44:51 -0400 Subject: [petsc-users] PETSc with mesh refinement In-Reply-To: References: Message-ID: On Sun, Sep 3, 2017 at 1:02 PM, Zou, Ling wrote: > Hi All, > > I wonder if there is a good example on how PETSc should work with mesh > refinement. I have done some online search/PETSc-user email search, but did > not find a good one. > > Here is my case. Currently, assuming NO mesh refinement (so I know exactly > the problem size, mesh connectivity, non-zero pattern of matrices, etc.) > In a transient simulation, my PETSc code looks like: > > ======================================================================== > SetupVectors(); // unknown vec, old unknown, etc. > SetupResFunc(); // residual function, basically pointing to the > discretized PDE > SetupMatrices(); // allocate memory for preconditioning matrix based on > non-zero pattern (so it is mesh dependent) > // I also use finite differencing to get > preconditioning matrix, so there is another MatFDColoring object associated > with the matrix > for (int step = 0; step < max_step; step++) > { > SNESSolve(); > } > ======================================================================== > > This setup works perfectly for my problems. > (I understand it is low efficient to use FD to get the preconditioning > matrix, but it is not a priority to me at this moment). > > However, now I would like to do some mesh refinement, so the previous > setup would not work, because the problem size, non-zero pattern of the > matrix and MatFDColoring change with mesh size (and connectivity). > I suppose my code should be changed like this: > > ======================================================================== > SetupVectors(); > SetupResFunc(); > SetupMatrices(); > > for (int step = 0; step < max_step; step++) > { > MeshRefinement(); // I will manage my mesh > > AdjustVectors(); // resize vector > AdjustMatrices(); // resize matrix, re-allocate memory, adjust non-zero > pattern, etc. > > SNESSolve(); > } > ======================================================================== > > The issue is that I understand one should not re-allocate memory to both > PETSc vector and matrix dynamically during simulation, which has been > discussed here: > https://scicomp.stackexchange.com/questions/373/is-it- > possible-to-dynamically-resize-a-sparse-matrix-in-the-petsc-library > > So, the final questions are: > 1) Any good example on PETSc working with mesh refinement? > 2) Any suggestion on my proposed code structure? > 1) No 2) Yes. We advocate using the DM to describe your data layout. This allows all Vec and Mats to be created automatically, and also tells the solver (SNESSetDM, etc.) the sizes and potential decomposition. We are working on a mesh adaptivity interface which will allow adaptivity to be used with all DM problems without any extra code. We should have some examples with this soon. Thanks, Matt > Best, > > Ling > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Sep 3 13:55:01 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 3 Sep 2017 13:55:01 -0500 Subject: [petsc-users] PETSc with mesh refinement In-Reply-To: References: Message-ID: <857EA795-72B9-4528-B3A8-EEC6A402330E@mcs.anl.gov> The TS/SNES/KSP/PC allocate various work vectors and matrices which they keep around for additional solvers. Thus if you want to change the size of your problem and/or number of degrees of freedom per process you need to do one of 2 things 1) destroy the old TS/SNES/ or KSP and create a new one each time you change your vectors to a different size or 2) call TSReset/SNESReset/or KSPReset each time you change your vectors, this causes all the internal work vectors and matrices to be freed so they can get replaced but keeps all the solver options etc. 2) is preferred to 1. Barry > On Sep 3, 2017, at 1:44 PM, Matthew Knepley wrote: > > On Sun, Sep 3, 2017 at 1:02 PM, Zou, Ling wrote: > Hi All, > > I wonder if there is a good example on how PETSc should work with mesh refinement. I have done some online search/PETSc-user email search, but did not find a good one. > > Here is my case. Currently, assuming NO mesh refinement (so I know exactly the problem size, mesh connectivity, non-zero pattern of matrices, etc.) > In a transient simulation, my PETSc code looks like: > > ======================================================================== > SetupVectors(); // unknown vec, old unknown, etc. > SetupResFunc(); // residual function, basically pointing to the discretized PDE > SetupMatrices(); // allocate memory for preconditioning matrix based on non-zero pattern (so it is mesh dependent) > // I also use finite differencing to get preconditioning matrix, so there is another MatFDColoring object associated with the matrix > for (int step = 0; step < max_step; step++) > { > SNESSolve(); > } > ======================================================================== > > This setup works perfectly for my problems. > (I understand it is low efficient to use FD to get the preconditioning matrix, but it is not a priority to me at this moment). > > However, now I would like to do some mesh refinement, so the previous setup would not work, because the problem size, non-zero pattern of the matrix and MatFDColoring change with mesh size (and connectivity). > I suppose my code should be changed like this: > > ======================================================================== > SetupVectors(); > SetupResFunc(); > SetupMatrices(); > > for (int step = 0; step < max_step; step++) > { > MeshRefinement(); // I will manage my mesh > > AdjustVectors(); // resize vector > AdjustMatrices(); // resize matrix, re-allocate memory, adjust non-zero pattern, etc. > > SNESSolve(); > } > ======================================================================== > > The issue is that I understand one should not re-allocate memory to both PETSc vector and matrix dynamically during simulation, which has been discussed here: > https://scicomp.stackexchange.com/questions/373/is-it-possible-to-dynamically-resize-a-sparse-matrix-in-the-petsc-library > > So, the final questions are: > 1) Any good example on PETSc working with mesh refinement? > 2) Any suggestion on my proposed code structure? > > 1) No > > 2) Yes. We advocate using the DM to describe your data layout. This allows all Vec and Mats to be created automatically, and > also tells the solver (SNESSetDM, etc.) the sizes and potential decomposition. > > We are working on a mesh adaptivity interface which will allow adaptivity to be used with all DM problems without any extra > code. We should have some examples with this soon. > > Thanks, > > Matt > > Best, > > Ling > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ From ling.zou at inl.gov Sun Sep 3 15:54:46 2017 From: ling.zou at inl.gov (Zou, Ling) Date: Sun, 3 Sep 2017 14:54:46 -0600 Subject: [petsc-users] PETSc with mesh refinement In-Reply-To: <857EA795-72B9-4528-B3A8-EEC6A402330E@mcs.anl.gov> References: <857EA795-72B9-4528-B3A8-EEC6A402330E@mcs.anl.gov> Message-ID: Thanks both of you. Great suggestions. Ling On Sun, Sep 3, 2017 at 12:55 PM, Barry Smith wrote: > > The TS/SNES/KSP/PC allocate various work vectors and matrices which they > keep around for additional solvers. Thus if you want to change the size of > your problem and/or number of degrees of freedom per process you need to do > one of 2 things > > 1) destroy the old TS/SNES/ or KSP and create a new one each time you > change your vectors to a different size or > > 2) call TSReset/SNESReset/or KSPReset each time you change your > vectors, this causes all the internal work vectors and matrices to be freed > so they can get replaced but keeps all the solver options etc. > > 2) is preferred to 1. > > Barry > > > On Sep 3, 2017, at 1:44 PM, Matthew Knepley wrote: > > > > On Sun, Sep 3, 2017 at 1:02 PM, Zou, Ling wrote: > > Hi All, > > > > I wonder if there is a good example on how PETSc should work with mesh > refinement. I have done some online search/PETSc-user email search, but did > not find a good one. > > > > Here is my case. Currently, assuming NO mesh refinement (so I know > exactly the problem size, mesh connectivity, non-zero pattern of matrices, > etc.) > > In a transient simulation, my PETSc code looks like: > > > > ======================================================================== > > SetupVectors(); // unknown vec, old unknown, etc. > > SetupResFunc(); // residual function, basically pointing to the > discretized PDE > > SetupMatrices(); // allocate memory for preconditioning matrix based on > non-zero pattern (so it is mesh dependent) > > // I also use finite differencing to get > preconditioning matrix, so there is another MatFDColoring object associated > with the matrix > > for (int step = 0; step < max_step; step++) > > { > > SNESSolve(); > > } > > ======================================================================== > > > > This setup works perfectly for my problems. > > (I understand it is low efficient to use FD to get the preconditioning > matrix, but it is not a priority to me at this moment). > > > > However, now I would like to do some mesh refinement, so the previous > setup would not work, because the problem size, non-zero pattern of the > matrix and MatFDColoring change with mesh size (and connectivity). > > I suppose my code should be changed like this: > > > > ======================================================================== > > SetupVectors(); > > SetupResFunc(); > > SetupMatrices(); > > > > for (int step = 0; step < max_step; step++) > > { > > MeshRefinement(); // I will manage my mesh > > > > AdjustVectors(); // resize vector > > AdjustMatrices(); // resize matrix, re-allocate memory, adjust > non-zero pattern, etc. > > > > SNESSolve(); > > } > > ======================================================================== > > > > The issue is that I understand one should not re-allocate memory to both > PETSc vector and matrix dynamically during simulation, which has been > discussed here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ > scicomp.stackexchange.com_questions_373_is-2Dit- > 2Dpossible-2Dto-2Ddynamically-2Dresize-2Da-2Dsparse- > 2Dmatrix-2Din-2Dthe-2Dpetsc-2Dlibrary&d=DwIFAg&c= > 54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r= > kuHHom1yjd94zUrBWecnYg&m=lEB-2RrZ7_xUBhwaeQnezW0fuTVkBbMZVcRtx8Fn > tz0&s=6JH1-QnmGFstCWaXbdRsDRn-lSHYB-gqGOlsCFyhDVQ&e= > > > > So, the final questions are: > > 1) Any good example on PETSc working with mesh refinement? > > 2) Any suggestion on my proposed code structure? > > > > 1) No > > > > 2) Yes. We advocate using the DM to describe your data layout. This > allows all Vec and Mats to be created automatically, and > > also tells the solver (SNESSetDM, etc.) the sizes and potential > decomposition. > > > > We are working on a mesh adaptivity interface which will allow > adaptivity to be used with all DM problems without any extra > > code. We should have some examples with this soon. > > > > Thanks, > > > > Matt > > > > Best, > > > > Ling > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www. > caam.rice.edu_-7Emk51_&d=DwIFAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > _aEkJFOKJFd00&r=kuHHom1yjd94zUrBWecnYg&m=lEB-2RrZ7_ > xUBhwaeQnezW0fuTVkBbMZVcRtx8Fntz0&s=URW-NglPj9JSNGOxqvL8qv9WilvxNzmYIw > 8A6IEn5tg&e= > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sun Sep 3 13:52:37 2017 From: jed at jedbrown.org (Jed Brown) Date: Sun, 03 Sep 2017 12:52:37 -0600 Subject: [petsc-users] Red-black block Jacobi in PETSc? In-Reply-To: References: <871snthgt6.fsf@jedbrown.org> Message-ID: <87d1778vii.fsf@jedbrown.org> (Moving this thread to petsc-dev.) PETSc factorization has an ordering PCFactorSetMatOrderingType. There exists a MatColoring class that can be extended by the user so I think I would emulate factorization by specifying a coloring. One potential difference is that a "coloring" relevant to SOR might be useful without being a valid coloring. (I.e., the colors might not actually be independent.) The MatSOR implementations support many variants already so it may take some time to get sufficiently familiar with the code to add this nontrivial feature. Ben Yee writes: > I am interested in implementing it in the SOR framework. My ultimate goal > is to find something that performs well for my problem, and I think I have > enough evidence from my simple 1D codes that it should work. If I'm going > to assess performance, I think it would be good to implement it reasonably > optimally. > > The manual page for PCSOR says that it is "Not a true parallel SOR, in > parallel this implementation corresponds to block Jacobi with SOR on each > block". If I'm reading this correctly, then that description sounds like > what I want to do except that it is missing the coloring scheme -- is that > correct? If that is correct, then is there a way to define a block size on > a MPIAIJ matrix, or would I have to use a BAIJ matrix? (With BJACOBI, > there's the set total number of blocks function for defining the block > sizes. Is there something equivalent for PCSOR?) > > Also, could you provide a brief overview of how I should go about altering > SOR to do the coloring? Thanks! > > > > On Tue, Aug 29, 2017 at 11:35 PM, Barry Smith wrote: > >> >> > On Aug 29, 2017, at 10:31 PM, Jed Brown wrote: >> > >> > Barry Smith writes: >> > >> >> Ok, you should be using the BAIJ matrix it supports point-block Jacobi >> and point-block Gauss-Seidel. >> >> >> >> We do not have a red-black Jacobi/Gauss-Seidel but you could easily >> add it. You will add it, not by using a any shell objects but by adding >> functionality to the PCPBJACOBI code in PETSc which is in >> src/ksp/pc/impls/pbjacobi/pbjacobi.c >> >> >> >> First you will need to add a routine to supply the colors (make it >> general for any number of colors since the code is basically the same as >> for only two colors) call it say >> >> >> >> PCPBJacobiSetColors(PC pc,PetscInt number of colors, PetscInt >> *sizeofeachcolor, PetscInt **idsforeachcolor); >> >> >> >> you will have to add additional fields in PC_PBJacobi to contain this >> information. >> >> >> >> Then copy the static PetscErrorCode PCApply_PBJacobi_N(PC pc,Vec x,Vec >> y) for your block size N (for example 3) and modify it to do what you >> want, so for example >> >> >> >> static PetscErrorCode PCApply_PBJacobi_2_Color(PC pc,Vec x,Vec y) >> >> { >> >> PC_PBJacobi *jac = (PC_PBJacobi*)pc->data; >> >> PetscErrorCode ierr; >> >> PetscInt i,m = jac->mbs; >> >> const MatScalar *diag = jac->diag; >> >> PetscScalar x0,x1,*yy; >> >> const PetscScalar *xx; >> >> >> >> PetscFunctionBegin; >> >> if (!jac->b) { >> >> ierr = VecDuplicate(x,&jac->b); >> >> ierr = VecDuplicate(x,&jac->work); >> >> } >> >> >> >> ierr = VecCopy(x,b);CHKERRQ(ierr); >> >> for (j=0; jnumberofcolors; j++) { >> >> >> >> ierr = VecGetArrayRead(b,&xx);CHKERRQ(ierr); >> >> ierr = VecGetArray(y,&yy);CHKERRQ(ierr); >> >> >> >> for (i=0; isizeofeachcolor[j]; i++) { >> >> ii = jac->idsforeachcolor[j][i]; >> >> diag = jac->diag + 4*ii; >> >> x0 = xx[2*ii]; x1 = xx[2*ii+1]; >> >> yy[2*ii] = diag[0]*x0 + diag[2]*x1; >> >> yy[2*ii+1] = diag[1]*x0 + diag[3]*x1; >> >> } >> >> ierr = VecRestoreArrayRead(b,&xx);CHKERRQ(ierr); >> >> ierr = VecRestoreArray(y,&yy);CHKERRQ(ierr); >> >> >> >> /* update residual */ >> >> if (i < jac->sizeofeachcolor[j]-1) { >> >> ierr = MatMult(pc->matrix,y,work2); >> >> ierr = VecAYPX(b,-1,work1); >> >> } >> >> } >> > >> > Why this way instead of adding the functionality to SOR? This global >> > residual update is horribly inefficient compared to merely computing the >> > residual on each color. >> >> True, but if this is for theoretical studies of convergence and not >> optimal performance it is easier. For example you do this and see the >> convergence is much better with coloring than without and then conclude it >> is worth spending the time to do it properly within an SOR framework. >> >> Barry >> >> > >> >> PetscFunctionReturn(0); >> >> } >> >> >> >> Finally in PCSetUp_PBJacobi() you would set the apply function pointer >> to the "color" version if the user has provided the coloring information. >> >> >> >> Pretty simple. >> >> >> >> Barry >> >> >> >> >> >>> On Aug 29, 2017, at 6:47 PM, Ben Yee wrote: >> >>> >> >>> I'm solving a coupled set of equations, so each "block" corresponds to >> a set of unknowns for a particular spatial cell. The matrix is structured >> such that all of the unknowns for a given spatial cell have adjacent global >> matrix indices (i.e., they're next to each other in the global solution >> vector). Effectively, I want to do red-black Gauss Seidel, but with >> blocks. Alternatively, it's the same as applying block Jacobi for all the >> red cells and then applying block Jacobi for all the black cells. >> >>> >> >>> The color of the block is determined from the geometry of the problem >> which is stored in various structures in the code I'm working on, >> independent of petsc. (Physically, I generally have a nice 3d cartesian >> spatial grid and the coloring is just a checkerboard in that case.) >> >>> >> >>> The reason I want to do this is for research purposes. I've >> implemented my own interpolation matrices for PCMG, and, in my simpler 1d >> codes and convergence analyses, I've found that doing a red-black smoothing >> significantly improved convergence for my particular problem (though I'm >> aware that this generally leads to poor cache efficiency). >> >>> >> >>> On Aug 29, 2017 7:33 PM, "Barry Smith" wrote: >> >>> >> >>> Ben, >> >>> >> >>> Please explain more what you mean by "a red-black block Jacobi >> smoother". What is your matrix structure? What are the blocks? How do you >> decide which ones are which color? Why do you wish to use some a smoother. >> >>> >> >>> Barry >> >>> >> >>>> On Aug 29, 2017, at 6:19 PM, Ben Yee wrote: >> >>>> >> >>>> Hi, >> >>>> >> >>>> For the smoother in PCMG, I want to use a red-black block Jacobi >> smoother. Is this available with the existing PETSc options? If so, how >> do I designate which blocks are red and which are black? >> >>>> >> >>>> If it is not possible, what would be the best way to implement this? >> Would I use KSPRICHARDSON+PCSHELL? >> >>>> >> >>>> Thanks! >> >>>> >> >>>> -- >> >>>> Ben Yee >> >>>> >> >>>> NERS PhD Candidate, University of Michigan >> >>>> B.S. Mech. & Nuclear Eng., U.C. Berkeley >> >>> >> >>> >> >> > > > -- > Ben Yee > > NERS PhD Candidate, University of Michigan > B.S. Mech. & Nuclear Eng., U.C. Berkeley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From zakaryah at gmail.com Mon Sep 4 15:48:47 2017 From: zakaryah at gmail.com (zakaryah .) Date: Mon, 4 Sep 2017 16:48:47 -0400 Subject: [petsc-users] A number of questions about DMDA with SNES and Quasi-Newton methods In-Reply-To: <876F0ACF-62C0-4DC9-B203-2893B25E3938@mcs.anl.gov> References: <020DAED9-AAAD-4741-976B-BB2EF6C79D3F@mcs.anl.gov> <5BF530F6-8D79-47E6-B2C5-B0702FF2AA59@mcs.anl.gov> <7EE84495-4346-4BFB-B9AD-BCAA3C722461@mcs.anl.gov> <08AA1273-062D-4FDD-8709-40AA1FE8AA92@mcs.anl.gov> <06932E72-E8F3-4EA8-889F-A570AD660E32@mcs.anl.gov> <98D45588-381F-44FB-9D20-65D1638E357F@mcs.anl.gov> <876F0ACF-62C0-4DC9-B203-2893B25E3938@mcs.anl.gov> Message-ID: One piece of information that would be useful is what ordering PETSc uses for the Jacobian in the snes_test_display. Is it a natural ordering, or the PETSc ordering? For debugging the Jacobian manually, the natural ordering is much easier to work with. For -n 1, are the orderings the same? If I use a MatStencil r to represent a field with 3 degrees of freedom, and the dimensions of my 3D DMDA are MxNxP, which row of the Jacobian corresponds to r.i=x, r.j=y, r.k=z, r.c=f? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Sep 4 15:58:24 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 4 Sep 2017 15:58:24 -0500 Subject: [petsc-users] A number of questions about DMDA with SNES and Quasi-Newton methods In-Reply-To: References: <020DAED9-AAAD-4741-976B-BB2EF6C79D3F@mcs.anl.gov> <5BF530F6-8D79-47E6-B2C5-B0702FF2AA59@mcs.anl.gov> <7EE84495-4346-4BFB-B9AD-BCAA3C722461@mcs.anl.gov> <08AA1273-062D-4FDD-8709-40AA1FE8AA92@mcs.anl.gov> <06932E72-E8F3-4EA8-889F-A570AD660E32@mcs.anl.gov> <98D45588-381F-44FB-9D20-65D1638E357F@mcs.anl.gov> <876F0ACF-62C0-4DC9-B203-2893B25E3938@mcs.anl.gov> Message-ID: <1D94CC99-2395-4BBE-A0F3-80D23D85FA45@mcs.anl.gov> > On Sep 4, 2017, at 3:48 PM, zakaryah . wrote: > > One piece of information that would be useful is what ordering PETSc uses for the Jacobian in the snes_test_display. Is it a natural ordering, or the PETSc ordering? For debugging the Jacobian manually, the natural ordering is much easier to work with. What is displayed is always the natural ordering (internally it is not the natural ordering). > For -n 1, are the orderings the same? yes > > If I use a MatStencil r to represent a field with 3 degrees of freedom, and the dimensions of my 3D DMDA are MxNxP, which row of the Jacobian corresponds to r.i=x, r.j=y, r.k=z, r.c=f? Internally it is complicated but for any viewing it is just the natural ordering and all the degrees of freedom for a single point are next to each other in the vector/matrix. From zakaryah at gmail.com Mon Sep 4 16:09:39 2017 From: zakaryah at gmail.com (zakaryah .) Date: Mon, 4 Sep 2017 17:09:39 -0400 Subject: [petsc-users] A number of questions about DMDA with SNES and Quasi-Newton methods In-Reply-To: <1D94CC99-2395-4BBE-A0F3-80D23D85FA45@mcs.anl.gov> References: <020DAED9-AAAD-4741-976B-BB2EF6C79D3F@mcs.anl.gov> <5BF530F6-8D79-47E6-B2C5-B0702FF2AA59@mcs.anl.gov> <7EE84495-4346-4BFB-B9AD-BCAA3C722461@mcs.anl.gov> <08AA1273-062D-4FDD-8709-40AA1FE8AA92@mcs.anl.gov> <06932E72-E8F3-4EA8-889F-A570AD660E32@mcs.anl.gov> <98D45588-381F-44FB-9D20-65D1638E357F@mcs.anl.gov> <876F0ACF-62C0-4DC9-B203-2893B25E3938@mcs.anl.gov> <1D94CC99-2395-4BBE-A0F3-80D23D85FA45@mcs.anl.gov> Message-ID: OK that is super helpful. Just to be sure - for MxNxP, the row r in the Jacobian is at r.i*P*N*3 + r.j*P*3 + r.k*3 + r.c? On Mon, Sep 4, 2017 at 4:58 PM, Barry Smith wrote: > > > On Sep 4, 2017, at 3:48 PM, zakaryah . wrote: > > > > One piece of information that would be useful is what ordering PETSc > uses for the Jacobian in the snes_test_display. Is it a natural ordering, > or the PETSc ordering? For debugging the Jacobian manually, the natural > ordering is much easier to work with. > > What is displayed is always the natural ordering (internally it is not > the natural ordering). > > > For -n 1, are the orderings the same? > > yes > > > > > > > If I use a MatStencil r to represent a field with 3 degrees of freedom, > and the dimensions of my 3D DMDA are MxNxP, which row of the Jacobian > corresponds to r.i=x, r.j=y, r.k=z, r.c=f? > > Internally it is complicated but for any viewing it is just the natural > ordering and all the degrees of freedom for a single point are next to each > other in the vector/matrix. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Sep 4 16:33:25 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 4 Sep 2017 16:33:25 -0500 Subject: [petsc-users] A number of questions about DMDA with SNES and Quasi-Newton methods In-Reply-To: References: <020DAED9-AAAD-4741-976B-BB2EF6C79D3F@mcs.anl.gov> <5BF530F6-8D79-47E6-B2C5-B0702FF2AA59@mcs.anl.gov> <7EE84495-4346-4BFB-B9AD-BCAA3C722461@mcs.anl.gov> <08AA1273-062D-4FDD-8709-40AA1FE8AA92@mcs.anl.gov> <06932E72-E8F3-4EA8-889F-A570AD660E32@mcs.anl.gov> <98D45588-381F-44FB-9D20-65D1638E357F@mcs.anl.gov> <876F0ACF-62C0-4DC9-B203-2893B25E3938@mcs.anl.gov> <1D94CC99-2395-4BBE-A0F3-80D23D85FA45@mcs.anl.gov> Message-ID: <66C5A502-B1F1-47FB-80CA-57708ECA613F@mcs.anl.gov> > On Sep 4, 2017, at 4:09 PM, zakaryah . wrote: > > OK that is super helpful. Just to be sure - for MxNxP, the row r in the Jacobian is at r.i*P*N*3 + r.j*P*3 + r.k*3 + r.c? It is that way, or the other way around r.k*M*N*3 + r.j*N*3 + r.k*3 + r.c > > > On Mon, Sep 4, 2017 at 4:58 PM, Barry Smith wrote: > > > On Sep 4, 2017, at 3:48 PM, zakaryah . wrote: > > > > One piece of information that would be useful is what ordering PETSc uses for the Jacobian in the snes_test_display. Is it a natural ordering, or the PETSc ordering? For debugging the Jacobian manually, the natural ordering is much easier to work with. > > What is displayed is always the natural ordering (internally it is not the natural ordering). > > > For -n 1, are the orderings the same? > > yes > > > > > > > If I use a MatStencil r to represent a field with 3 degrees of freedom, and the dimensions of my 3D DMDA are MxNxP, which row of the Jacobian corresponds to r.i=x, r.j=y, r.k=z, r.c=f? > > Internally it is complicated but for any viewing it is just the natural ordering and all the degrees of freedom for a single point are next to each other in the vector/matrix. > > From zakaryah at gmail.com Mon Sep 4 22:26:36 2017 From: zakaryah at gmail.com (zakaryah .) Date: Mon, 4 Sep 2017 23:26:36 -0400 Subject: [petsc-users] A number of questions about DMDA with SNES and Quasi-Newton methods In-Reply-To: References: <020DAED9-AAAD-4741-976B-BB2EF6C79D3F@mcs.anl.gov> <5BF530F6-8D79-47E6-B2C5-B0702FF2AA59@mcs.anl.gov> <7EE84495-4346-4BFB-B9AD-BCAA3C722461@mcs.anl.gov> <08AA1273-062D-4FDD-8709-40AA1FE8AA92@mcs.anl.gov> <06932E72-E8F3-4EA8-889F-A570AD660E32@mcs.anl.gov> <98D45588-381F-44FB-9D20-65D1638E357F@mcs.anl.gov> <876F0ACF-62C0-4DC9-B203-2893B25E3938@mcs.anl.gov> <1D94CC99-2395-4BBE-A0F3-80D23D85FA45@mcs.anl.gov> <66C5A502-B1F1-47FB-80CA-57708ECA613F@mcs.anl.gov> Message-ID: OK - I've checked the Jacobian and function very thoroughly and I am confident there are no errors. I suspect that I am having problems with a bad starting point, and the SNES cannot find the global minimum from there. I know that the global minimum (with residual zero) exists in all cases but I would like the methods for finding it to be more robust to the starting value. The problem comes from the physics of finite deformations of elastic materials. In short, I have a functional of smooth maps on a 3D domain to itself. The functional contains two terms. The first term represents forces which come from external data, and in the Lagrangian this term only involves the values of the map at the point in question. The second term penalizes fluctuations in the map, and can take various forms. The simplest form is just the Dirichlet energy, but I'm also interested in the infinitesimal strain energy and the finite strain energy. The first two have terms in the Lagrangian which are first order in the second spatial derivatives of the map, while the third (finite strain energy) has terms which are up to third order in the first and second spatial derivatives of the map. It is the finite strain energy term which has been problematic. The Euler-Lagrange equations are discretized on a cubic grid, with equal interval spacing in each dimension. The map is the dependent variable, i.e. the x in F(x) = 0. I prefer Neumann boundary conditions. Because the spatial derivatives of the map are usually small, the Jacobian typically has large values in 3x3 blocks along the diagonal (which depend on the map and the external data), and up to 54 values which are functions of the spatial derivatives of the map and tend to be smaller. Do you have any advice on diagnosing and improving situations in which Newton's method finds a stationary point that is not the state with globally minimal residual? One hint is that -snes_type newtonls does not find as good a solution as -snes_type newtontr but I don't know much about these trust region methods, or how to customize and assess them. I'm grateful for any advice. On Mon, Sep 4, 2017 at 5:44 PM, zakaryah . wrote: > Yes, it looks like it IS the other way around, and I think the row is > > r.c + r.i*3 + r.j*3*M + r.k*3*M*N, where r.i is in [0,M-1], r.j is in > [0,N-1], and r.k is in [0,P-1]. > > That matches the boundary conditions in the displayed Jacobian. > > On Mon, Sep 4, 2017 at 5:33 PM, Barry Smith wrote: > >> >> > On Sep 4, 2017, at 4:09 PM, zakaryah . wrote: >> > >> > OK that is super helpful. Just to be sure - for MxNxP, the row r in >> the Jacobian is at r.i*P*N*3 + r.j*P*3 + r.k*3 + r.c? >> >> It is that way, or the other way around r.k*M*N*3 + r.j*N*3 + r.k*3 + >> r.c >> > >> > >> > On Mon, Sep 4, 2017 at 4:58 PM, Barry Smith wrote: >> > >> > > On Sep 4, 2017, at 3:48 PM, zakaryah . wrote: >> > > >> > > One piece of information that would be useful is what ordering PETSc >> uses for the Jacobian in the snes_test_display. Is it a natural ordering, >> or the PETSc ordering? For debugging the Jacobian manually, the natural >> ordering is much easier to work with. >> > >> > What is displayed is always the natural ordering (internally it is >> not the natural ordering). >> > >> > > For -n 1, are the orderings the same? >> > >> > yes >> > >> > >> > >> > > >> > > If I use a MatStencil r to represent a field with 3 degrees of >> freedom, and the dimensions of my 3D DMDA are MxNxP, which row of the >> Jacobian corresponds to r.i=x, r.j=y, r.k=z, r.c=f? >> > >> > Internally it is complicated but for any viewing it is just the natural >> ordering and all the degrees of freedom for a single point are next to each >> other in the vector/matrix. >> > >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davydden at gmail.com Tue Sep 5 07:26:02 2017 From: davydden at gmail.com (Denis Davydov) Date: Tue, 5 Sep 2017 14:26:02 +0200 Subject: [petsc-users] Elemental support is outdated? Message-ID: <544DEC8D-ED6E-4DF9-A3DC-FF5BAED22DE7@gmail.com> Dear developers, I just tried compiling PETSc with Elemental 0.87.7 and got compiler errors (see below) which suggests that the interface in PETSc is outdated. Looking at the ?elemental.py? https://bitbucket.org/petsc/petsc/src/90564b43f6b05485163c147b464b5d6d28cde3ef/config/BuildSystem/config/packages/elemental.py?at=hongzh%2Fswe&fileviewer=file-view-default I see that the suggested Elemental's commit is from Nov 2015 https://github.com/elemental/Elemental/commit/e81005dc3af9b331d43e16e4d221e889378e2c49 whereas 0.87.7 is from Feb. 2017; There are 8 releases in between. Regards, Denis. === petsc-3.7.6/src/mat/impls/elemental/matelem.cxx:653:7: error: no matching function for call to 'SolveAfter' El::lu::SolveAfter(El::NORMAL,*a->emat,*a->pivot,xer); ^~~~~~~~~~~~~~~~~~ elemental-0.87.7-arf6zkytd5mmyckplcrjiin3zapbhvyt/include/El/lapack_like/factor.hpp:552:6: note: candidate function [with Field = double] not viable: no known conversion from 'El::DistMatrix' (aka 'DistMatrix') to 'const El::DistPermutation' for 3rd argument -------------- next part -------------- An HTML attachment was scrubbed... URL: From rupp at iue.tuwien.ac.at Tue Sep 5 08:10:05 2017 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Tue, 5 Sep 2017 08:10:05 -0500 Subject: [petsc-users] Elemental support is outdated? In-Reply-To: <544DEC8D-ED6E-4DF9-A3DC-FF5BAED22DE7@gmail.com> References: <544DEC8D-ED6E-4DF9-A3DC-FF5BAED22DE7@gmail.com> Message-ID: Hi Denis, > I just tried compiling PETSc with Elemental 0.87.7 and got compiler > errors (see below) which suggests that the interface in PETSc is outdated. > Looking at the ?elemental.py? > https://bitbucket.org/petsc/petsc/src/90564b43f6b05485163c147b464b5d6d28cde3ef/config/BuildSystem/config/packages/elemental.py?at=hongzh%2Fswe&fileviewer=file-view-default > > > I see that the suggested Elemental's commit is from Nov 2015 > https://github.com/elemental/Elemental/commit/e81005dc3af9b331d43e16e4d221e889378e2c49 > > whereas 0.87.7 is from Feb. 2017; There are 8 releases in between. the master branch of PETSc uses version 0.87.4 from November 2016: https://bitbucket.org/petsc/petsc/src/743f4829774bac6ed514af01aa0f24f18a42afc0/config/BuildSystem/config/packages/elemental.py?at=master&fileviewer=file-view-default Best regards, Karli From davydden at gmail.com Tue Sep 5 08:12:51 2017 From: davydden at gmail.com (Denis Davydov) Date: Tue, 5 Sep 2017 15:12:51 +0200 Subject: [petsc-users] Elemental support is outdated? In-Reply-To: References: <544DEC8D-ED6E-4DF9-A3DC-FF5BAED22DE7@gmail.com> Message-ID: <142ECA51-92FA-4D94-943F-517F91C0B64E@gmail.com> Hi Karl, Thanks for the prompt reply, I did not notice that I was not looking at the master branch. I will try with 0.87.4... It would still be good if PETSc supports the most recent Elemental. Regards, Denis. > On 5 Sep 2017, at 15:10, Karl Rupp wrote: > > Hi Denis, > >> I just tried compiling PETSc with Elemental 0.87.7 and got compiler errors (see below) which suggests that the interface in PETSc is outdated. >> Looking at the ?elemental.py? https://bitbucket.org/petsc/petsc/src/90564b43f6b05485163c147b464b5d6d28cde3ef/config/BuildSystem/config/packages/elemental.py?at=hongzh%2Fswe&fileviewer=file-view-default I see that the suggested Elemental's commit is from Nov 2015 https://github.com/elemental/Elemental/commit/e81005dc3af9b331d43e16e4d221e889378e2c49 whereas 0.87.7 is from Feb. 2017; There are 8 releases in between. > > the master branch of PETSc uses version 0.87.4 from November 2016: > https://bitbucket.org/petsc/petsc/src/743f4829774bac6ed514af01aa0f24f18a42afc0/config/BuildSystem/config/packages/elemental.py?at=master&fileviewer=file-view-default > > Best regards, > Karli From jed at jedbrown.org Tue Sep 5 08:39:13 2017 From: jed at jedbrown.org (Jed Brown) Date: Tue, 05 Sep 2017 07:39:13 -0600 Subject: [petsc-users] A number of questions about DMDA with SNES and Quasi-Newton methods In-Reply-To: References: <08AA1273-062D-4FDD-8709-40AA1FE8AA92@mcs.anl.gov> <06932E72-E8F3-4EA8-889F-A570AD660E32@mcs.anl.gov> <98D45588-381F-44FB-9D20-65D1638E357F@mcs.anl.gov> <876F0ACF-62C0-4DC9-B203-2893B25E3938@mcs.anl.gov> <1D94CC99-2395-4BBE-A0F3-80D23D85FA45@mcs.anl.gov> <66C5A502-B1F1-47FB-80CA-57708ECA613F@mcs.anl.gov> Message-ID: <87o9qp5kou.fsf@jedbrown.org> "zakaryah ." writes: > OK - I've checked the Jacobian and function very thoroughly and I am > confident there are no errors. Does Newton converge quadratically when you have a good enough initial guess? Globalization of large deformation elasticity is a persistent engineering challenge. The standard approach is to use a continuation, often in the form of load increments. Regarding trust region documentation, the man page says The basic algorithm is taken from "The Minpack Project", by More', Sorensen, Garbow, Hillstrom, pages 88-111 of "Sources and Development of Mathematical Software", Wayne Cowell, editor. You should be able to make sense of it reading from any other source on trust region methods. > I suspect that I am having problems with a bad starting point, and the SNES > cannot find the global minimum from there. I know that the global minimum > (with residual zero) exists in all cases but I would like the methods for > finding it to be more robust to the starting value. > > The problem comes from the physics of finite deformations of elastic > materials. In short, I have a functional of smooth maps on a 3D domain to > itself. The functional contains two terms. The first term represents > forces which come from external data, and in the Lagrangian this term only > involves the values of the map at the point in question. The second term > penalizes fluctuations in the map, and can take various forms. The > simplest form is just the Dirichlet energy, but I'm also interested in the > infinitesimal strain energy and the finite strain energy. The first two > have terms in the Lagrangian which are first order in the second spatial > derivatives of the map, while the third (finite strain energy) has terms > which are up to third order in the first and second spatial derivatives of > the map. It is the finite strain energy term which has been problematic. > > The Euler-Lagrange equations are discretized on a cubic grid, with equal > interval spacing in each dimension. The map is the dependent variable, > i.e. the x in F(x) = 0. I prefer Neumann boundary conditions. Because the > spatial derivatives of the map are usually small, the Jacobian typically > has large values in 3x3 blocks along the diagonal (which depend on the map > and the external data), and up to 54 values which are functions of the > spatial derivatives of the map and tend to be smaller. > > Do you have any advice on diagnosing and improving situations in which > Newton's method finds a stationary point that is not the state with > globally minimal residual? One hint is that -snes_type newtonls does not > find as good a solution as -snes_type newtontr but I don't know much about > these trust region methods, or how to customize and assess them. I'm > grateful for any advice. > > On Mon, Sep 4, 2017 at 5:44 PM, zakaryah . wrote: > >> Yes, it looks like it IS the other way around, and I think the row is >> >> r.c + r.i*3 + r.j*3*M + r.k*3*M*N, where r.i is in [0,M-1], r.j is in >> [0,N-1], and r.k is in [0,P-1]. >> >> That matches the boundary conditions in the displayed Jacobian. >> >> On Mon, Sep 4, 2017 at 5:33 PM, Barry Smith wrote: >> >>> >>> > On Sep 4, 2017, at 4:09 PM, zakaryah . wrote: >>> > >>> > OK that is super helpful. Just to be sure - for MxNxP, the row r in >>> the Jacobian is at r.i*P*N*3 + r.j*P*3 + r.k*3 + r.c? >>> >>> It is that way, or the other way around r.k*M*N*3 + r.j*N*3 + r.k*3 + >>> r.c >>> > >>> > >>> > On Mon, Sep 4, 2017 at 4:58 PM, Barry Smith wrote: >>> > >>> > > On Sep 4, 2017, at 3:48 PM, zakaryah . wrote: >>> > > >>> > > One piece of information that would be useful is what ordering PETSc >>> uses for the Jacobian in the snes_test_display. Is it a natural ordering, >>> or the PETSc ordering? For debugging the Jacobian manually, the natural >>> ordering is much easier to work with. >>> > >>> > What is displayed is always the natural ordering (internally it is >>> not the natural ordering). >>> > >>> > > For -n 1, are the orderings the same? >>> > >>> > yes >>> > >>> > >>> > >>> > > >>> > > If I use a MatStencil r to represent a field with 3 degrees of >>> freedom, and the dimensions of my 3D DMDA are MxNxP, which row of the >>> Jacobian corresponds to r.i=x, r.j=y, r.k=z, r.c=f? >>> > >>> > Internally it is complicated but for any viewing it is just the natural >>> ordering and all the degrees of freedom for a single point are next to each >>> other in the vector/matrix. >>> > >>> > >>> >>> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From rupp at iue.tuwien.ac.at Tue Sep 5 09:41:26 2017 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Tue, 5 Sep 2017 09:41:26 -0500 Subject: [petsc-users] Elemental support is outdated? In-Reply-To: <142ECA51-92FA-4D94-943F-517F91C0B64E@gmail.com> References: <544DEC8D-ED6E-4DF9-A3DC-FF5BAED22DE7@gmail.com> <142ECA51-92FA-4D94-943F-517F91C0B64E@gmail.com> Message-ID: Hi Denis, > Thanks for the prompt reply, > I did not notice that I was not looking at the master branch. > I will try with 0.87.4... > > It would still be good if PETSc supports the most recent Elemental. you are right. I checked the latest Elemental release and the upgrade was without problems. It is now merged to branch next for testing: https://bitbucket.org/petsc/petsc/commits/ce8ca746282174cc1e4f5a02c6a306c44d99f4f9 Thanks and best regards, Karli > >> On 5 Sep 2017, at 15:10, Karl Rupp wrote: >> >> Hi Denis, >> >>> I just tried compiling PETSc with Elemental 0.87.7 and got compiler errors (see below) which suggests that the interface in PETSc is outdated. >>> Looking at the ?elemental.py? https://bitbucket.org/petsc/petsc/src/90564b43f6b05485163c147b464b5d6d28cde3ef/config/BuildSystem/config/packages/elemental.py?at=hongzh%2Fswe&fileviewer=file-view-default I see that the suggested Elemental's commit is from Nov 2015 https://github.com/elemental/Elemental/commit/e81005dc3af9b331d43e16e4d221e889378e2c49 whereas 0.87.7 is from Feb. 2017; There are 8 releases in between. >> >> the master branch of PETSc uses version 0.87.4 from November 2016: >> https://bitbucket.org/petsc/petsc/src/743f4829774bac6ed514af01aa0f24f18a42afc0/config/BuildSystem/config/packages/elemental.py?at=master&fileviewer=file-view-default >> >> Best regards, >> Karli > From balay at mcs.anl.gov Tue Sep 5 09:43:30 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 5 Sep 2017 09:43:30 -0500 Subject: [petsc-users] Elemental support is outdated? In-Reply-To: <142ECA51-92FA-4D94-943F-517F91C0B64E@gmail.com> References: <544DEC8D-ED6E-4DF9-A3DC-FF5BAED22DE7@gmail.com> <142ECA51-92FA-4D94-943F-517F91C0B64E@gmail.com> Message-ID: You could try (with petsc master): --download-elemental-commit=v0.87.7 Latest elemental master appears to have major changes [including C++14 reqirement] - so that might require more work.. Satish On Tue, 5 Sep 2017, Denis Davydov wrote: > Hi Karl, > > Thanks for the prompt reply, > I did not notice that I was not looking at the master branch. > I will try with 0.87.4... > > It would still be good if PETSc supports the most recent Elemental. > > Regards, > Denis. > > > On 5 Sep 2017, at 15:10, Karl Rupp wrote: > > > > Hi Denis, > > > >> I just tried compiling PETSc with Elemental 0.87.7 and got compiler errors (see below) which suggests that the interface in PETSc is outdated. > >> Looking at the ?elemental.py? https://bitbucket.org/petsc/petsc/src/90564b43f6b05485163c147b464b5d6d28cde3ef/config/BuildSystem/config/packages/elemental.py?at=hongzh%2Fswe&fileviewer=file-view-default I see that the suggested Elemental's commit is from Nov 2015 https://github.com/elemental/Elemental/commit/e81005dc3af9b331d43e16e4d221e889378e2c49 whereas 0.87.7 is from Feb. 2017; There are 8 releases in between. > > > > the master branch of PETSc uses version 0.87.4 from November 2016: > > https://bitbucket.org/petsc/petsc/src/743f4829774bac6ed514af01aa0f24f18a42afc0/config/BuildSystem/config/packages/elemental.py?at=master&fileviewer=file-view-default > > > > Best regards, > > Karli > > From davydden at gmail.com Tue Sep 5 09:49:35 2017 From: davydden at gmail.com (Denis Davydov) Date: Tue, 5 Sep 2017 16:49:35 +0200 Subject: [petsc-users] Elemental support is outdated? In-Reply-To: References: <544DEC8D-ED6E-4DF9-A3DC-FF5BAED22DE7@gmail.com> <142ECA51-92FA-4D94-943F-517F91C0B64E@gmail.com> Message-ID: I already tried building PETSc with 0.87.7, the error is: petsc-3.7.6/src/mat/impls/elemental/matelem.cxx:653:7: error: no matching function for call to 'SolveAfter' El::lu::SolveAfter(El::NORMAL,*a->emat,*a->pivot,xer); ^~~~~~~~~~~~~~~~~~ elemental-0.87.7-arf6zkytd5mmyckplcrjiin3zapbhvyt/include/El/lapack_like/factor.hpp:552:6: note: candidate function [with Field = double] not viable: no known conversion from 'El::DistMatrix' (aka 'DistMatrix') to 'const El::DistPermutation' for 3rd argument strangely enough I have the same issues with 0.87.4: /petsc-3.7.6/src/mat/impls/elemental/matelem.cxx:653:7: error: no matching function for call to 'SolveAfter' El::lu::SolveAfter(El::NORMAL,*a->emat,*a->pivot,xer); ^~~~~~~~~~~~~~~~~~ /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:549:6: note: candidate function [with F = double] not viable: no known conversion from 'El::DistMatrix' (aka 'DistMatrix') to 'const El::DistPermutation' for 3rd argument void SolveAfter ^ /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:543:6: note: candidate template ignored: could not match 'Matrix' against 'DistMatrix' void SolveAfter ^ /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:530:6: note: candidate function template not viable: requires 3 arguments, but 4 were provided void SolveAfter ^ /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:535:6: note: candidate function template not viable: requires 3 arguments, but 4 were provided void SolveAfter ^ /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:558:6: note: candidate function template not viable: requires 5 arguments, but 4 were provided void SolveAfter ^ /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:565:6: note: candidate function template not viable: requires 5 arguments, but 4 were provided void SolveAfter ^ Regards, Denis > On 5 Sep 2017, at 16:43, Satish Balay wrote: > > You could try (with petsc master): > > --download-elemental-commit=v0.87.7 > > Latest elemental master appears to have major changes [including C++14 > reqirement] - so that might require more work.. > > Satish > > On Tue, 5 Sep 2017, Denis Davydov wrote: > >> Hi Karl, >> >> Thanks for the prompt reply, >> I did not notice that I was not looking at the master branch. >> I will try with 0.87.4... >> >> It would still be good if PETSc supports the most recent Elemental. >> >> Regards, >> Denis. >> >>> On 5 Sep 2017, at 15:10, Karl Rupp wrote: >>> >>> Hi Denis, >>> >>>> I just tried compiling PETSc with Elemental 0.87.7 and got compiler errors (see below) which suggests that the interface in PETSc is outdated. >>>> Looking at the ?elemental.py? https://bitbucket.org/petsc/petsc/src/90564b43f6b05485163c147b464b5d6d28cde3ef/config/BuildSystem/config/packages/elemental.py?at=hongzh%2Fswe&fileviewer=file-view-default I see that the suggested Elemental's commit is from Nov 2015 https://github.com/elemental/Elemental/commit/e81005dc3af9b331d43e16e4d221e889378e2c49 whereas 0.87.7 is from Feb. 2017; There are 8 releases in between. >>> >>> the master branch of PETSc uses version 0.87.4 from November 2016: >>> https://bitbucket.org/petsc/petsc/src/743f4829774bac6ed514af01aa0f24f18a42afc0/config/BuildSystem/config/packages/elemental.py?at=master&fileviewer=file-view-default >>> >>> Best regards, >>> Karli >> >> From balay at mcs.anl.gov Tue Sep 5 09:52:14 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 5 Sep 2017 09:52:14 -0500 Subject: [petsc-users] Elemental support is outdated? In-Reply-To: References: <544DEC8D-ED6E-4DF9-A3DC-FF5BAED22DE7@gmail.com> <142ECA51-92FA-4D94-943F-517F91C0B64E@gmail.com> Message-ID: You need 'master' branch from petsc git repo for this to work - not petsc-3.7.6 Satish On Tue, 5 Sep 2017, Denis Davydov wrote: > I already tried building PETSc with 0.87.7, the error is: > > petsc-3.7.6/src/mat/impls/elemental/matelem.cxx:653:7: error: no matching function for call to 'SolveAfter' > El::lu::SolveAfter(El::NORMAL,*a->emat,*a->pivot,xer); > ^~~~~~~~~~~~~~~~~~ > elemental-0.87.7-arf6zkytd5mmyckplcrjiin3zapbhvyt/include/El/lapack_like/factor.hpp:552:6: note: candidate function [with Field = double] not viable: no known conversion from 'El::DistMatrix' (aka 'DistMatrix') to 'const El::DistPermutation' for 3rd argument > > > strangely enough I have the same issues with 0.87.4: > > > /petsc-3.7.6/src/mat/impls/elemental/matelem.cxx:653:7: error: no matching function for call to 'SolveAfter' > El::lu::SolveAfter(El::NORMAL,*a->emat,*a->pivot,xer); > ^~~~~~~~~~~~~~~~~~ > /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:549:6: note: candidate function [with F = double] not viable: no known conversion from 'El::DistMatrix' (aka 'DistMatrix') to 'const El::DistPermutation' for 3rd argument > void SolveAfter > ^ > /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:543:6: note: candidate template ignored: could not match 'Matrix' against 'DistMatrix' > void SolveAfter > ^ > /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:530:6: note: candidate function template not viable: requires 3 arguments, but 4 were provided > void SolveAfter > ^ > /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:535:6: note: candidate function template not viable: requires 3 arguments, but 4 were provided > void SolveAfter > ^ > /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:558:6: note: candidate function template not viable: requires 5 arguments, but 4 were provided > void SolveAfter > ^ > /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:565:6: note: candidate function template not viable: requires 5 arguments, but 4 were provided > void SolveAfter > ^ > > Regards, > Denis > > > On 5 Sep 2017, at 16:43, Satish Balay wrote: > > > > You could try (with petsc master): > > > > --download-elemental-commit=v0.87.7 > > > > Latest elemental master appears to have major changes [including C++14 > > reqirement] - so that might require more work.. > > > > Satish > > > > On Tue, 5 Sep 2017, Denis Davydov wrote: > > > >> Hi Karl, > >> > >> Thanks for the prompt reply, > >> I did not notice that I was not looking at the master branch. > >> I will try with 0.87.4... > >> > >> It would still be good if PETSc supports the most recent Elemental. > >> > >> Regards, > >> Denis. > >> > >>> On 5 Sep 2017, at 15:10, Karl Rupp wrote: > >>> > >>> Hi Denis, > >>> > >>>> I just tried compiling PETSc with Elemental 0.87.7 and got compiler errors (see below) which suggests that the interface in PETSc is outdated. > >>>> Looking at the ?elemental.py? https://bitbucket.org/petsc/petsc/src/90564b43f6b05485163c147b464b5d6d28cde3ef/config/BuildSystem/config/packages/elemental.py?at=hongzh%2Fswe&fileviewer=file-view-default I see that the suggested Elemental's commit is from Nov 2015 https://github.com/elemental/Elemental/commit/e81005dc3af9b331d43e16e4d221e889378e2c49 whereas 0.87.7 is from Feb. 2017; There are 8 releases in between. > >>> > >>> the master branch of PETSc uses version 0.87.4 from November 2016: > >>> https://bitbucket.org/petsc/petsc/src/743f4829774bac6ed514af01aa0f24f18a42afc0/config/BuildSystem/config/packages/elemental.py?at=master&fileviewer=file-view-default > >>> > >>> Best regards, > >>> Karli > >> > >> > > From davydden at gmail.com Tue Sep 5 10:53:22 2017 From: davydden at gmail.com (Denis Davydov) Date: Tue, 5 Sep 2017 17:53:22 +0200 Subject: [petsc-users] Elemental support is outdated? In-Reply-To: References: <544DEC8D-ED6E-4DF9-A3DC-FF5BAED22DE7@gmail.com> <142ECA51-92FA-4D94-943F-517F91C0B64E@gmail.com> Message-ID: <87B5CBFD-3568-49A9-B814-78F93F3E5CC2@gmail.com> Thanks, did not notice that. I can confirm that PETSc builds with Elemental 0.87.7 from master branch on macOS. Regards, Denis. > On 5 Sep 2017, at 16:52, Satish Balay wrote: > > You need 'master' branch from petsc git repo for this to work - not petsc-3.7.6 > > Satish > > On Tue, 5 Sep 2017, Denis Davydov wrote: > >> I already tried building PETSc with 0.87.7, the error is: >> >> petsc-3.7.6/src/mat/impls/elemental/matelem.cxx:653:7: error: no matching function for call to 'SolveAfter' >> El::lu::SolveAfter(El::NORMAL,*a->emat,*a->pivot,xer); >> ^~~~~~~~~~~~~~~~~~ >> elemental-0.87.7-arf6zkytd5mmyckplcrjiin3zapbhvyt/include/El/lapack_like/factor.hpp:552:6: note: candidate function [with Field = double] not viable: no known conversion from 'El::DistMatrix' (aka 'DistMatrix') to 'const El::DistPermutation' for 3rd argument >> >> >> strangely enough I have the same issues with 0.87.4: >> >> >> /petsc-3.7.6/src/mat/impls/elemental/matelem.cxx:653:7: error: no matching function for call to 'SolveAfter' >> El::lu::SolveAfter(El::NORMAL,*a->emat,*a->pivot,xer); >> ^~~~~~~~~~~~~~~~~~ >> /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:549:6: note: candidate function [with F = double] not viable: no known conversion from 'El::DistMatrix' (aka 'DistMatrix') to 'const El::DistPermutation' for 3rd argument >> void SolveAfter >> ^ >> /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:543:6: note: candidate template ignored: could not match 'Matrix' against 'DistMatrix' >> void SolveAfter >> ^ >> /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:530:6: note: candidate function template not viable: requires 3 arguments, but 4 were provided >> void SolveAfter >> ^ >> /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:535:6: note: candidate function template not viable: requires 3 arguments, but 4 were provided >> void SolveAfter >> ^ >> /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:558:6: note: candidate function template not viable: requires 5 arguments, but 4 were provided >> void SolveAfter >> ^ >> /elemental-0.87.4-xas7fy5wlvgtr2a4i7kqm52metc73tqk/include/El/lapack_like/factor.hpp:565:6: note: candidate function template not viable: requires 5 arguments, but 4 were provided >> void SolveAfter >> ^ >> >> Regards, >> Denis >> >>> On 5 Sep 2017, at 16:43, Satish Balay wrote: >>> >>> You could try (with petsc master): >>> >>> --download-elemental-commit=v0.87.7 >>> >>> Latest elemental master appears to have major changes [including C++14 >>> reqirement] - so that might require more work.. >>> >>> Satish >>> >>> On Tue, 5 Sep 2017, Denis Davydov wrote: >>> >>>> Hi Karl, >>>> >>>> Thanks for the prompt reply, >>>> I did not notice that I was not looking at the master branch. >>>> I will try with 0.87.4... >>>> >>>> It would still be good if PETSc supports the most recent Elemental. >>>> >>>> Regards, >>>> Denis. >>>> >>>>> On 5 Sep 2017, at 15:10, Karl Rupp wrote: >>>>> >>>>> Hi Denis, >>>>> >>>>>> I just tried compiling PETSc with Elemental 0.87.7 and got compiler errors (see below) which suggests that the interface in PETSc is outdated. >>>>>> Looking at the ?elemental.py? https://bitbucket.org/petsc/petsc/src/90564b43f6b05485163c147b464b5d6d28cde3ef/config/BuildSystem/config/packages/elemental.py?at=hongzh%2Fswe&fileviewer=file-view-default I see that the suggested Elemental's commit is from Nov 2015 https://github.com/elemental/Elemental/commit/e81005dc3af9b331d43e16e4d221e889378e2c49 whereas 0.87.7 is from Feb. 2017; There are 8 releases in between. >>>>> >>>>> the master branch of PETSc uses version 0.87.4 from November 2016: >>>>> https://bitbucket.org/petsc/petsc/src/743f4829774bac6ed514af01aa0f24f18a42afc0/config/BuildSystem/config/packages/elemental.py?at=master&fileviewer=file-view-default >>>>> >>>>> Best regards, >>>>> Karli >>>> >>>> >> >> From barna.becsek at artorg.unibe.ch Tue Sep 5 11:18:36 2017 From: barna.becsek at artorg.unibe.ch (barna.becsek at artorg.unibe.ch) Date: Tue, 5 Sep 2017 16:18:36 +0000 Subject: [petsc-users] CUDA Dense Matrix Type Message-ID: <001FF40C-50CC-4439-B804-58FDDC6AFB62@artorg.unibe.ch> Hello, I was wondering whether there was an analogous type in petsc to MatDense for CUDA. If there isn?t, what would be the best way to represent a dense matrix with CUDA types? Sincerely, Barna -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 5 12:04:07 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 5 Sep 2017 13:04:07 -0400 Subject: [petsc-users] CUDA Dense Matrix Type In-Reply-To: <001FF40C-50CC-4439-B804-58FDDC6AFB62@artorg.unibe.ch> References: <001FF40C-50CC-4439-B804-58FDDC6AFB62@artorg.unibe.ch> Message-ID: On Tue, Sep 5, 2017 at 12:18 PM, wrote: > Hello, > > I was wondering whether there was an analogous type in petsc to MatDense > for CUDA. If there isn?t, what would be the best way to represent a dense > matrix with CUDA types? > We have not added dense support for CUDA. We would happily accept a contribution in this area, but we have received no other requests for the functionality. Thanks, Matt > Sincerely, > > Barna > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Wed Sep 6 13:29:59 2017 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Wed, 6 Sep 2017 19:29:59 +0100 Subject: [petsc-users] Pulling submeshes out of a DMPlex object Message-ID: Dear all, for various reasons I'd like, given an existing DMPlex mesh, to be able to pull submeshes out. These might take various forms: 1. The locally visible part of a distributed mesh (so that I can discretise and solve independent problems on each local part) 2. Some subset of entities of a given codimension (for example, the boundary of a domain, or the skeleton of the mesh). So I think conceptually what I want is: Create a DMLabel that marks entities of a given codimension. Say DMPlexCreateSubmesh(dm, label, value, height, comm, &subdm) This should walk all entities of the specified height and gather them if they are labelled with the provided value, then the subdm is created by treating those entities as the cells of a new mesh and adding their transitive closure. There appears to be some submesh functionality in plex right now, but it doesn't appear to do what I want: DMPlexCreateSubmesh - Extract a hypersurface from the mesh using vertices defined by a label DMPlexFilter - Extract a subset of mesh cells defined by a label as a separate mesh The latter looks close to what I want for 1, except the change of communicator. The former seems somewhat odd. I would have thought the way to extract a hypersurface would be to mark the faces of the hypersurface and build a mesh as a I described above. It looks like it should be possible to reuse some of the existing submesh routines to do what I want to do, but I'm a little lost. What I tried was just adding an option to DMPlexCreateSubmeshGeneric_Interpolated to mark a submesh by traversing the closure of points marked with a given value. But that swiftly ran into problems because the resulting submesh didn't have depth strata as expected, and looking at the existing code I'm quite confused. The subpointMap that a submesh DM has labels the points of the parents with their depth (in the original mesh?). But this seems to be used to determine the depth of the points in the submesh. Which seems odd to me. I realise this is all rather woolly right now, perhaps in attempting to clarify any queries I will explain better what I want to do. Cheers, Lawrence From knepley at gmail.com Wed Sep 6 14:58:35 2017 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 6 Sep 2017 15:58:35 -0400 Subject: [petsc-users] Pulling submeshes out of a DMPlex object In-Reply-To: References: Message-ID: On Wed, Sep 6, 2017 at 2:29 PM, Lawrence Mitchell < lawrence.mitchell at imperial.ac.uk> wrote: > Dear all, > > for various reasons I'd like, given an existing DMPlex mesh, to be > able to pull submeshes out. These might take various forms: > > 1. The locally visible part of a distributed mesh (so that I can > discretise and solve independent problems on each local part) > This is already a standalone Plex. If you delete the PetscSF, it never knows anyone else is out there :) > 2. Some subset of entities of a given codimension (for example, the > boundary of a domain, or the skeleton of the mesh). > There is a "depth" label in a Plex (DMPlexGetDepthLabel) that does this. > So I think conceptually what I want is: > > Create a DMLabel that marks entities of a given codimension. > > Say > > DMPlexCreateSubmesh(dm, label, value, height, comm, &subdm) > > This should walk all entities of the specified height and gather them > if they are labelled with the provided value, then the subdm is > created by treating those entities as the cells of a new mesh and > adding their transitive closure. > Yes, this makes sense. I also tend to include a map from the submesh back to the original. > There appears to be some submesh functionality in plex right now, but > it doesn't appear to do what I want: > > DMPlexCreateSubmesh - Extract a hypersurface from the mesh using > vertices defined by a label > DMPlexFilter - Extract a subset of mesh cells defined by a label as > a separate mesh > > The latter looks close to what I want for 1, except the change of > communicator. > Can you explain this problem? I think this does what you want. > The former seems somewhat odd. I would have thought the way to > extract a hypersurface would be to mark the faces of the hypersurface > and build a mesh as a I described above. > It just shows you that everyone wants something different. Engineers and scientists tend to only know about cells and vertices, so they always want to the interface to use those. Earthquake people want to extract faults, but they also want to cells attached to either side to make things understandable. > It looks like it should be possible to reuse some of the existing > submesh routines to do what I want to do, but I'm a little lost. > > What I tried was just adding an option to > DMPlexCreateSubmeshGeneric_Interpolated to mark a submesh by > traversing the closure of points marked with a given value. > It should do that automatically. Did you try it as is? Maybe you could give me a small Plex and show me why its giving something wrong. Thanks, Matt > But that swiftly ran into problems because the resulting submesh > didn't have depth strata as expected, and looking at the existing code > I'm quite confused. The subpointMap that a submesh DM has labels the > points of the parents with their depth (in the original mesh?). But > this seems to be used to determine the depth of the points in the > submesh. Which seems odd to me. > > I realise this is all rather woolly right now, perhaps in attempting > to clarify any queries I will explain better what I want to do. > > Cheers, > > Lawrence > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Wed Sep 6 15:25:28 2017 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Wed, 6 Sep 2017 21:25:28 +0100 Subject: [petsc-users] Pulling submeshes out of a DMPlex object In-Reply-To: References: Message-ID: On 6 Sep 2017, at 20:58, Matthew Knepley wrote: >> 1. The locally visible part of a distributed mesh (so that I can >> discretise and solve independent problems on each local part) > > This is already a standalone Plex. If you delete the PetscSF, it never knows anyone else is out there :) So I'd Dmclone and then delete the point SF. Do I not also need to replace the communicator on the DM so that VecCreate DTRT. I'll follow up to the rest with some examples probably next week. Lawrence -------------- next part -------------- An HTML attachment was scrubbed... URL: From giuntoli1991 at gmail.com Fri Sep 8 06:10:41 2017 From: giuntoli1991 at gmail.com (Guido Giuntoli) Date: Fri, 8 Sep 2017 13:10:41 +0200 Subject: [petsc-users] Zero diagonal jacobian Message-ID: Hi I want to solve a problem with a matrix of this kind K 1 J = 1 0 that represents an elastic problem with Lagrange multipliers, this system is really small so I want to try a direct solver so I put -pc_type lu and it fails, probably because of the 0 diagonal. Is there any way, using petsc, to solve this in direct way ? like doing a shifting of columns or something like that. Thank you, Guido. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Sep 8 07:13:09 2017 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 8 Sep 2017 08:13:09 -0400 Subject: [petsc-users] Zero diagonal jacobian In-Reply-To: References: Message-ID: On Fri, Sep 8, 2017 at 7:10 AM, Guido Giuntoli wrote: > Hi I want to solve a problem with a matrix of this kind > > K 1 > J = > 1 0 > > that represents an elastic problem with Lagrange multipliers, this system > is really small so I want to try a direct solver so I put -pc_type lu and > it fails, probably because of the 0 diagonal. Is there any way, using > petsc, to solve this in direct way ? like doing a shifting of columns or > something like that. > 1) You can use SuperLU, which has pivoting. 2) You can use PCFIELDSPLIT, giving the block split above, with type schur and direct solves on each block. 3) You can use reordering, *-pc_factor_nonzeros_along_diagonal* Thanks, Matt > Thank you, Guido. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Sun Sep 10 01:39:46 2017 From: cpraveen at gmail.com (Praveen C) Date: Sun, 10 Sep 2017 12:09:46 +0530 Subject: [petsc-users] Example for solving system of pde with snes Message-ID: Dear all I am solving 3 coupled first order PDE in 2-D with SNES. The stencil is (3+1+3) in both directions due to weno scheme. I want to use matrix-free gmres. For preconditioner, I will use first order scheme with stencil (1+1+1) in both directions. I am using DMDA. Two things that I am looking for are: How to setup the matrix ? Since I have 3 variables at each grid point, how to fill the jacobian matrix. Is there an existing example that comes close to this situation. My search was not fruitful. Thank you praveen -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Sep 10 07:06:04 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 10 Sep 2017 08:06:04 -0400 Subject: [petsc-users] Example for solving system of pde with snes In-Reply-To: References: Message-ID: On Sun, Sep 10, 2017 at 2:39 AM, Praveen C wrote: > Dear all > > I am solving 3 coupled first order PDE in 2-D with SNES. The stencil is > (3+1+3) in both directions due to weno scheme. I want to use matrix-free > gmres. For preconditioner, I will use first order scheme with stencil > (1+1+1) in both directions. I am using DMDA. > > Two things that I am looking for are: > > How to setup the matrix ? > > Since I have 3 variables at each grid point, how to fill the jacobian > matrix. > > Is there an existing example that comes close to this situation. My search > was not fruitful. > No. You want 2 DAs, 1 for evaluating residuals with s = 3, and one for making a Jacobian with s = 1. You give these two DAs to http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/DMDASNESSetFunctionLocal.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/DMDASNESSetJacobianLocal.html and I think it should work as you want. Its too bad we cannot handle this case with SNESSetDM() since it only takes a single DM and there is no way to specialize the Jacobian. Thanks, Matt Thank you > praveen > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Sun Sep 10 07:29:26 2017 From: cpraveen at gmail.com (Praveen C) Date: Sun, 10 Sep 2017 17:59:26 +0530 Subject: [petsc-users] Example for solving system of pde with snes In-Reply-To: References: Message-ID: Hello Matt > No. You want 2 DAs, 1 for evaluating residuals with s = 3, and one for > making a Jacobian with s = 1. You give these > two DAs to > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/ > DMDASNESSetFunctionLocal.html > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/ > DMDASNESSetJacobianLocal.html > > and I think it should work as you want. Its too bad we cannot handle this > case with SNESSetDM() since it only takes > a single DM and there is no way to specialize the Jacobian. > > ok, I was on wrong track trying to use SNESSetFunction and SNESSetJacobian which wouldnt have worked. I still have some doubt. If da0 --> for residual da1 --> for jacobian I still need to call SNESetDM. So here I pass da0 ? But then how will snes know about da1 ? Thanks praveen -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sun Sep 10 07:43:22 2017 From: jed at jedbrown.org (Jed Brown) Date: Sun, 10 Sep 2017 06:43:22 -0600 Subject: [petsc-users] Example for solving system of pde with snes In-Reply-To: References: Message-ID: <87377uzpud.fsf@jedbrown.org> Praveen C writes: > Dear all > > I am solving 3 coupled first order PDE in 2-D with SNES. The stencil is > (3+1+3) in both directions due to weno scheme. I want to use matrix-free > gmres. For preconditioner, I will use first order scheme with stencil > (1+1+1) in both directions. I am using DMDA. > > Two things that I am looking for are: > > How to setup the matrix ? > > Since I have 3 variables at each grid point, how to fill the jacobian > matrix. I would recommend using a MatSetValuesBlocked interface. For a finite difference method, you'd typically set one block row at a time. So you'll create a 3x15 array and call MatSetValuesBlockedStencil indicating the stencils for the one block row and five block columns. > Is there an existing example that comes close to this situation. My search > was not fruitful. There are finite element examples, like SNES ex48.c, but I'm not sure if there is a SNES FD example. > Thank you > praveen -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From jed at jedbrown.org Sun Sep 10 08:07:39 2017 From: jed at jedbrown.org (Jed Brown) Date: Sun, 10 Sep 2017 07:07:39 -0600 Subject: [petsc-users] Example for solving system of pde with snes In-Reply-To: References: Message-ID: <87zia2ya5g.fsf@jedbrown.org> Praveen C writes: > Hello Matt > > >> No. You want 2 DAs, 1 for evaluating residuals with s = 3, and one for >> making a Jacobian with s = 1. You give these >> two DAs to >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/ >> DMDASNESSetFunctionLocal.html >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/ >> DMDASNESSetJacobianLocal.html >> >> and I think it should work as you want. Its too bad we cannot handle this >> case with SNESSetDM() since it only takes >> a single DM and there is no way to specialize the Jacobian. >> >> > ok, I was on wrong track trying to use SNESSetFunction and SNESSetJacobian > which wouldnt have worked. You can use SNESSetJacobian, but DMSNESSetJacobianLocal is more convenient in my opinion. > I still have some doubt. If > > da0 --> for residual > da1 --> for jacobian > > I still need to call SNESetDM. So here I pass da0 ? But then how will snes > know about da1 ? This should work. DMCreateMatrix(da1, &Jpre); SNESSetDM(snes, da0); SNESSetJacobian(snes, NULL, Jpre, NULL, NULL); DMDASNESSetJacobianLocal(da0, func, ctx); Your func will assemble into the Pmat argument. You might run with -snes_mf_operator to apply the action of the true Jacobian while preconditioning with the preconditioning matrix. See also src/snes/examples/tutorials/ex15.c and its handling of "-alloc_star". -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From cpraveen at gmail.com Sun Sep 10 08:15:31 2017 From: cpraveen at gmail.com (Praveen C) Date: Sun, 10 Sep 2017 18:45:31 +0530 Subject: [petsc-users] Example for solving system of pde with snes In-Reply-To: <87zia2ya5g.fsf@jedbrown.org> References: <87zia2ya5g.fsf@jedbrown.org> Message-ID: > > > You can use SNESSetJacobian, but DMSNESSetJacobianLocal is more > convenient in my opinion. > > > I still have some doubt. If > > > > da0 --> for residual > > da1 --> for jacobian > > > > I still need to call SNESetDM. So here I pass da0 ? But then how will > snes > > know about da1 ? > > This should work. > > DMCreateMatrix(da1, &Jpre); > SNESSetDM(snes, da0); > SNESSetJacobian(snes, NULL, Jpre, NULL, NULL); > DMDASNESSetJacobianLocal(da0, func, ctx); > > > In the last line above, should it not be da1 ? Thanks a lot for the examples. Best praveen -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Sep 10 08:19:44 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 10 Sep 2017 09:19:44 -0400 Subject: [petsc-users] Example for solving system of pde with snes In-Reply-To: References: <87zia2ya5g.fsf@jedbrown.org> Message-ID: On Sun, Sep 10, 2017 at 9:15 AM, Praveen C wrote: > > >> >> You can use SNESSetJacobian, but DMSNESSetJacobianLocal is more >> convenient in my opinion. >> >> > I still have some doubt. If >> > >> > da0 --> for residual >> > da1 --> for jacobian >> > >> > I still need to call SNESetDM. So here I pass da0 ? But then how will >> snes >> > know about da1 ? >> >> This should work. >> >> DMCreateMatrix(da1, &Jpre); >> SNESSetDM(snes, da0); >> SNESSetJacobian(snes, NULL, Jpre, NULL, NULL); >> DMDASNESSetJacobianLocal(da0, func, ctx); >> >> >> In the last line above, should it not be da1 ? > No, Jed is saying that using a DA with a wider stencil on that Jacobian is alright as long as you do not try to input values outside the more restricted sparsity pattern. Thanks, Matt > Thanks a lot for the examples. > > Best > praveen > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sun Sep 10 08:29:45 2017 From: jed at jedbrown.org (Jed Brown) Date: Sun, 10 Sep 2017 07:29:45 -0600 Subject: [petsc-users] Example for solving system of pde with snes In-Reply-To: References: <87zia2ya5g.fsf@jedbrown.org> Message-ID: <87wp56y94m.fsf@jedbrown.org> Matthew Knepley writes: > On Sun, Sep 10, 2017 at 9:15 AM, Praveen C wrote: > >> >> >>> >>> You can use SNESSetJacobian, but DMSNESSetJacobianLocal is more >>> convenient in my opinion. >>> >>> > I still have some doubt. If >>> > >>> > da0 --> for residual >>> > da1 --> for jacobian >>> > >>> > I still need to call SNESetDM. So here I pass da0 ? But then how will >>> snes >>> > know about da1 ? >>> >>> This should work. >>> >>> DMCreateMatrix(da1, &Jpre); >>> SNESSetDM(snes, da0); >>> SNESSetJacobian(snes, NULL, Jpre, NULL, NULL); >>> DMDASNESSetJacobianLocal(da0, func, ctx); >>> >>> >>> In the last line above, should it not be da1 ? >> > > No, Jed is saying that using a DA with a wider stencil on that Jacobian is > alright as long as > you do not try to input values outside the more restricted sparsity pattern. Yeah, SNES only knows about one DM and will use it for communication, etc. An alternative is to implement a function for SNESSetJacobian -- if you do this, you need to get the local work space and do communication explicitly. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From davegwebb10 at gmail.com Sun Sep 10 16:51:26 2017 From: davegwebb10 at gmail.com (David Gross) Date: Sun, 10 Sep 2017 23:51:26 +0200 Subject: [petsc-users] Matrix dot product Message-ID: Hello, I was wondering if there was a matrix equivalent to the vecDot function (Frobenius inner product)? As far as I can tell the closest thing is MatNorm with NORM_FROBENIUS, but obviously this is acting on only one matrix. If there is not a built in function, what is the best way to compute this? I am working fortran90. Regards, Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Sep 10 19:19:18 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 10 Sep 2017 20:19:18 -0400 Subject: [petsc-users] Matrix dot product In-Reply-To: References: Message-ID: On Sun, Sep 10, 2017 at 5:51 PM, David Gross wrote: > Hello, > I was wondering if there was a matrix equivalent to the vecDot function > (Frobenius inner product)? As far as I can tell the closest thing is > MatNorm with NORM_FROBENIUS, but obviously this is acting on only one > matrix. > > If there is not a built in function, what is the best way to compute this? > I am working fortran90. > We do not have this. However, it would be trivial to add since we have http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatAXPY.html since you just replace + with * in our code. You could argue that we should have written for a general ring, but C makes this cumbersome. Do you think you could make the change? What are you using this for? Thanks, Matt > Regards, > Dave > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Sep 11 11:16:53 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 11 Sep 2017 12:16:53 -0400 Subject: [petsc-users] possibility of short term subcontract work with PETSc Message-ID: PETSc folks, Argonne National Laboratory has recently set up a system that may make it possible for the PETSc group at ANL to subcontract particular PETSc contribution projects to developers in most of the world. These could be from one week projects to months. If you have particular projects in mind, for example "add feature XXX to package YYY in PETSc" please contact me directly bsmith at mcs.anl.gov with an outline of the idea, how it could help many other PETSc users and how long you think it make take. And we can iterate to see if it is something appropriate to fund. Barry From ryantmorley7298 at gmail.com Mon Sep 11 11:21:30 2017 From: ryantmorley7298 at gmail.com (Ryan Morley) Date: Mon, 11 Sep 2017 11:21:30 -0500 Subject: [petsc-users] [PETSc] How to read a Vector and Matrix from an ASCII file into PETSc? Message-ID: Dear all, I am working on a program in Petsc that will read a vector and matrix from an ASCII file and store it into a vector and matrix object in Petsc so I can eventually solve. It needs to be an ASCII file because I will be outputting the vector and matrix to an ASCII file from code in Delphi. I am starting out trying to accomplish printing a vector to an ASCII file and then reading the vector back into Petsc by reading from the ASCII file. I was able to successfully print a vector to an ASCII file using the following code: PetscViewerASCIIOpen(PETSC_COMM_WORLD, "RyansVector.output", &viewer); VecView(u, viewer); Where "PETSC_COMM_WORLD" is the MPI communicator, "RyansVector.output" is the name of the ASCII file, and "viewer" is the name of the PetscViewer to use with the specified file. Also "u" is a vector object in Petsc. However, now when I want to read from an ASCII file and store the values into a vector I cannot seem to accomplish this. Looking around online I cannot find any evidence that it is possible to read from an ASCII file to produce a vector object in Petsc. Is it only possible to read into Petsc objects with Binary and HDF5 file types? Is it possible to use VecLoad when reading? If so, how? I was not able to get VecLoad to work. Thank you so much! Ryan Morley -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Mon Sep 11 12:04:42 2017 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 11 Sep 2017 20:04:42 +0300 Subject: [petsc-users] [PETSc] How to read a Vector and Matrix from an ASCII file into PETSc? In-Reply-To: References: Message-ID: On 11 September 2017 at 19:21, Ryan Morley wrote: > Dear all, > > I am working on a program in Petsc that will read a vector and matrix from > an ASCII file and store it into a vector and matrix object in Petsc so I can > eventually solve. It needs to be an ASCII file because I will be outputting > the vector and matrix to an ASCII file from code in Delphi. > Would it be OK to use an intermediate program to read the ASCII output and dump in PETSc binary format? That would be rather trivial to do with a Python script. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From epscodes at gmail.com Mon Sep 11 15:22:15 2017 From: epscodes at gmail.com (Xiangdong) Date: Mon, 11 Sep 2017 16:22:15 -0400 Subject: [petsc-users] MatMult for block matrix Message-ID: Hello everyone, I have a block matrix B and vec x, and need to compute y=Bx many times. I found that each time before I call MatMult(B,x,y,ierr), I need to call VecSet(y,0.0,ierr). Otherwise, the result y is not correct (sometimes, but not always). I have this puzzle even if x and y are created by MatCreateVecs. I am using petsc 3.7.6 fortran api. If I create the same B in non-block format, it works fine without the extra vecset. It seems that I cannot reuse the allocation of vec y for block matrix without a reset. Is there any clue to explain this behavior? Thank you. Best, Xiangdong -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Sep 11 17:11:56 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 11 Sep 2017 18:11:56 -0400 Subject: [petsc-users] MatMult for block matrix In-Reply-To: References: Message-ID: <35D6445F-3A65-46E7-81E1-2F19B95A052B@mcs.anl.gov> This is either a bug in PETSc or some misunderstanding. You certainly should not have to call VecSet() on y first. Can you send (or point to) us a code that demonstrates the problem so we can debug? Barry > On Sep 11, 2017, at 4:22 PM, Xiangdong wrote: > > Hello everyone, > > I have a block matrix B and vec x, and need to compute y=Bx many times. I found that each time before I call MatMult(B,x,y,ierr), I need to call VecSet(y,0.0,ierr). Otherwise, the result y is not correct (sometimes, but not always). I have this puzzle even if x and y are created by MatCreateVecs. I am using petsc 3.7.6 fortran api. > > If I create the same B in non-block format, it works fine without the extra vecset. It seems that I cannot reuse the allocation of vec y for block matrix without a reset. > > Is there any clue to explain this behavior? > > Thank you. > > Best, > Xiangdong > > > > From imilian.hartig at gmail.com Tue Sep 12 11:52:40 2017 From: imilian.hartig at gmail.com (Maximilian Hartig) Date: Tue, 12 Sep 2017 18:52:40 +0200 Subject: [petsc-users] Gradient in DMProjectField Message-ID: <2A6B60B4-CA4C-454F-94B7-1E219540A07B@gmail.com> Hello, in my attempt to use an incremental formulation for elastoplasticity I wanted to create an auxiliary field in which the stresses from the past tilmestep are stored. I managed to create an auxiliary DM on which I have a field for the displacement, one for the stress and one for the inner variables. I transfer the displacements to a global vector on the auxiliary field, then calculate the stresses and inner variables using the DMProjectField function. Transfer of the displacements works fine and they are correct. The problem is with the stresses. While on some gauss points I get the correct stress, others have random stresses. I construct the strain using the gradient of the displacements and I believe this is where the problem lies. However, I have not been able to pinpoint it. When I just transfer the displacements to the auxiliary fields and then use a_x in the residual function instead, the stresses are correct. Is there a trick with using u_x in the DMProjectField? Thanks, Max From knepley at gmail.com Tue Sep 12 14:26:41 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 12 Sep 2017 15:26:41 -0400 Subject: [petsc-users] Gradient in DMProjectField In-Reply-To: <2A6B60B4-CA4C-454F-94B7-1E219540A07B@gmail.com> References: <2A6B60B4-CA4C-454F-94B7-1E219540A07B@gmail.com> Message-ID: On Tue, Sep 12, 2017 at 12:52 PM, Maximilian Hartig < imilian.hartig at gmail.com> wrote: > Hello, > > in my attempt to use an incremental formulation for elastoplasticity I > wanted to create an auxiliary field in which the stresses from the past > tilmestep are stored. I managed to create an auxiliary DM on which I have a > field for the displacement, one for the stress and one for the inner > variables. I transfer the displacements to a global vector on the auxiliary > field, then calculate the stresses and inner variables using the > DMProjectField function. Transfer of the displacements works fine and they > are correct. The problem is with the stresses. While on some gauss points I > get the correct stress, others have random stresses. I construct the strain > using the gradient of the displacements and I believe this is where the > problem lies. However, I have not been able to pinpoint it. > When I just transfer the displacements to the auxiliary fields and then > use a_x in the residual function instead, the stresses are correct. Is > there a trick with using u_x in the DMProjectField? > Its possible that there is a bug. Just so I understand, you are saying that DMProjectField() has the wrong gradient of the input Vec? It would really be great if you could demonstrate it with a standalone example. Thanks, Matt > Thanks, > Max -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fe.wallner at gmail.com Tue Sep 12 15:30:40 2017 From: fe.wallner at gmail.com (Felipe Giacomelli) Date: Tue, 12 Sep 2017 17:30:40 -0300 Subject: [petsc-users] Solving a coupled linear system of equations in parallel Message-ID: Hello, I would like to use PETSc to solve coupled linear systems, such as the ones that originate from the discretization of Navier Stokes equations. In a two dimensions, incompressible, steady state case, one would have the following set of equations (Finite Volume Method): A_uu 0 A_up u b_u 0 A_vv A_vp v = b_v A_pu A_pv 0 p b_p What would be the standard approach to solve this linear system? How can one ?split? this linear system among several processes? When there is only one variable involved, as in heat transfer problems, I use METIS to decompose the domain (graph partition). Thus, each process build its block of the major linear system of equations. However, if there?s more than one variable per node, I don?t know what would be the best way to assemble the system of equations in a parallel context. Notes: The discretization method employed is the Element based Finite Volume Method. METIS is used to decompose the domain (graph partition). I understand how PETSc is used to solve linear systems of equation when there is only one variable per node. I would like to keep the domain decomposition, if that?s possible. Articles or other reading suggestions would be appreciated. Thank you, -- Felipe M Wallner Giacomelli -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Sep 12 15:41:32 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 12 Sep 2017 16:41:32 -0400 Subject: [petsc-users] Solving a coupled linear system of equations in parallel In-Reply-To: References: Message-ID: <03804322-BAB3-44BB-8075-82EE165EF949@mcs.anl.gov> You will "interlace" the variables; that is store the first u followed by the first v followed by the first p followed by the second u etc. Do the same partitioning of the mesh as you do for the heat equation. You will use the PCFIELDSPLIT as the solver, likely with multigrid for the A_uu and A_vv matrices Barry > On Sep 12, 2017, at 4:30 PM, Felipe Giacomelli wrote: > > Hello, > I would like to use PETSc to solve coupled linear systems, such as the ones that originate from the discretization of Navier Stokes equations. In a two dimensions, incompressible, steady state case, one would have the following set of equations (Finite Volume Method): > > A_uu 0 A_up u b_u > 0 A_vv A_vp v = b_v > A_pu A_pv 0 p b_p > > What would be the standard approach to solve this linear system? How can one ?split? this linear system among several processes? > > When there is only one variable involved, as in heat transfer problems, I use METIS to decompose the domain (graph partition). Thus, each process build its block of the major linear system of equations. However, if there?s more than one variable per node, I don?t know what would be the best way to assemble the system of equations in a parallel context. > > Notes: > The discretization method employed is the Element based Finite Volume Method. > METIS is used to decompose the domain (graph partition). > I understand how PETSc is used to solve linear systems of equation when there is only one variable per node. > I would like to keep the domain decomposition, if that?s possible. > Articles or other reading suggestions would be appreciated. > > Thank you, > > -- > Felipe M Wallner Giacomelli From imilian.hartig at gmail.com Wed Sep 13 04:22:51 2017 From: imilian.hartig at gmail.com (Maximilian Hartig) Date: Wed, 13 Sep 2017 11:22:51 +0200 Subject: [petsc-users] Gradient in DMProjectField In-Reply-To: References: <2A6B60B4-CA4C-454F-94B7-1E219540A07B@gmail.com> Message-ID: <8B0FE399-8523-47A3-A0CD-2E15B2B0E819@gmail.com> > On 12. Sep 2017, at 21:26, Matthew Knepley wrote: > > On Tue, Sep 12, 2017 at 12:52 PM, Maximilian Hartig > wrote: > Hello, > > in my attempt to use an incremental formulation for elastoplasticity I wanted to create an auxiliary field in which the stresses from the past tilmestep are stored. I managed to create an auxiliary DM on which I have a field for the displacement, one for the stress and one for the inner variables. I transfer the displacements to a global vector on the auxiliary field, then calculate the stresses and inner variables using the DMProjectField function. Transfer of the displacements works fine and they are correct. The problem is with the stresses. While on some gauss points I get the correct stress, others have random stresses. I construct the strain using the gradient of the displacements and I believe this is where the problem lies. However, I have not been able to pinpoint it. > When I just transfer the displacements to the auxiliary fields and then use a_x in the residual function instead, the stresses are correct. Is there a trick with using u_x in the DMProjectField? > > Its possible that there is a bug. Just so I understand, you are saying that DMProjectField() has the wrong gradient of the input Vec? > It would really be great if you could demonstrate it with a standalone example. That is correct, and I suspect it has something to do with the Dirichlet boundary conditions I impose. If I comment out the following line, projected result and expected result are the same: ierr = DMAddBoundary(dm, PETSC_TRUE, "fixed", "Face Sets",0,Ncomp,restrictall,(void (*)()) zero_vector, Nfid,fid,NULL);CHKERRQ(ierr); Please find attached the test example and the test mesh. I run it with: -disp_petscspace_order 2 -stress_petscspace_order 2 Thanks, Max > > Thanks, > > Matt > > Thanks, > Max > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: projectfieldtest.c Type: application/octet-stream Size: 8596 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testcube.msh Type: application/octet-stream Size: 841 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From federico.golfre at gmail.com Wed Sep 13 10:25:20 2017 From: federico.golfre at gmail.com (=?UTF-8?Q?Federico_Golfr=C3=A8_Andreasi?=) Date: Wed, 13 Sep 2017 17:25:20 +0200 Subject: [petsc-users] mg pre-conditioner default setup from PETSc-3.4 to PETSc-3.7 Message-ID: Dear PETSc users/developers, I recently switched from PETSc-3.4 to PETSc-3.7 and found that some default setup for the "mg" (mutigrid) preconditioner have changed. We were solving a linear system passing, throug command line, the following options: -ksp_type fgmres -ksp_max_it 100000 -ksp_rtol 0.000001 -pc_type mg -ksp_view The output of the KSP view is as follow: KSP Object: 128 MPI processes type: fgmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=100000, initial guess is zero tolerances: relative=1e-06, absolute=1e-50, divergence=10000 right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 128 MPI processes type: mg MG: type is MULTIPLICATIVE, levels=1 cycles=v Cycles per PCApply=1 Not using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (mg_levels_0_) 128 MPI processes type: chebyshev Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903 Chebyshev: estimated using: [0 0.1; 0 1.1] KSP Object: (mg_levels_0_est_) 128 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_0_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix followed by preconditioner matrix: Matrix Object: 128 MPI processes type: mpiaij rows=279669, cols=279669 total: nonzeros=6427943, allocated nonzeros=6427943 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Matrix Object: 128 MPI processes type: mpiaij rows=279669, cols=279669 total: nonzeros=6427943, allocated nonzeros=6427943 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_0_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix followed by preconditioner matrix: Matrix Object: 128 MPI processes type: mpiaij rows=279669, cols=279669 total: nonzeros=6427943, allocated nonzeros=6427943 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Matrix Object: 128 MPI processes type: mpiaij rows=279669, cols=279669 total: nonzeros=6427943, allocated nonzeros=6427943 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines linear system matrix followed by preconditioner matrix: Matrix Object: 128 MPI processes type: mpiaij rows=279669, cols=279669 total: nonzeros=6427943, allocated nonzeros=6427943 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Matrix Object: 128 MPI processes type: mpiaij rows=279669, cols=279669 total: nonzeros=6427943, allocated nonzeros=6427943 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines When I build the same program using PETSc-3.7 and run it with the same options we observe that the runtime increases and the convergence is slightly different. The output of the KSP view is: KSP Object: 128 MPI processes type: fgmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=100000, initial guess is zero tolerances: relative=1e-06, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 128 MPI processes type: mg MG: type is MULTIPLICATIVE, levels=1 cycles=v Cycles per PCApply=1 Not using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (mg_levels_0_) 128 MPI processes type: chebyshev Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903 Chebyshev: eigenvalues estimated using gmres with translations [0. 0.1; 0. 1.1] KSP Object: (mg_levels_0_esteig_) 128 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10, initial guess is zero *tolerances: relative=1e-12*, absolute=1e-50, divergence=10000. left preconditioning *using PRECONDITIONED norm type for convergence test* maximum iterations=2, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_0_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix followed by preconditioner matrix: Mat Object: 128 MPI processes type: mpiaij rows=279669, cols=279669 total: nonzeros=6427943, allocated nonzeros=6427943 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Mat Object: 128 MPI processes type: mpiaij rows=279669, cols=279669 total: nonzeros=6427943, allocated nonzeros=6427943 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines linear system matrix followed by preconditioner matrix: Mat Object: 128 MPI processes type: mpiaij rows=279669, cols=279669 total: nonzeros=6427943, allocated nonzeros=6427943 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Mat Object: 128 MPI processes type: mpiaij rows=279669, cols=279669 total: nonzeros=6427943, allocated nonzeros=6427943 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines I was able to get a closer solution adding the following options: -mg_levels_0_esteig_ksp_norm_type none -mg_levels_0_esteig_ksp_rtol 1.0e-5 -mg_levels_ksp_max_it 1 But I still can reach the same runtime we were observing with PETSc-3.4, could you please advice me if I should specify any other options? Thank you very much for your support, Federico Golfre' Andreasi -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Sep 13 10:29:56 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 13 Sep 2017 10:29:56 -0500 Subject: [petsc-users] mg pre-conditioner default setup from PETSc-3.4 to PETSc-3.7 In-Reply-To: References: Message-ID: <3E0D16E2-BA8F-4792-BBC3-8C55611292CC@mcs.anl.gov> There will likely always be slight differences in convergence over that many releases. Lots of little defaults etc get changed over time as we learn from users and increase the robustness of the defaults. So in your case do the differences matter? 1) What is the time to solution in both cases, is it a few percent different or now much slower? 2) What about number of iterations? Almost identical (say 1 or 2 different) or does it now take 30 iterations when it use to take 5? Barry > On Sep 13, 2017, at 10:25 AM, Federico Golfr? Andreasi wrote: > > Dear PETSc users/developers, > > I recently switched from PETSc-3.4 to PETSc-3.7 and found that some default setup for the "mg" (mutigrid) preconditioner have changed. > > We were solving a linear system passing, throug command line, the following options: > -ksp_type fgmres > -ksp_max_it 100000 > -ksp_rtol 0.000001 > -pc_type mg > -ksp_view > > The output of the KSP view is as follow: > > KSP Object: 128 MPI processes > type: fgmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=100000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 128 MPI processes > type: mg > MG: type is MULTIPLICATIVE, levels=1 cycles=v > Cycles per PCApply=1 > Not using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (mg_levels_0_) 128 MPI processes > type: chebyshev > Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903 > Chebyshev: estimated using: [0 0.1; 0 1.1] > KSP Object: (mg_levels_0_est_) 128 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_levels_0_) 128 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > linear system matrix followed by preconditioner matrix: > Matrix Object: 128 MPI processes > type: mpiaij > rows=279669, cols=279669 > total: nonzeros=6427943, allocated nonzeros=6427943 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Matrix Object: 128 MPI processes > type: mpiaij > rows=279669, cols=279669 > total: nonzeros=6427943, allocated nonzeros=6427943 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_levels_0_) 128 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > linear system matrix followed by preconditioner matrix: > Matrix Object: 128 MPI processes > type: mpiaij > rows=279669, cols=279669 > total: nonzeros=6427943, allocated nonzeros=6427943 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Matrix Object: 128 MPI processes > type: mpiaij > rows=279669, cols=279669 > total: nonzeros=6427943, allocated nonzeros=6427943 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > linear system matrix followed by preconditioner matrix: > Matrix Object: 128 MPI processes > type: mpiaij > rows=279669, cols=279669 > total: nonzeros=6427943, allocated nonzeros=6427943 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Matrix Object: 128 MPI processes > type: mpiaij > rows=279669, cols=279669 > total: nonzeros=6427943, allocated nonzeros=6427943 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > When I build the same program using PETSc-3.7 and run it with the same options we observe that the runtime increases and the convergence is slightly different. The output of the KSP view is: > > KSP Object: 128 MPI processes > type: fgmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=100000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000. > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 128 MPI processes > type: mg > MG: type is MULTIPLICATIVE, levels=1 cycles=v > Cycles per PCApply=1 > Not using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (mg_levels_0_) 128 MPI processes > type: chebyshev > Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903 > Chebyshev: eigenvalues estimated using gmres with translations [0. 0.1; 0. 1.1] > KSP Object: (mg_levels_0_esteig_) 128 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10, initial guess is zero > tolerances: relative=1e-12, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > maximum iterations=2, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_levels_0_) 128 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > linear system matrix followed by preconditioner matrix: > Mat Object: 128 MPI processes > type: mpiaij > rows=279669, cols=279669 > total: nonzeros=6427943, allocated nonzeros=6427943 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Mat Object: 128 MPI processes > type: mpiaij > rows=279669, cols=279669 > total: nonzeros=6427943, allocated nonzeros=6427943 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > linear system matrix followed by preconditioner matrix: > Mat Object: 128 MPI processes > type: mpiaij > rows=279669, cols=279669 > total: nonzeros=6427943, allocated nonzeros=6427943 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Mat Object: 128 MPI processes > type: mpiaij > rows=279669, cols=279669 > total: nonzeros=6427943, allocated nonzeros=6427943 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > I was able to get a closer solution adding the following options: > -mg_levels_0_esteig_ksp_norm_type none > -mg_levels_0_esteig_ksp_rtol 1.0e-5 > -mg_levels_ksp_max_it 1 > > But I still can reach the same runtime we were observing with PETSc-3.4, could you please advice me if I should specify any other options? > > Thank you very much for your support, > Federico Golfre' Andreasi > From hzhang at mcs.anl.gov Wed Sep 13 10:35:06 2017 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 13 Sep 2017 10:35:06 -0500 Subject: [petsc-users] mg pre-conditioner default setup from PETSc-3.4 to PETSc-3.7 In-Reply-To: References: Message-ID: Federico : > > Coarse grid solver -- level ------------------------------- > KSP Object: (mg_levels_0_) 128 MPI processes > type: chebyshev > Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903 > Chebyshev: eigenvalues estimated using gmres with translations > [0. 0.1; 0. 1.1] > KSP Object: (mg_levels_0_esteig_) 128 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10, initial guess is zero > *tolerances: relative=1e-12*, absolute=1e-50, divergence=10000. > left preconditioning > *using PRECONDITIONED norm type for convergence test* > maximum iterations=2, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > Chebyshev requires an estimate of operator eigenvalues, for which we use few gmres iterations. These default options are used for eigenvalue estimates. Hong -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Sep 13 10:48:05 2017 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 13 Sep 2017 11:48:05 -0400 Subject: [petsc-users] mg pre-conditioner default setup from PETSc-3.4 to PETSc-3.7 In-Reply-To: References: Message-ID: Two iterations for the eigen estimate is too low and gmres converges slowly. I'm surprised this does not diverge, or just die, for a Laplacian because you need to get an upper bound. Cheby will scale the estimate up by some safety factor (is it really large now?). Try: -mg_levels_esteig_ksp_max_it 10 (the old default). I usually use 5. Also, I would suggest using cg (-mg_levels_esteig_ksp_type cg), it converges much faster. If your problem is not very asymmetric, it is fine. On Wed, Sep 13, 2017 at 11:35 AM, Hong wrote: > Federico : > >> >> Coarse grid solver -- level ------------------------------- >> KSP Object: (mg_levels_0_) 128 MPI processes >> type: chebyshev >> Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903 >> Chebyshev: eigenvalues estimated using gmres with translations >> [0. 0.1; 0. 1.1] >> KSP Object: (mg_levels_0_esteig_) 128 MPI processes >> type: gmres >> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> GMRES: happy breakdown tolerance 1e-30 >> maximum iterations=10, initial guess is zero >> *tolerances: relative=1e-12*, absolute=1e-50, >> divergence=10000. >> left preconditioning >> *using PRECONDITIONED norm type for convergence test* >> maximum iterations=2, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> left preconditioning >> using NONE norm type for convergence test >> > > Chebyshev requires an estimate of operator eigenvalues, for which we use > few gmres iterations. These default options are used for eigenvalue > estimates. > > Hong > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From federico.golfre at gmail.com Wed Sep 13 10:56:11 2017 From: federico.golfre at gmail.com (=?UTF-8?Q?Federico_Golfr=C3=A8_Andreasi?=) Date: Wed, 13 Sep 2017 17:56:11 +0200 Subject: [petsc-users] mg pre-conditioner default setup from PETSc-3.4 to PETSc-3.7 In-Reply-To: <3E0D16E2-BA8F-4792-BBC3-8C55611292CC@mcs.anl.gov> References: <3E0D16E2-BA8F-4792-BBC3-8C55611292CC@mcs.anl.gov> Message-ID: Hi Barry, I understand and perfectly agree with you that the behavior increase after the release due to better tuning. In my case, the difference in the solution is negligible, but the runtime increases up to +70% (with the same number of ksp_iterations). So I was wondering if maybe there were just some flags related to memory preallocation or re-usage of intermediate solution that before was defaulted. Thank you, Federico On 13 September 2017 at 17:29, Barry Smith wrote: > > There will likely always be slight differences in convergence over that > many releases. Lots of little defaults etc get changed over time as we > learn from users and increase the robustness of the defaults. > > So in your case do the differences matter? > > 1) What is the time to solution in both cases, is it a few percent > different or now much slower? > > 2) What about number of iterations? Almost identical (say 1 or 2 > different) or does it now take 30 iterations when it use to take 5? > > Barry > > > On Sep 13, 2017, at 10:25 AM, Federico Golfr? Andreasi < > federico.golfre at gmail.com> wrote: > > > > Dear PETSc users/developers, > > > > I recently switched from PETSc-3.4 to PETSc-3.7 and found that some > default setup for the "mg" (mutigrid) preconditioner have changed. > > > > We were solving a linear system passing, throug command line, the > following options: > > -ksp_type fgmres > > -ksp_max_it 100000 > > -ksp_rtol 0.000001 > > -pc_type mg > > -ksp_view > > > > The output of the KSP view is as follow: > > > > KSP Object: 128 MPI processes > > type: fgmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=100000, initial guess is zero > > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > > right preconditioning > > using UNPRECONDITIONED norm type for convergence test > > PC Object: 128 MPI processes > > type: mg > > MG: type is MULTIPLICATIVE, levels=1 cycles=v > > Cycles per PCApply=1 > > Not using Galerkin computed coarse grid matrices > > Coarse grid solver -- level ------------------------------- > > KSP Object: (mg_levels_0_) 128 MPI processes > > type: chebyshev > > Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903 > > Chebyshev: estimated using: [0 0.1; 0 1.1] > > KSP Object: (mg_levels_0_est_) 128 MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (mg_levels_0_) 128 MPI processes > > type: sor > > SOR: type = local_symmetric, iterations = 1, local > iterations = 1, omega = 1 > > linear system matrix followed by preconditioner matrix: > > Matrix Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > Matrix Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > maximum iterations=1, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (mg_levels_0_) 128 MPI processes > > type: sor > > SOR: type = local_symmetric, iterations = 1, local iterations = > 1, omega = 1 > > linear system matrix followed by preconditioner matrix: > > Matrix Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > Matrix Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > linear system matrix followed by preconditioner matrix: > > Matrix Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > Matrix Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > > > When I build the same program using PETSc-3.7 and run it with the same > options we observe that the runtime increases and the convergence is > slightly different. The output of the KSP view is: > > > > KSP Object: 128 MPI processes > > type: fgmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=100000, initial guess is zero > > tolerances: relative=1e-06, absolute=1e-50, divergence=10000. > > right preconditioning > > using UNPRECONDITIONED norm type for convergence test > > PC Object: 128 MPI processes > > type: mg > > MG: type is MULTIPLICATIVE, levels=1 cycles=v > > Cycles per PCApply=1 > > Not using Galerkin computed coarse grid matrices > > Coarse grid solver -- level ------------------------------- > > KSP Object: (mg_levels_0_) 128 MPI processes > > type: chebyshev > > Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903 > > Chebyshev: eigenvalues estimated using gmres with translations > [0. 0.1; 0. 1.1] > > KSP Object: (mg_levels_0_esteig_) 128 MPI > processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10, initial guess is zero > > tolerances: relative=1e-12, absolute=1e-50, divergence=10000. > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > maximum iterations=2, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (mg_levels_0_) 128 MPI processes > > type: sor > > SOR: type = local_symmetric, iterations = 1, local iterations = > 1, omega = 1. > > linear system matrix followed by preconditioner matrix: > > Mat Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > Mat Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > linear system matrix followed by preconditioner matrix: > > Mat Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > Mat Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > > > I was able to get a closer solution adding the following options: > > -mg_levels_0_esteig_ksp_norm_type none > > -mg_levels_0_esteig_ksp_rtol 1.0e-5 > > -mg_levels_ksp_max_it 1 > > > > But I still can reach the same runtime we were observing with PETSc-3.4, > could you please advice me if I should specify any other options? > > > > Thank you very much for your support, > > Federico Golfre' Andreasi > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Sep 13 10:58:33 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 13 Sep 2017 10:58:33 -0500 Subject: [petsc-users] mg pre-conditioner default setup from PETSc-3.4 to PETSc-3.7 In-Reply-To: References: <3E0D16E2-BA8F-4792-BBC3-8C55611292CC@mcs.anl.gov> Message-ID: <6BDE8DAF-F70B-42D7-A92B-8CD682BAC928@mcs.anl.gov> > On Sep 13, 2017, at 10:56 AM, Federico Golfr? Andreasi wrote: > > Hi Barry, > > I understand and perfectly agree with you that the behavior increase after the release due to better tuning. > > In my case, the difference in the solution is negligible, but the runtime increases up to +70% (with the same number of ksp_iterations). Ok this is an important (and bad) difference. > So I was wondering if maybe there were just some flags related to memory preallocation or re-usage of intermediate solution that before was defaulted. Note likely it is this. Are both compiled with the same level of compiler optimization? Please run both with -log_summary and send the output, this will tell us WHAT parts are now slower. Barry > > Thank you, > Federico > > > > On 13 September 2017 at 17:29, Barry Smith wrote: > > There will likely always be slight differences in convergence over that many releases. Lots of little defaults etc get changed over time as we learn from users and increase the robustness of the defaults. > > So in your case do the differences matter? > > 1) What is the time to solution in both cases, is it a few percent different or now much slower? > > 2) What about number of iterations? Almost identical (say 1 or 2 different) or does it now take 30 iterations when it use to take 5? > > Barry > > > On Sep 13, 2017, at 10:25 AM, Federico Golfr? Andreasi wrote: > > > > Dear PETSc users/developers, > > > > I recently switched from PETSc-3.4 to PETSc-3.7 and found that some default setup for the "mg" (mutigrid) preconditioner have changed. > > > > We were solving a linear system passing, throug command line, the following options: > > -ksp_type fgmres > > -ksp_max_it 100000 > > -ksp_rtol 0.000001 > > -pc_type mg > > -ksp_view > > > > The output of the KSP view is as follow: > > > > KSP Object: 128 MPI processes > > type: fgmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=100000, initial guess is zero > > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > > right preconditioning > > using UNPRECONDITIONED norm type for convergence test > > PC Object: 128 MPI processes > > type: mg > > MG: type is MULTIPLICATIVE, levels=1 cycles=v > > Cycles per PCApply=1 > > Not using Galerkin computed coarse grid matrices > > Coarse grid solver -- level ------------------------------- > > KSP Object: (mg_levels_0_) 128 MPI processes > > type: chebyshev > > Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903 > > Chebyshev: estimated using: [0 0.1; 0 1.1] > > KSP Object: (mg_levels_0_est_) 128 MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (mg_levels_0_) 128 MPI processes > > type: sor > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > > linear system matrix followed by preconditioner matrix: > > Matrix Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > Matrix Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > maximum iterations=1, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (mg_levels_0_) 128 MPI processes > > type: sor > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > > linear system matrix followed by preconditioner matrix: > > Matrix Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > Matrix Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > linear system matrix followed by preconditioner matrix: > > Matrix Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > Matrix Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > > > When I build the same program using PETSc-3.7 and run it with the same options we observe that the runtime increases and the convergence is slightly different. The output of the KSP view is: > > > > KSP Object: 128 MPI processes > > type: fgmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=100000, initial guess is zero > > tolerances: relative=1e-06, absolute=1e-50, divergence=10000. > > right preconditioning > > using UNPRECONDITIONED norm type for convergence test > > PC Object: 128 MPI processes > > type: mg > > MG: type is MULTIPLICATIVE, levels=1 cycles=v > > Cycles per PCApply=1 > > Not using Galerkin computed coarse grid matrices > > Coarse grid solver -- level ------------------------------- > > KSP Object: (mg_levels_0_) 128 MPI processes > > type: chebyshev > > Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903 > > Chebyshev: eigenvalues estimated using gmres with translations [0. 0.1; 0. 1.1] > > KSP Object: (mg_levels_0_esteig_) 128 MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10, initial guess is zero > > tolerances: relative=1e-12, absolute=1e-50, divergence=10000. > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > maximum iterations=2, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (mg_levels_0_) 128 MPI processes > > type: sor > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > > linear system matrix followed by preconditioner matrix: > > Mat Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > Mat Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > linear system matrix followed by preconditioner matrix: > > Mat Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > Mat Object: 128 MPI processes > > type: mpiaij > > rows=279669, cols=279669 > > total: nonzeros=6427943, allocated nonzeros=6427943 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > > > I was able to get a closer solution adding the following options: > > -mg_levels_0_esteig_ksp_norm_type none > > -mg_levels_0_esteig_ksp_rtol 1.0e-5 > > -mg_levels_ksp_max_it 1 > > > > But I still can reach the same runtime we were observing with PETSc-3.4, could you please advice me if I should specify any other options? > > > > Thank you very much for your support, > > Federico Golfre' Andreasi > > > > From sean at farley.io Wed Sep 13 13:41:37 2017 From: sean at farley.io (Sean Farley) Date: Wed, 13 Sep 2017 11:41:37 -0700 Subject: [petsc-users] [petsc-dev] possibility of short term subcontract work with PETSc In-Reply-To: References: Message-ID: Barry Smith writes: > PETSc folks, > > Argonne National Laboratory has recently set up a system that may make it possible for the PETSc group at ANL to subcontract particular PETSc contribution projects to developers in most of the world. These could be from one week projects to months. Nice, that's pretty cool! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 800 bytes Desc: not available URL: From davegwebb10 at gmail.com Wed Sep 13 17:14:22 2017 From: davegwebb10 at gmail.com (David Gross) Date: Thu, 14 Sep 2017 00:14:22 +0200 Subject: [petsc-users] Matrix dot product In-Reply-To: References: Message-ID: Hi Matt, Thank you for getting back to me. Your answer confirms what I thought in terms of existing functionality. I think it shouldn't be too hard to make a copy of MatAXPY to MatAXY where it performs Xij = A*Xij*Yij (or without the A). I could then do the MatNorm of the resulting matrix to get what I need. Is a MatAXY function desirable as a source contribution? I am hoping to use PETSc for performing basic vector and matrix operations on dense matrices and 1D vectors. The main uses are matmult, matmatmult and matrix additions and scaling. The application is for implementing a parallel version on an existing Pan-Reif matrix inversion algorithm. The choice of using PETSc is mostly due to us already using it in the same program to solve sparse matrices (with MUMPS) with the goal of avoiding adding yet another package (ex ScaLAPACK/PBLAS) into the code even if other packages may be more directly oriented towards my application. Regards, Dave On Mon, Sep 11, 2017 at 2:19 AM, Matthew Knepley wrote: > On Sun, Sep 10, 2017 at 5:51 PM, David Gross > wrote: > >> Hello, >> I was wondering if there was a matrix equivalent to the vecDot function >> (Frobenius inner product)? As far as I can tell the closest thing is >> MatNorm with NORM_FROBENIUS, but obviously this is acting on only one >> matrix. >> >> If there is not a built in function, what is the best way to compute >> this? I am working fortran90. >> > > We do not have this. However, it would be trivial to add since we have > > http://www.mcs.anl.gov/petsc/petsc-current/docs/ > manualpages/Mat/MatAXPY.html > > since you just replace + with * in our code. You could argue that we > should have written for > a general ring, but C makes this cumbersome. Do you think you could make > the change? > > What are you using this for? > > Thanks, > > Matt > > >> Regards, >> Dave >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spandey2 at wpi.edu Wed Sep 13 17:27:57 2017 From: spandey2 at wpi.edu (Pandey, Siddhant) Date: Wed, 13 Sep 2017 22:27:57 +0000 Subject: [petsc-users] SLEPC - Hermitian matrix gives complex eigenvalues Message-ID: <14C3EF90-C404-45AD-9516-63D3F9CA4D10@wpi.edu> Hello all, I am diagonalizing a hermitian matrix (MatIsHermitian = 1 up to a tolerance of 1e-15) using krylovschur, with EPS_TARGET_MAGNITUDE = 0, but getting complex eigenvalues. Also, I am getting seemingly spurious eigenvalues of magnitude very close to 0, whose relative errors are much larger than my set tolerance. Can anyone indicate what might be the cause and possible solutions. I have attached the files eps_view, A_mat_binary, and B_mat_binary files, which show the settings I have used, and contain the A and B matrices respectively (in binary). The eigenvalues I get are: # Eigenvalue ? ---------------------------------------------------------------- 0 0.0000000000000077 + i -0.0000000000000000 1 0.0000000000000265 + i -0.0000000000000008 2 0.0000000000000340 + i 0.0000000000000002 3 0.0000000000000373 + i 0.0000000000000001 4 0.0000000000000444 + i -0.0000000000000007 5 0.0000000000000448 + i -0.0000000000000017 6 0.0000000000000470 + i 0.0000000000000002 7 0.0000000000000489 + i 0.0000000000000030 8 0.0000000000000548 + i 0.0000000000000006 9 0.0000000000000585 + i 0.0000000000000001 10 0.0000000000000643 + i 0.0000000000000005 11 0.0000000000000714 + i -0.0000000000000008 12 0.0000000000000750 + i -0.0000000000000019 13 0.0000000000000752 + i 0.0000000000000031 14 0.0000000000000769 + i -0.0000000000000002 15 0.0000000000000784 + i -0.0000000000000037 16 0.0000000000000860 + i 0.0000000000000002 17 0.0000000000000880 + i -0.0000000000000004 18 0.0000000000000968 + i -0.0000000000000001 19 0.0000000000000979 + i 0.0000000000000013 20 0.0000000000001045 + i -0.0000000000000001 21 0.0000000000001150 + i -0.0000000000000011 22 0.0000000000001348 + i -0.0000000000000012 23 0.0000000000001446 + i -0.0000000000000019 24 0.0000000000001454 + i 0.0000000000000011 25 0.0000000000001555 + i 0.0000000000000003 26 0.0000000000002513 + i -0.0000000000000009 27 5.0908854514230413 + i -0.0022004178122762 28 5.2768106039842175 + i 0.1043464906789375 29 5.3003883062604187 + i -0.0757735907905433 30 11.5143655883932929 + i 0.0049838692042474 31 11.8821523259838653 + i -0.1475608751440501 32 11.9515216995487101 + i 0.1857395729336506 33 12.0909362158339384 + i 0.0049114285397287 34 12.0905704159492675 + i -0.1522880348537981 35 12.2205398661469111 + i -0.0781999802933937 36 12.4807156964720480 + i -0.2850122604907908 37 12.6082849940660289 + i 0.0560456679728079 38 12.9384278480576125 + i 0.0238826907631012 39 17.5230234731441357 + i -0.1824807274488794 40 17.5503678395543901 + i 0.1785356473404145 41 17.6933953160112409 + i -0.0626631055149425 42 19.1692824404480930 + i -0.6351729266691462 43 19.4158452684509797 + i 0.3151965488807310 44 19.6020750507704591 + i -0.2559887580276014 45 19.6443102906562181 + i 1.1005601705485646 46 19.7948713379697452 + i 0.1230015140422697 47 19.8791098474284347 + i -0.1943322911563744 48 20.1268732265661860 + i -0.0340890856219265 49 20.1368588182453898 + i 0.2673370459460956 Thanks in advance, Sidd Worcester Polytechnic, U.S.A. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: A_mat_binary Type: application/octet-stream Size: 2816632 bytes Desc: A_mat_binary URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: B_mat_binary Type: application/octet-stream Size: 2816632 bytes Desc: B_mat_binary URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: eps_view_pre.txt URL: From knepley at gmail.com Wed Sep 13 19:49:36 2017 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 13 Sep 2017 20:49:36 -0400 Subject: [petsc-users] Matrix dot product In-Reply-To: References: Message-ID: On Wed, Sep 13, 2017 at 6:14 PM, David Gross wrote: > Hi Matt, > Thank you for getting back to me. Your answer confirms what I thought in > terms of existing functionality. I think it shouldn't be too hard to make a > copy of MatAXPY to MatAXY where it performs Xij = A*Xij*Yij (or without the > A). I could then do the MatNorm of the resulting matrix to get what I need. > > Is a MatAXY function desirable as a source contribution? > Yes. I would prefer calling it MatPointwiseMult, since you can see it as VecPointwiseMult on a Vec obtained from forgetting the linear operator structure of the matrix (forgetful functor). > I am hoping to use PETSc for performing basic vector and matrix operations > on dense matrices and 1D vectors. The main uses are matmult, matmatmult and > matrix additions and scaling. The application is for implementing a > parallel version on an existing Pan-Reif matrix inversion algorithm. > Is this Newton's method on the vector space of matrices? Thanks, Matt > The choice of using PETSc is mostly due to us already using it in the same > program to solve sparse matrices (with MUMPS) with the goal of avoiding > adding yet another package (ex ScaLAPACK/PBLAS) into the code even if other > packages may be more directly oriented towards my application. > > Regards, > Dave > > > > On Mon, Sep 11, 2017 at 2:19 AM, Matthew Knepley > wrote: > >> On Sun, Sep 10, 2017 at 5:51 PM, David Gross >> wrote: >> >>> Hello, >>> I was wondering if there was a matrix equivalent to the vecDot function >>> (Frobenius inner product)? As far as I can tell the closest thing is >>> MatNorm with NORM_FROBENIUS, but obviously this is acting on only one >>> matrix. >>> >>> If there is not a built in function, what is the best way to compute >>> this? I am working fortran90. >>> >> >> We do not have this. However, it would be trivial to add since we have >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpage >> s/Mat/MatAXPY.html >> >> since you just replace + with * in our code. You could argue that we >> should have written for >> a general ring, but C makes this cumbersome. Do you think you could make >> the change? >> >> What are you using this for? >> >> Thanks, >> >> Matt >> >> >>> Regards, >>> Dave >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Thu Sep 14 03:33:01 2017 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 14 Sep 2017 10:33:01 +0200 Subject: [petsc-users] SLEPC - Hermitian matrix gives complex eigenvalues In-Reply-To: <14C3EF90-C404-45AD-9516-63D3F9CA4D10@wpi.edu> References: <14C3EF90-C404-45AD-9516-63D3F9CA4D10@wpi.edu> Message-ID: <071CC27A-4A0D-4529-8B62-67CA1A44042F@dsic.upv.es> > El 14 sept 2017, a las 0:27, Pandey, Siddhant escribi?: > > Hello all, > > I am diagonalizing a hermitian matrix (MatIsHermitian = 1 up to a tolerance of 1e-15) using krylovschur, with EPS_TARGET_MAGNITUDE = 0, but getting complex eigenvalues. Also, I am getting seemingly spurious eigenvalues of magnitude very close to 0, whose relative errors are much larger than my set tolerance. Can anyone indicate what might be the cause and possible solutions. > > I have attached the files eps_view, A_mat_binary, and B_mat_binary files, which show the settings I have used, and contain the A and B matrices respectively (in binary). > > The eigenvalues I get are: > > # > > Eigenvalue > ? ---------------------------------------------------------------- > 0 0.0000000000000077 + i -0.0000000000000000 > 1 0.0000000000000265 + i -0.0000000000000008 > 2 0.0000000000000340 + i 0.0000000000000002 > 3 0.0000000000000373 + i 0.0000000000000001 > 4 0.0000000000000444 + i -0.0000000000000007 > 5 0.0000000000000448 + i -0.0000000000000017 > 6 0.0000000000000470 + i 0.0000000000000002 > 7 0.0000000000000489 + i 0.0000000000000030 > 8 0.0000000000000548 + i 0.0000000000000006 > 9 0.0000000000000585 + i 0.0000000000000001 > 10 0.0000000000000643 + i 0.0000000000000005 > 11 0.0000000000000714 + i -0.0000000000000008 > 12 0.0000000000000750 + i -0.0000000000000019 > 13 0.0000000000000752 + i 0.0000000000000031 > 14 0.0000000000000769 + i -0.0000000000000002 > 15 0.0000000000000784 + i -0.0000000000000037 > 16 0.0000000000000860 + i 0.0000000000000002 > 17 0.0000000000000880 + i -0.0000000000000004 > 18 0.0000000000000968 + i -0.0000000000000001 > 19 0.0000000000000979 + i 0.0000000000000013 > 20 0.0000000000001045 + i -0.0000000000000001 > 21 0.0000000000001150 + i -0.0000000000000011 > 22 0.0000000000001348 + i -0.0000000000000012 > 23 0.0000000000001446 + i -0.0000000000000019 > 24 0.0000000000001454 + i 0.0000000000000011 > 25 0.0000000000001555 + i 0.0000000000000003 > 26 0.0000000000002513 + i -0.0000000000000009 > 27 5.0908854514230413 + i -0.0022004178122762 > 28 5.2768106039842175 + i 0.1043464906789375 > 29 5.3003883062604187 + i -0.0757735907905433 > 30 11.5143655883932929 + i 0.0049838692042474 > 31 11.8821523259838653 + i -0.1475608751440501 > 32 11.9515216995487101 + i 0.1857395729336506 > 33 12.0909362158339384 + i 0.0049114285397287 > 34 12.0905704159492675 + i -0.1522880348537981 > 35 12.2205398661469111 + i -0.0781999802933937 > 36 12.4807156964720480 + i -0.2850122604907908 > 37 12.6082849940660289 + i 0.0560456679728079 > 38 12.9384278480576125 + i 0.0238826907631012 > 39 17.5230234731441357 + i -0.1824807274488794 > 40 17.5503678395543901 + i 0.1785356473404145 > 41 17.6933953160112409 + i -0.0626631055149425 > 42 19.1692824404480930 + i -0.6351729266691462 > 43 19.4158452684509797 + i 0.3151965488807310 > 44 19.6020750507704591 + i -0.2559887580276014 > 45 19.6443102906562181 + i 1.1005601705485646 > 46 19.7948713379697452 + i 0.1230015140422697 > 47 19.8791098474284347 + i -0.1943322911563744 > 48 20.1268732265661860 + i -0.0340890856219265 > 49 20.1368588182453898 + i 0.2673370459460956 > > > Thanks in advance, > Sidd > Worcester Polytechnic, U.S.A. > Your B matrix has full rank, but rank(A)=1002. This means your eigenproblem has 27 zero eigenvalues. So no wonder that you get tiny eigenvalues (they are numerically zero in the working precision). Since you problem is Hermitian, you should solve it as such, adding -eps_gen_hermitian. Then the accuracy will be a bit better. Still, the relative error for the tiny eigenvalues will be large (division by zero), but you can see that the *absolute* error shows that the eigenvector is computed correctly. See output below. Another thing is that since your A matrix is singular, you should not use target=0, because then the solver will invert matrix A. MUMPS is good at working with singular matrices, but still the error will propagate to the computed eigensolutions. You have to move away from zero, e.g. target=0.1 Finally, do not set ncv=1029. By doing this, the built subspace is equal to the whole space of the original problem, so the cost will be even higher than calling LAPACK directly on your matrices. Let SLEPc choose the ncv value. Jose $ ./ex7 -f1 ~/tmp/pandey/A_mat_binary -f2 ~/tmp/pandey/B_mat_binary -st_pc_factor_mat_solver_package mumps -st_type sinvert -eps_target 0.1 -eps_nev 50 -eps_error_absolute ::ascii_info_detail Generalized eigenproblem stored in file. Reading COMPLEX matrices from binary files... ---------------------- -------------------- k ||Ax-kBx|| ---------------------- -------------------- 0.000000+0.000000i 7.65812e-12 0.000000-0.000000i 2.78585e-12 0.000000-0.000000i 6.30965e-12 0.000000+0.000000i 1.90477e-12 0.000000-0.000000i 4.18541e-12 0.000000+0.000000i 2.90717e-12 0.000000+0.000000i 2.90732e-12 0.000000-0.000000i 6.69097e-12 0.000000+0.000000i 2.01707e-12 0.000000-0.000000i 6.2849e-12 0.000000-0.000000i 6.30369e-12 0.000000-0.000000i 6.79209e-12 0.000000+0.000000i 3.3761e-12 0.000000-0.000000i 6.19772e-12 0.000000+0.000000i 3.87231e-12 0.000000-0.000000i 2.75338e-12 0.000000+0.000000i 9.50872e-12 0.000000-0.000000i 4.59981e-12 0.000000-0.000000i 7.62071e-13 0.000000-0.000000i 7.24146e-12 0.000000+0.000000i 2.98146e-12 0.000000-0.000000i 1.49165e-12 0.000000-0.000000i 1.33641e-12 0.000000-0.000000i 3.02551e-12 0.000000-0.000000i 1.76448e-12 0.000000+0.000000i 2.03659e-12 0.000000-0.000000i 5.71796e-12 5.238680+0.000000i 3.67926e-11 5.238680+0.000000i 3.85213e-11 5.238680+0.000000i 3.3809e-11 12.047661+0.000000i 3.74891e-11 12.047661+0.000000i 3.36437e-11 12.047661+0.000000i 4.68278e-11 12.079279+0.000000i 4.27835e-11 12.079279+0.000000i 7.02149e-11 12.079279-0.000000i 1.14514e-10 12.954275+0.000000i 1.13398e-10 12.954275-0.000000i 1.0298e-10 12.955799-0.000000i 6.2543e-11 17.515033+0.000000i 1.25423e-10 17.515033-0.000000i 6.11696e-11 17.647490+0.000000i 2.56307e-10 19.770424-0.000000i 1.43313e-07 19.770424-0.000000i 4.1167e-11 19.770424-0.000000i 7.67294e-11 19.770424-0.000000i 5.92692e-11 19.770424+0.000000i 7.66231e-11 19.770424+0.000000i 9.85743e-11 20.199838+0.000000i 8.37107e-11 20.199838+0.000000i 3.56701e-11 20.199838-0.000000i 8.82029e-11 20.199838-0.000000i 8.52395e-11 20.199838-0.000000i 8.82165e-11 20.199838-0.000000i 4.32813e-07 24.112262+0.000000i 2.25578e-10 24.112262-0.000000i 1.27094e-10 24.112262+0.000000i 1.14679e-10 24.227579-0.000000i 9.77701e-11 24.227579+0.000000i 1.41993e-10 24.227579-0.000000i 1.84562e-10 29.035473-0.000000i 2.69505e-10 29.035473-0.000000i 1.72167e-10 29.035473-0.000000i 1.87434e-10 29.649484-0.000000i 2.44039e-10 29.649484-0.000000i 3.84759e-10 32.112699+0.000000i 1.35182e-10 32.112699+0.000000i 1.91162e-10 32.112699-0.000000i 3.58118e-10 32.115257+0.000000i 3.58743e-10 32.115257-0.000000i 3.46267e-10 32.115257+0.000000i 4.55903e-10 50.121754+0.000000i 6.04554e-10 50.121754-0.000000i 6.26139e-10 ---------------------- -------------------- Number of iterations of the method: 3 Number of linear iterations of the method: 168 Solution method: krylovschur Number of requested eigenvalues: 50 Stopping condition: tol=1e-08, maxit=100 Linear eigensolve converged (73 eigenpairs) due to CONVERGED_TOL; iterations 3 ---------------------- -------------------- k ||Ax-kBx||/||kx|| ---------------------- -------------------- 0.000000+0.000000i 31.3252 0.000000-0.000000i 18.2211 0.000000-0.000000i 44.631 0.000000+0.000000i 13.836 0.000000-0.000000i 30.8722 0.000000+0.000000i 26.1132 0.000000+0.000000i 28.6077 0.000000-0.000000i 74.5526 0.000000+0.000000i 23.1291 0.000000-0.000000i 74.0475 0.000000-0.000000i 74.7671 0.000000-0.000000i 90.664 0.000000+0.000000i 45.6476 0.000000-0.000000i 84.4024 0.000000+0.000000i 55.9734 0.000000-0.000000i 42.2662 0.000000+0.000000i 155.367 0.000000-0.000000i 82.083 0.000000-0.000000i 14.409 0.000000-0.000000i 163.883 0.000000+0.000000i 70.2693 0.000000-0.000000i 38.2544 0.000000-0.000000i 35.2561 0.000000-0.000000i 87.8327 0.000000-0.000000i 73.0972 0.000000+0.000000i 102.899 0.000000-0.000000i 841.374 5.238680+0.000000i 7.02326e-12 5.238680+0.000000i 7.35325e-12 5.238680+0.000000i 6.45373e-12 12.047661+0.000000i 3.11174e-12 12.047661+0.000000i 2.79255e-12 12.047661+0.000000i 3.88688e-12 12.079279+0.000000i 3.54189e-12 12.079279+0.000000i 5.81284e-12 12.079279-0.000000i 9.48017e-12 12.954275+0.000000i 8.75372e-12 12.954275-0.000000i 7.94948e-12 12.955799-0.000000i 4.82741e-12 17.515033+0.000000i 7.16088e-12 17.515033-0.000000i 3.49241e-12 17.647490+0.000000i 1.45237e-11 19.770424-0.000000i 7.24883e-09 19.770424-0.000000i 2.08225e-12 19.770424-0.000000i 3.88102e-12 19.770424-0.000000i 2.99787e-12 19.770424+0.000000i 3.87565e-12 19.770424+0.000000i 4.98595e-12 20.199838+0.000000i 4.14413e-12 20.199838+0.000000i 1.76586e-12 20.199838-0.000000i 4.36652e-12 20.199838-0.000000i 4.21981e-12 20.199838-0.000000i 4.36719e-12 20.199838-0.000000i 2.14265e-08 24.112262+0.000000i 9.35532e-12 24.112262-0.000000i 5.27091e-12 24.112262+0.000000i 4.75603e-12 24.227579-0.000000i 4.03549e-12 24.227579+0.000000i 5.8608e-12 24.227579-0.000000i 7.61787e-12 29.035473-0.000000i 9.28191e-12 29.035473-0.000000i 5.92953e-12 29.035473-0.000000i 6.45536e-12 29.649484-0.000000i 8.23079e-12 29.649484-0.000000i 1.29769e-11 32.112699+0.000000i 4.20961e-12 32.112699+0.000000i 5.95285e-12 32.112699-0.000000i 1.11519e-11 32.115257+0.000000i 1.11705e-11 32.115257-0.000000i 1.0782e-11 32.115257+0.000000i 1.41958e-11 50.121754+0.000000i 1.20617e-11 50.121754-0.000000i 1.24924e-11 ---------------------- -------------------- From adrimacortes at gmail.com Thu Sep 14 09:30:25 2017 From: adrimacortes at gmail.com (=?UTF-8?Q?Adriano_C=C3=B4rtes?=) Date: Thu, 14 Sep 2017 11:30:25 -0300 Subject: [petsc-users] SNES ex12 visualization Message-ID: Dear all, I am running the SNES ex12 and I'm passing the options -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append to generate an output file. The .h5 file is generated, but I'm not being able to load it in Paraview (5.4.0-64bits). Paraview recognizes the file and offers severel options to read it, here is the complete list Chombo Files GTC Files M3DC1 Files Multilevel 3D Plasma Files PFLOTRAN Files Pixie Files Tetrad Files UNIC Files VizSchema Files The problem is none of the options above work :( I'm using the configure option '-download-hdf5' and it installs hdf5 version 1.8.18 Any hint of how to fix it and have the visualization working? Best regards, Adriano. -- Adriano C?rtes ================================================= *Campus Duque de Caxias and* *High-performance Computing Center (NACAD/COPPE)* Federal University of Rio de Janeiro (UFRJ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 14 10:00:41 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 14 Sep 2017 11:00:41 -0400 Subject: [petsc-users] SNES ex12 visualization In-Reply-To: References: Message-ID: On Thu, Sep 14, 2017 at 10:30 AM, Adriano C?rtes wrote: > Dear all, > > I am running the SNES ex12 and I'm passing the options -dm_view > hdf5:sol.h5 -vec_view hdf5:sol.h5::append to generate an output file. The > .h5 file is generated, but I'm not being able to load it in Paraview > (5.4.0-64bits). Paraview recognizes the file and offers severel options to > read it, here is the complete list > > Chombo Files > GTC Files > M3DC1 Files > Multilevel 3D Plasma Files > PFLOTRAN Files > Pixie Files > Tetrad Files > UNIC Files > VizSchema Files > > The problem is none of the options above work :( > I'm using the configure option '-download-hdf5' and it installs hdf5 > version 1.8.18 > Any hint of how to fix it and have the visualization working? > Yes, Paraview does not directly read HDF5. It needs you to tell it what the data in the HDF5 file means. You do this by creating a *.xdmf file, which is XML. We provide a tool $PETSC_DIR/bin/petsc_gen_xdmf.py which should automatically produce this file for you. Let us know if it does not work. Thanks, Matt > > Best regards, > Adriano. > > -- > Adriano C?rtes > ================================================= > *Campus Duque de Caxias and* > *High-performance Computing Center (NACAD/COPPE)* > Federal University of Rio de Janeiro (UFRJ) > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From adrimacortes at gmail.com Thu Sep 14 10:43:44 2017 From: adrimacortes at gmail.com (=?UTF-8?Q?Adriano_C=C3=B4rtes?=) Date: Thu, 14 Sep 2017 12:43:44 -0300 Subject: [petsc-users] SNES ex12 visualization In-Reply-To: References: Message-ID: Dear Matthew, Thank you for your return. It worked, but this prompts another question. So why PetscViewer does not write both files (.h5 and .xmf) directly, instead of having to post-proc the .h5 file (in serial)? And what about big 3D simulations? PETSc always serialize the output of the distributed dmplex? Is there a way to output one .h5 per mesh partition? Best regards, Adriano. 2017-09-14 12:00 GMT-03:00 Matthew Knepley : > On Thu, Sep 14, 2017 at 10:30 AM, Adriano C?rtes > wrote: > >> Dear all, >> >> I am running the SNES ex12 and I'm passing the options -dm_view >> hdf5:sol.h5 -vec_view hdf5:sol.h5::append to generate an output file. The >> .h5 file is generated, but I'm not being able to load it in Paraview >> (5.4.0-64bits). Paraview recognizes the file and offers severel options to >> read it, here is the complete list >> >> Chombo Files >> GTC Files >> M3DC1 Files >> Multilevel 3D Plasma Files >> PFLOTRAN Files >> Pixie Files >> Tetrad Files >> UNIC Files >> VizSchema Files >> >> The problem is none of the options above work :( >> I'm using the configure option '-download-hdf5' and it installs hdf5 >> version 1.8.18 >> Any hint of how to fix it and have the visualization working? >> > > Yes, Paraview does not directly read HDF5. It needs you to tell it what > the data in the HDF5 file means. You do > this by creating a *.xdmf file, which is XML. We provide a tool > > $PETSC_DIR/bin/petsc_gen_xdmf.py > > which should automatically produce this file for you. Let us know if it > does not work. > > Thanks, > > Matt > > >> >> Best regards, >> Adriano. >> >> -- >> Adriano C?rtes >> ================================================= >> *Campus Duque de Caxias and* >> *High-performance Computing Center (NACAD/COPPE)* >> Federal University of Rio de Janeiro (UFRJ) >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > -- Adriano C?rtes ================================================= *Campus Duque de Caxias and* *High-performance Computing Center (NACAD/COPPE)* Federal University of Rio de Janeiro (UFRJ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 14 10:47:26 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 14 Sep 2017 11:47:26 -0400 Subject: [petsc-users] SNES ex12 visualization In-Reply-To: References: Message-ID: On Thu, Sep 14, 2017 at 11:43 AM, Adriano C?rtes wrote: > Dear Matthew, > > Thank you for your return. It worked, but this prompts another question. > So why PetscViewer does not write both files (.h5 and .xmf) directly, > instead of having to post-proc the .h5 file (in serial)? > 1) Maintenance: Changing the Python is much easier than changing the C you would add to generate it 2) Performance: On big parallel system, writing files is expensive so I wanted to minimize what I had to do. 3) Robustness: Moving 1 file around is much easier than remembering 2. I just always regenerate the xdmf when needed. > And what about big 3D simulations? PETSc always serialize the output of > the distributed dmplex? Is there a way to output one .h5 per mesh > partition? > Given the way I/O is structured on big machines, we believe the multiple file route is a huge mistake. Also, all our measurements say that sending some data on the network is not noticeable given the disk access costs. Thanks, Matt > Best regards, > Adriano. > > > 2017-09-14 12:00 GMT-03:00 Matthew Knepley : > >> On Thu, Sep 14, 2017 at 10:30 AM, Adriano C?rtes >> wrote: >> >>> Dear all, >>> >>> I am running the SNES ex12 and I'm passing the options -dm_view >>> hdf5:sol.h5 -vec_view hdf5:sol.h5::append to generate an output file. The >>> .h5 file is generated, but I'm not being able to load it in Paraview >>> (5.4.0-64bits). Paraview recognizes the file and offers severel options to >>> read it, here is the complete list >>> >>> Chombo Files >>> GTC Files >>> M3DC1 Files >>> Multilevel 3D Plasma Files >>> PFLOTRAN Files >>> Pixie Files >>> Tetrad Files >>> UNIC Files >>> VizSchema Files >>> >>> The problem is none of the options above work :( >>> I'm using the configure option '-download-hdf5' and it installs hdf5 >>> version 1.8.18 >>> Any hint of how to fix it and have the visualization working? >>> >> >> Yes, Paraview does not directly read HDF5. It needs you to tell it what >> the data in the HDF5 file means. You do >> this by creating a *.xdmf file, which is XML. We provide a tool >> >> $PETSC_DIR/bin/petsc_gen_xdmf.py >> >> which should automatically produce this file for you. Let us know if it >> does not work. >> >> Thanks, >> >> Matt >> >> >>> >>> Best regards, >>> Adriano. >>> >>> -- >>> Adriano C?rtes >>> ================================================= >>> *Campus Duque de Caxias and* >>> *High-performance Computing Center (NACAD/COPPE)* >>> Federal University of Rio de Janeiro (UFRJ) >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ >> > > > > -- > Adriano C?rtes > ================================================= > *Campus Duque de Caxias and* > *High-performance Computing Center (NACAD/COPPE)* > Federal University of Rio de Janeiro (UFRJ) > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Thu Sep 14 11:10:53 2017 From: fande.kong at inl.gov (Kong, Fande) Date: Thu, 14 Sep 2017 10:10:53 -0600 Subject: [petsc-users] SNES ex12 visualization In-Reply-To: References: Message-ID: On Thu, Sep 14, 2017 at 9:47 AM, Matthew Knepley wrote: > On Thu, Sep 14, 2017 at 11:43 AM, Adriano C?rtes > wrote: > >> Dear Matthew, >> >> Thank you for your return. It worked, but this prompts another question. >> So why PetscViewer does not write both files (.h5 and .xmf) directly, >> instead of having to post-proc the .h5 file (in serial)? >> > > 1) Maintenance: Changing the Python is much easier than changing the C you > would add to generate it > > 2) Performance: On big parallel system, writing files is expensive so I > wanted to minimize what I had to do. > > 3) Robustness: Moving 1 file around is much easier than remembering 2. I > just always regenerate the xdmf when needed. > > >> And what about big 3D simulations? PETSc always serialize the output of >> the distributed dmplex? Is there a way to output one .h5 per mesh >> partition? >> > > Given the way I/O is structured on big machines, we believe the multiple > file route is a huge mistake. Also, all our measurements > say that sending some data on the network is not noticeable given the disk > access costs. > I have slightly different things here. We tried the serial output, it looks really slow for large-scale problems, and the first processor often runs out of memory because of gathering all data from other processor cores. The parallel IO runs smoothly and much faster than I excepted. We have done experiments with ten thousands of cores for a problem with 1 billion of unknowns. I did not see any concern so far. Fande, > > Thanks, > > Matt > > >> Best regards, >> Adriano. >> >> >> 2017-09-14 12:00 GMT-03:00 Matthew Knepley : >> >>> On Thu, Sep 14, 2017 at 10:30 AM, Adriano C?rtes >> > wrote: >>> >>>> Dear all, >>>> >>>> I am running the SNES ex12 and I'm passing the options -dm_view >>>> hdf5:sol.h5 -vec_view hdf5:sol.h5::append to generate an output file. The >>>> .h5 file is generated, but I'm not being able to load it in Paraview >>>> (5.4.0-64bits). Paraview recognizes the file and offers severel options to >>>> read it, here is the complete list >>>> >>>> Chombo Files >>>> GTC Files >>>> M3DC1 Files >>>> Multilevel 3D Plasma Files >>>> PFLOTRAN Files >>>> Pixie Files >>>> Tetrad Files >>>> UNIC Files >>>> VizSchema Files >>>> >>>> The problem is none of the options above work :( >>>> I'm using the configure option '-download-hdf5' and it installs hdf5 >>>> version 1.8.18 >>>> Any hint of how to fix it and have the visualization working? >>>> >>> >>> Yes, Paraview does not directly read HDF5. It needs you to tell it what >>> the data in the HDF5 file means. You do >>> this by creating a *.xdmf file, which is XML. We provide a tool >>> >>> $PETSC_DIR/bin/petsc_gen_xdmf.py >>> >>> which should automatically produce this file for you. Let us know if it >>> does not work. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> >>>> Best regards, >>>> Adriano. >>>> >>>> -- >>>> Adriano C?rtes >>>> ================================================= >>>> *Campus Duque de Caxias and* >>>> *High-performance Computing Center (NACAD/COPPE)* >>>> Federal University of Rio de Janeiro (UFRJ) >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> http://www.caam.rice.edu/~mk51/ >>> >>> >> >> >> >> -- >> Adriano C?rtes >> ================================================= >> *Campus Duque de Caxias and* >> *High-performance Computing Center (NACAD/COPPE)* >> Federal University of Rio de Janeiro (UFRJ) >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Sep 14 11:29:15 2017 From: jed at jedbrown.org (Jed Brown) Date: Thu, 14 Sep 2017 10:29:15 -0600 Subject: [petsc-users] SNES ex12 visualization In-Reply-To: References: Message-ID: <87mv5xqm5g.fsf@jedbrown.org> "Kong, Fande" writes: >> Given the way I/O is structured on big machines, we believe the multiple >> file route is a huge mistake. Also, all our measurements >> say that sending some data on the network is not noticeable given the disk >> access costs. >> > > I have slightly different things here. We tried the serial output, it looks > really slow for large-scale problems, and the first processor often runs > out of memory because of gathering all data from other processor cores. The > parallel IO runs smoothly and much faster than I excepted. We have done > experiments with ten thousands of cores for a problem with 1 billion of > unknowns. I did not see any concern so far. I think there are two different issues here. Writing a separate file per MPI rank (often also per time step, etc.) creates a filesystem *metadata* bottleneck. It's the open() and close() that are more painful than the write() when you have lots of files. (You'd also want to be careful about your naming convention because merely running "ls" on a directory with many files is usually quite painful.) MPI-IO collectives offer a solution -- each rank writes parts of a file efficiently using the parallel file system. MPI-IO was introduced in MPI-2 (standardized in 1997) and PETSc has thus far avoided a hard dependency on this standard because some implementations were very slow to adopt it. In my opinion, any IO in PETSc that is intended to be highly scalable should use MPI-IO. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From bsmith at mcs.anl.gov Thu Sep 14 11:35:29 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 14 Sep 2017 11:35:29 -0500 Subject: [petsc-users] SNES ex12 visualization In-Reply-To: References: Message-ID: > On Sep 14, 2017, at 11:10 AM, Kong, Fande wrote: > > > > On Thu, Sep 14, 2017 at 9:47 AM, Matthew Knepley wrote: > On Thu, Sep 14, 2017 at 11:43 AM, Adriano C?rtes wrote: > Dear Matthew, > > Thank you for your return. It worked, but this prompts another question. So why PetscViewer does not write both files (.h5 and .xmf) directly, instead of having to post-proc the .h5 file (in serial)? > > 1) Maintenance: Changing the Python is much easier than changing the C you would add to generate it > > 2) Performance: On big parallel system, writing files is expensive so I wanted to minimize what I had to do. > > 3) Robustness: Moving 1 file around is much easier than remembering 2. I just always regenerate the xdmf when needed. > > And what about big 3D simulations? PETSc always serialize the output of the distributed dmplex? Is there a way to output one .h5 per mesh partition? > > Given the way I/O is structured on big machines, we believe the multiple file route is a huge mistake. Also, all our measurements > say that sending some data on the network is not noticeable given the disk access costs. > > I have slightly different things here. We tried the serial output, it looks really slow for large-scale problems, and the first processor often runs out of memory because of gathering all data from other processor cores. Where in PETSc is this? What type of viewer? Is there an example that reproduces the problem? Even when we do not use MPI IO in PETSc we attempt to not "put the entire object on the first process" so memory should not be an issue. For example VecVew() should memory scale both with or without MPI IO > The parallel IO runs smoothly and much faster than I excepted. We have done experiments with ten thousands of cores for a problem with 1 billion of unknowns. Is this your own canned IO or something in PETSc? > I did not see any concern so far. Ten thousand files is possibly manageable but I question 2 million. > > > Fande, > > > Thanks, > > Matt > > Best regards, > Adriano. > > > 2017-09-14 12:00 GMT-03:00 Matthew Knepley : > On Thu, Sep 14, 2017 at 10:30 AM, Adriano C?rtes wrote: > Dear all, > > I am running the SNES ex12 and I'm passing the options -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append to generate an output file. The .h5 file is generated, but I'm not being able to load it in Paraview (5.4.0-64bits). Paraview recognizes the file and offers severel options to read it, here is the complete list > > Chombo Files > GTC Files > M3DC1 Files > Multilevel 3D Plasma Files > PFLOTRAN Files > Pixie Files > Tetrad Files > UNIC Files > VizSchema Files > > The problem is none of the options above work :( > I'm using the configure option '-download-hdf5' and it installs hdf5 version 1.8.18 > Any hint of how to fix it and have the visualization working? > > Yes, Paraview does not directly read HDF5. It needs you to tell it what the data in the HDF5 file means. You do > this by creating a *.xdmf file, which is XML. We provide a tool > > $PETSC_DIR/bin/petsc_gen_xdmf.py > > which should automatically produce this file for you. Let us know if it does not work. > > Thanks, > > Matt > > > Best regards, > Adriano. > > -- > Adriano C?rtes > ================================================= > Campus Duque de Caxias and > High-performance Computing Center (NACAD/COPPE) > Federal University of Rio de Janeiro (UFRJ) > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > > > > -- > Adriano C?rtes > ================================================= > Campus Duque de Caxias and > High-performance Computing Center (NACAD/COPPE) > Federal University of Rio de Janeiro (UFRJ) > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > From fande.kong at inl.gov Thu Sep 14 12:07:24 2017 From: fande.kong at inl.gov (Kong, Fande) Date: Thu, 14 Sep 2017 11:07:24 -0600 Subject: [petsc-users] SNES ex12 visualization In-Reply-To: References: Message-ID: On Thu, Sep 14, 2017 at 10:35 AM, Barry Smith wrote: > > > On Sep 14, 2017, at 11:10 AM, Kong, Fande wrote: > > > > > > > > On Thu, Sep 14, 2017 at 9:47 AM, Matthew Knepley > wrote: > > On Thu, Sep 14, 2017 at 11:43 AM, Adriano C?rtes > wrote: > > Dear Matthew, > > > > Thank you for your return. It worked, but this prompts another question. > So why PetscViewer does not write both files (.h5 and .xmf) directly, > instead of having to post-proc the .h5 file (in serial)? > > > > 1) Maintenance: Changing the Python is much easier than changing the C > you would add to generate it > > > > 2) Performance: On big parallel system, writing files is expensive so I > wanted to minimize what I had to do. > > > > 3) Robustness: Moving 1 file around is much easier than remembering 2. I > just always regenerate the xdmf when needed. > > > > And what about big 3D simulations? PETSc always serialize the output of > the distributed dmplex? Is there a way to output one .h5 per mesh partition? > > > > Given the way I/O is structured on big machines, we believe the multiple > file route is a huge mistake. Also, all our measurements > > say that sending some data on the network is not noticeable given the > disk access costs. > > > > I have slightly different things here. We tried the serial output, it > looks really slow for large-scale problems, and the first processor often > runs out of memory because of gathering all data from other processor cores. > > Where in PETSc is this? What type of viewer? Is there an example that > reproduces the problem? Even when we do not use MPI IO in PETSc we attempt > to not "put the entire object on the first process" so memory should not be > an issue. For example VecVew() should memory scale both with or without MPI > IO > We manually gather all data to the first processor core, and write it as a single vtk file. > > > > The parallel IO runs smoothly and much faster than I excepted. We have > done experiments with ten thousands of cores for a problem with 1 billion > of unknowns. > > Is this your own canned IO or something in PETSc? > We implement the writer based on the ISView and VecView with HDF5 viewer in PETSc to output all data as a single HDF. ISView and VecView do the magic job for me. > > > I did not see any concern so far. > > Ten thousand files is possibly manageable but I question 2 million. > Just one single HDF5 file. Fande, > > > > > > > Fande, > > > > > > Thanks, > > > > Matt > > > > Best regards, > > Adriano. > > > > > > 2017-09-14 12:00 GMT-03:00 Matthew Knepley : > > On Thu, Sep 14, 2017 at 10:30 AM, Adriano C?rtes > wrote: > > Dear all, > > > > I am running the SNES ex12 and I'm passing the options -dm_view > hdf5:sol.h5 -vec_view hdf5:sol.h5::append to generate an output file. The > .h5 file is generated, but I'm not being able to load it in Paraview > (5.4.0-64bits). Paraview recognizes the file and offers severel options to > read it, here is the complete list > > > > Chombo Files > > GTC Files > > M3DC1 Files > > Multilevel 3D Plasma Files > > PFLOTRAN Files > > Pixie Files > > Tetrad Files > > UNIC Files > > VizSchema Files > > > > The problem is none of the options above work :( > > I'm using the configure option '-download-hdf5' and it installs hdf5 > version 1.8.18 > > Any hint of how to fix it and have the visualization working? > > > > Yes, Paraview does not directly read HDF5. It needs you to tell it what > the data in the HDF5 file means. You do > > this by creating a *.xdmf file, which is XML. We provide a tool > > > > $PETSC_DIR/bin/petsc_gen_xdmf.py > > > > which should automatically produce this file for you. Let us know if it > does not work. > > > > Thanks, > > > > Matt > > > > > > Best regards, > > Adriano. > > > > -- > > Adriano C?rtes > > ================================================= > > Campus Duque de Caxias and > > High-performance Computing Center (NACAD/COPPE) > > Federal University of Rio de Janeiro (UFRJ) > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www. > caam.rice.edu_-7Emk51_&d=DwIFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m= > YTLjMkjfS0tYLZ3RxmJFoe8BT56h48axFCzaadZwBXA&s= > iLsaHQugaY4gj4DKE9gq8XdBt7q3ejdpDRfJ8RFerE0&e= > > > > > > > > -- > > Adriano C?rtes > > ================================================= > > Campus Duque de Caxias and > > High-performance Computing Center (NACAD/COPPE) > > Federal University of Rio de Janeiro (UFRJ) > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www. > caam.rice.edu_-7Emk51_&d=DwIFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m= > YTLjMkjfS0tYLZ3RxmJFoe8BT56h48axFCzaadZwBXA&s= > iLsaHQugaY4gj4DKE9gq8XdBt7q3ejdpDRfJ8RFerE0&e= > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 14 12:26:47 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 14 Sep 2017 13:26:47 -0400 Subject: [petsc-users] SNES ex12 visualization In-Reply-To: References: Message-ID: On Thu, Sep 14, 2017 at 1:07 PM, Kong, Fande wrote: > > > On Thu, Sep 14, 2017 at 10:35 AM, Barry Smith wrote: > >> >> > On Sep 14, 2017, at 11:10 AM, Kong, Fande wrote: >> > >> > >> > >> > On Thu, Sep 14, 2017 at 9:47 AM, Matthew Knepley >> wrote: >> > On Thu, Sep 14, 2017 at 11:43 AM, Adriano C?rtes < >> adrimacortes at gmail.com> wrote: >> > Dear Matthew, >> > >> > Thank you for your return. It worked, but this prompts another >> question. So why PetscViewer does not write both files (.h5 and .xmf) >> directly, instead of having to post-proc the .h5 file (in serial)? >> > >> > 1) Maintenance: Changing the Python is much easier than changing the C >> you would add to generate it >> > >> > 2) Performance: On big parallel system, writing files is expensive so I >> wanted to minimize what I had to do. >> > >> > 3) Robustness: Moving 1 file around is much easier than remembering 2. >> I just always regenerate the xdmf when needed. >> > >> > And what about big 3D simulations? PETSc always serialize the output of >> the distributed dmplex? Is there a way to output one .h5 per mesh partition? >> > >> > Given the way I/O is structured on big machines, we believe the >> multiple file route is a huge mistake. Also, all our measurements >> > say that sending some data on the network is not noticeable given the >> disk access costs. >> > >> > I have slightly different things here. We tried the serial output, it >> looks really slow for large-scale problems, and the first processor often >> runs out of memory because of gathering all data from other processor cores. >> >> Where in PETSc is this? What type of viewer? Is there an example that >> reproduces the problem? Even when we do not use MPI IO in PETSc we attempt >> to not "put the entire object on the first process" so memory should not be >> an issue. For example VecVew() should memory scale both with or without MPI >> IO >> > > We manually gather all data to the first processor core, and write it as a > single vtk file. > Of course I am not doing that. I reduce everything to an ISView or a VecView call. That way it uses MPI I/O if its turned on. Matt > >> >> > The parallel IO runs smoothly and much faster than I excepted. We have >> done experiments with ten thousands of cores for a problem with 1 billion >> of unknowns. >> >> Is this your own canned IO or something in PETSc? >> > > We implement the writer based on the ISView and VecView with HDF5 viewer > in PETSc to output all data as a single HDF. ISView and VecView do the > magic job for me. > > > >> >> > I did not see any concern so far. >> >> Ten thousand files is possibly manageable but I question 2 million. >> > > Just one single HDF5 file. > > Fande, > > >> >> > >> > >> > Fande, >> > >> > >> > Thanks, >> > >> > Matt >> > >> > Best regards, >> > Adriano. >> > >> > >> > 2017-09-14 12:00 GMT-03:00 Matthew Knepley : >> > On Thu, Sep 14, 2017 at 10:30 AM, Adriano C?rtes < >> adrimacortes at gmail.com> wrote: >> > Dear all, >> > >> > I am running the SNES ex12 and I'm passing the options -dm_view >> hdf5:sol.h5 -vec_view hdf5:sol.h5::append to generate an output file. The >> .h5 file is generated, but I'm not being able to load it in Paraview >> (5.4.0-64bits). Paraview recognizes the file and offers severel options to >> read it, here is the complete list >> > >> > Chombo Files >> > GTC Files >> > M3DC1 Files >> > Multilevel 3D Plasma Files >> > PFLOTRAN Files >> > Pixie Files >> > Tetrad Files >> > UNIC Files >> > VizSchema Files >> > >> > The problem is none of the options above work :( >> > I'm using the configure option '-download-hdf5' and it installs hdf5 >> version 1.8.18 >> > Any hint of how to fix it and have the visualization working? >> > >> > Yes, Paraview does not directly read HDF5. It needs you to tell it what >> the data in the HDF5 file means. You do >> > this by creating a *.xdmf file, which is XML. We provide a tool >> > >> > $PETSC_DIR/bin/petsc_gen_xdmf.py >> > >> > which should automatically produce this file for you. Let us know if it >> does not work. >> > >> > Thanks, >> > >> > Matt >> > >> > >> > Best regards, >> > Adriano. >> > >> > -- >> > Adriano C?rtes >> > ================================================= >> > Campus Duque de Caxias and >> > High-performance Computing Center (NACAD/COPPE) >> > Federal University of Rio de Janeiro (UFRJ) >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> > >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.caam >> .rice.edu_-7Emk51_&d=DwIFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ >> _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY& >> m=YTLjMkjfS0tYLZ3RxmJFoe8BT56h48axFCzaadZwBXA&s=iLsaHQugaY4g >> j4DKE9gq8XdBt7q3ejdpDRfJ8RFerE0&e= >> > >> > >> > >> > -- >> > Adriano C?rtes >> > ================================================= >> > Campus Duque de Caxias and >> > High-performance Computing Center (NACAD/COPPE) >> > Federal University of Rio de Janeiro (UFRJ) >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> > >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.caam >> .rice.edu_-7Emk51_&d=DwIFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ >> _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY& >> m=YTLjMkjfS0tYLZ3RxmJFoe8BT56h48axFCzaadZwBXA&s=iLsaHQugaY4g >> j4DKE9gq8XdBt7q3ejdpDRfJ8RFerE0&e= >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Thu Sep 14 12:56:15 2017 From: fande.kong at inl.gov (Kong, Fande) Date: Thu, 14 Sep 2017 11:56:15 -0600 Subject: [petsc-users] SNES ex12 visualization In-Reply-To: References: Message-ID: On Thu, Sep 14, 2017 at 11:26 AM, Matthew Knepley wrote: > On Thu, Sep 14, 2017 at 1:07 PM, Kong, Fande wrote: > >> >> >> On Thu, Sep 14, 2017 at 10:35 AM, Barry Smith wrote: >> >>> >>> > On Sep 14, 2017, at 11:10 AM, Kong, Fande wrote: >>> > >>> > >>> > >>> > On Thu, Sep 14, 2017 at 9:47 AM, Matthew Knepley >>> wrote: >>> > On Thu, Sep 14, 2017 at 11:43 AM, Adriano C?rtes < >>> adrimacortes at gmail.com> wrote: >>> > Dear Matthew, >>> > >>> > Thank you for your return. It worked, but this prompts another >>> question. So why PetscViewer does not write both files (.h5 and .xmf) >>> directly, instead of having to post-proc the .h5 file (in serial)? >>> > >>> > 1) Maintenance: Changing the Python is much easier than changing the C >>> you would add to generate it >>> > >>> > 2) Performance: On big parallel system, writing files is expensive so >>> I wanted to minimize what I had to do. >>> > >>> > 3) Robustness: Moving 1 file around is much easier than remembering 2. >>> I just always regenerate the xdmf when needed. >>> > >>> > And what about big 3D simulations? PETSc always serialize the output >>> of the distributed dmplex? Is there a way to output one .h5 per mesh >>> partition? >>> > >>> > Given the way I/O is structured on big machines, we believe the >>> multiple file route is a huge mistake. Also, all our measurements >>> > say that sending some data on the network is not noticeable given the >>> disk access costs. >>> > >>> > I have slightly different things here. We tried the serial output, it >>> looks really slow for large-scale problems, and the first processor often >>> runs out of memory because of gathering all data from other processor cores. >>> >>> Where in PETSc is this? What type of viewer? Is there an example that >>> reproduces the problem? Even when we do not use MPI IO in PETSc we attempt >>> to not "put the entire object on the first process" so memory should not be >>> an issue. For example VecVew() should memory scale both with or without MPI >>> IO >>> >> >> We manually gather all data to the first processor core, and write it as >> a single vtk file. >> > > Of course I am not doing that. I reduce everything to an ISView or a > VecView call. That way it uses MPI I/O if its turned on. > I meant Fande manually gathers all data to the first processor core in his in-house code. > > Matt > > >> >>> >>> > The parallel IO runs smoothly and much faster than I excepted. We have >>> done experiments with ten thousands of cores for a problem with 1 billion >>> of unknowns. >>> >>> Is this your own canned IO or something in PETSc? >>> >> >> We implement the writer based on the ISView and VecView with HDF5 viewer >> in PETSc to output all data as a single HDF. ISView and VecView do the >> magic job for me. >> >> >> >>> >>> > I did not see any concern so far. >>> >>> Ten thousand files is possibly manageable but I question 2 million. >>> >> >> Just one single HDF5 file. >> >> Fande, >> >> >>> >>> > >>> > >>> > Fande, >>> > >>> > >>> > Thanks, >>> > >>> > Matt >>> > >>> > Best regards, >>> > Adriano. >>> > >>> > >>> > 2017-09-14 12:00 GMT-03:00 Matthew Knepley : >>> > On Thu, Sep 14, 2017 at 10:30 AM, Adriano C?rtes < >>> adrimacortes at gmail.com> wrote: >>> > Dear all, >>> > >>> > I am running the SNES ex12 and I'm passing the options -dm_view >>> hdf5:sol.h5 -vec_view hdf5:sol.h5::append to generate an output file. The >>> .h5 file is generated, but I'm not being able to load it in Paraview >>> (5.4.0-64bits). Paraview recognizes the file and offers severel options to >>> read it, here is the complete list >>> > >>> > Chombo Files >>> > GTC Files >>> > M3DC1 Files >>> > Multilevel 3D Plasma Files >>> > PFLOTRAN Files >>> > Pixie Files >>> > Tetrad Files >>> > UNIC Files >>> > VizSchema Files >>> > >>> > The problem is none of the options above work :( >>> > I'm using the configure option '-download-hdf5' and it installs hdf5 >>> version 1.8.18 >>> > Any hint of how to fix it and have the visualization working? >>> > >>> > Yes, Paraview does not directly read HDF5. It needs you to tell it >>> what the data in the HDF5 file means. You do >>> > this by creating a *.xdmf file, which is XML. We provide a tool >>> > >>> > $PETSC_DIR/bin/petsc_gen_xdmf.py >>> > >>> > which should automatically produce this file for you. Let us know if >>> it does not work. >>> > >>> > Thanks, >>> > >>> > Matt >>> > >>> > >>> > Best regards, >>> > Adriano. >>> > >>> > -- >>> > Adriano C?rtes >>> > ================================================= >>> > Campus Duque de Caxias and >>> > High-performance Computing Center (NACAD/COPPE) >>> > Federal University of Rio de Janeiro (UFRJ) >>> > >>> > >>> > >>> > -- >>> > What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> > -- Norbert Wiener >>> > >>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.caam >>> .rice.edu_-7Emk51_&d=DwIFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ >>> _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY& >>> m=YTLjMkjfS0tYLZ3RxmJFoe8BT56h48axFCzaadZwBXA&s=iLsaHQugaY4g >>> j4DKE9gq8XdBt7q3ejdpDRfJ8RFerE0&e= >>> > >>> > >>> > >>> > -- >>> > Adriano C?rtes >>> > ================================================= >>> > Campus Duque de Caxias and >>> > High-performance Computing Center (NACAD/COPPE) >>> > Federal University of Rio de Janeiro (UFRJ) >>> > >>> > >>> > >>> > -- >>> > What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> > -- Norbert Wiener >>> > >>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.caam >>> .rice.edu_-7Emk51_&d=DwIFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ >>> _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY& >>> m=YTLjMkjfS0tYLZ3RxmJFoe8BT56h48axFCzaadZwBXA&s=iLsaHQugaY4g >>> j4DKE9gq8XdBt7q3ejdpDRfJ8RFerE0&e= >>> > >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davegwebb10 at gmail.com Thu Sep 14 15:07:11 2017 From: davegwebb10 at gmail.com (David Gross) Date: Thu, 14 Sep 2017 22:07:11 +0200 Subject: [petsc-users] Matrix dot product In-Reply-To: References: Message-ID: Hi Matt, Noted for the naming convention. Yes, it is a Newton method (see Pan, V. and Reif, J., Fast and Efficient Parallel Solution of Dense Linear Systems, Computers Math. Applications. Vol. 17, No. 11, pp. 1481-1491, 1989) The dense matrix I have is repeatedly inverted while slowly changing such that the previous inverse is a near perfect guess for the new inverse. Regards, Dave On Thu, Sep 14, 2017 at 2:49 AM, Matthew Knepley wrote: > On Wed, Sep 13, 2017 at 6:14 PM, David Gross > wrote: > >> Hi Matt, >> Thank you for getting back to me. Your answer confirms what I thought in >> terms of existing functionality. I think it shouldn't be too hard to make a >> copy of MatAXPY to MatAXY where it performs Xij = A*Xij*Yij (or without the >> A). I could then do the MatNorm of the resulting matrix to get what I need. >> >> Is a MatAXY function desirable as a source contribution? >> > > Yes. I would prefer calling it MatPointwiseMult, since you can see it as > VecPointwiseMult on a Vec obtained > from forgetting the linear operator structure of the matrix (forgetful > functor). > > >> I am hoping to use PETSc for performing basic vector and matrix >> operations on dense matrices and 1D vectors. The main uses are matmult, >> matmatmult and matrix additions and scaling. The application is for >> implementing a parallel version on an existing Pan-Reif matrix inversion >> algorithm. >> > > Is this Newton's method on the vector space of matrices? > > Thanks, > > Matt > > >> The choice of using PETSc is mostly due to us already using it in the >> same program to solve sparse matrices (with MUMPS) with the goal of >> avoiding adding yet another package (ex ScaLAPACK/PBLAS) into the code even >> if other packages may be more directly oriented towards my application. >> >> Regards, >> Dave >> >> >> >> On Mon, Sep 11, 2017 at 2:19 AM, Matthew Knepley >> wrote: >> >>> On Sun, Sep 10, 2017 at 5:51 PM, David Gross >>> wrote: >>> >>>> Hello, >>>> I was wondering if there was a matrix equivalent to the vecDot function >>>> (Frobenius inner product)? As far as I can tell the closest thing is >>>> MatNorm with NORM_FROBENIUS, but obviously this is acting on only one >>>> matrix. >>>> >>>> If there is not a built in function, what is the best way to compute >>>> this? I am working fortran90. >>>> >>> >>> We do not have this. However, it would be trivial to add since we have >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpage >>> s/Mat/MatAXPY.html >>> >>> since you just replace + with * in our code. You could argue that we >>> should have written for >>> a general ring, but C makes this cumbersome. Do you think you could make >>> the change? >>> >>> What are you using this for? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Regards, >>>> Dave >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> http://www.caam.rice.edu/~mk51/ >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spandey2 at wpi.edu Thu Sep 14 17:09:59 2017 From: spandey2 at wpi.edu (Pandey, Siddhant) Date: Thu, 14 Sep 2017 22:09:59 +0000 Subject: [petsc-users] SLEPC - Hermitian matrix gives complex eigenvalues In-Reply-To: <071CC27A-4A0D-4529-8B62-67CA1A44042F@dsic.upv.es> References: <14C3EF90-C404-45AD-9516-63D3F9CA4D10@wpi.edu>, <071CC27A-4A0D-4529-8B62-67CA1A44042F@dsic.upv.es> Message-ID: How do I calculate the rank of my A matrix? I couldn't find anything in PETSC documentation. It seems like a mumps functionality, but I cannot get any output regarding null pivots/rank deficiency. Sidd CCNS, WPI ________________________________ From: Jose E. Roman Sent: Thursday, September 14, 2017 4:33:01 AM To: Pandey, Siddhant Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] SLEPC - Hermitian matrix gives complex eigenvalues > El 14 sept 2017, a las 0:27, Pandey, Siddhant escribi?: > > Hello all, > > I am diagonalizing a hermitian matrix (MatIsHermitian = 1 up to a tolerance of 1e-15) using krylovschur, with EPS_TARGET_MAGNITUDE = 0, but getting complex eigenvalues. Also, I am getting seemingly spurious eigenvalues of magnitude very close to 0, whose relative errors are much larger than my set tolerance. Can anyone indicate what might be the cause and possible solutions. > > I have attached the files eps_view, A_mat_binary, and B_mat_binary files, which show the settings I have used, and contain the A and B matrices respectively (in binary). > > The eigenvalues I get are: > > # > > Eigenvalue > ? ---------------------------------------------------------------- > 0 0.0000000000000077 + i -0.0000000000000000 > 1 0.0000000000000265 + i -0.0000000000000008 > 2 0.0000000000000340 + i 0.0000000000000002 > 3 0.0000000000000373 + i 0.0000000000000001 > 4 0.0000000000000444 + i -0.0000000000000007 > 5 0.0000000000000448 + i -0.0000000000000017 > 6 0.0000000000000470 + i 0.0000000000000002 > 7 0.0000000000000489 + i 0.0000000000000030 > 8 0.0000000000000548 + i 0.0000000000000006 > 9 0.0000000000000585 + i 0.0000000000000001 > 10 0.0000000000000643 + i 0.0000000000000005 > 11 0.0000000000000714 + i -0.0000000000000008 > 12 0.0000000000000750 + i -0.0000000000000019 > 13 0.0000000000000752 + i 0.0000000000000031 > 14 0.0000000000000769 + i -0.0000000000000002 > 15 0.0000000000000784 + i -0.0000000000000037 > 16 0.0000000000000860 + i 0.0000000000000002 > 17 0.0000000000000880 + i -0.0000000000000004 > 18 0.0000000000000968 + i -0.0000000000000001 > 19 0.0000000000000979 + i 0.0000000000000013 > 20 0.0000000000001045 + i -0.0000000000000001 > 21 0.0000000000001150 + i -0.0000000000000011 > 22 0.0000000000001348 + i -0.0000000000000012 > 23 0.0000000000001446 + i -0.0000000000000019 > 24 0.0000000000001454 + i 0.0000000000000011 > 25 0.0000000000001555 + i 0.0000000000000003 > 26 0.0000000000002513 + i -0.0000000000000009 > 27 5.0908854514230413 + i -0.0022004178122762 > 28 5.2768106039842175 + i 0.1043464906789375 > 29 5.3003883062604187 + i -0.0757735907905433 > 30 11.5143655883932929 + i 0.0049838692042474 > 31 11.8821523259838653 + i -0.1475608751440501 > 32 11.9515216995487101 + i 0.1857395729336506 > 33 12.0909362158339384 + i 0.0049114285397287 > 34 12.0905704159492675 + i -0.1522880348537981 > 35 12.2205398661469111 + i -0.0781999802933937 > 36 12.4807156964720480 + i -0.2850122604907908 > 37 12.6082849940660289 + i 0.0560456679728079 > 38 12.9384278480576125 + i 0.0238826907631012 > 39 17.5230234731441357 + i -0.1824807274488794 > 40 17.5503678395543901 + i 0.1785356473404145 > 41 17.6933953160112409 + i -0.0626631055149425 > 42 19.1692824404480930 + i -0.6351729266691462 > 43 19.4158452684509797 + i 0.3151965488807310 > 44 19.6020750507704591 + i -0.2559887580276014 > 45 19.6443102906562181 + i 1.1005601705485646 > 46 19.7948713379697452 + i 0.1230015140422697 > 47 19.8791098474284347 + i -0.1943322911563744 > 48 20.1268732265661860 + i -0.0340890856219265 > 49 20.1368588182453898 + i 0.2673370459460956 > > > Thanks in advance, > Sidd > Worcester Polytechnic, U.S.A. > Your B matrix has full rank, but rank(A)=1002. This means your eigenproblem has 27 zero eigenvalues. So no wonder that you get tiny eigenvalues (they are numerically zero in the working precision). Since you problem is Hermitian, you should solve it as such, adding -eps_gen_hermitian. Then the accuracy will be a bit better. Still, the relative error for the tiny eigenvalues will be large (division by zero), but you can see that the *absolute* error shows that the eigenvector is computed correctly. See output below. Another thing is that since your A matrix is singular, you should not use target=0, because then the solver will invert matrix A. MUMPS is good at working with singular matrices, but still the error will propagate to the computed eigensolutions. You have to move away from zero, e.g. target=0.1 Finally, do not set ncv=1029. By doing this, the built subspace is equal to the whole space of the original problem, so the cost will be even higher than calling LAPACK directly on your matrices. Let SLEPc choose the ncv value. Jose $ ./ex7 -f1 ~/tmp/pandey/A_mat_binary -f2 ~/tmp/pandey/B_mat_binary -st_pc_factor_mat_solver_package mumps -st_type sinvert -eps_target 0.1 -eps_nev 50 -eps_error_absolute ::ascii_info_detail Generalized eigenproblem stored in file. Reading COMPLEX matrices from binary files... ---------------------- -------------------- k ||Ax-kBx|| ---------------------- -------------------- 0.000000+0.000000i 7.65812e-12 0.000000-0.000000i 2.78585e-12 0.000000-0.000000i 6.30965e-12 0.000000+0.000000i 1.90477e-12 0.000000-0.000000i 4.18541e-12 0.000000+0.000000i 2.90717e-12 0.000000+0.000000i 2.90732e-12 0.000000-0.000000i 6.69097e-12 0.000000+0.000000i 2.01707e-12 0.000000-0.000000i 6.2849e-12 0.000000-0.000000i 6.30369e-12 0.000000-0.000000i 6.79209e-12 0.000000+0.000000i 3.3761e-12 0.000000-0.000000i 6.19772e-12 0.000000+0.000000i 3.87231e-12 0.000000-0.000000i 2.75338e-12 0.000000+0.000000i 9.50872e-12 0.000000-0.000000i 4.59981e-12 0.000000-0.000000i 7.62071e-13 0.000000-0.000000i 7.24146e-12 0.000000+0.000000i 2.98146e-12 0.000000-0.000000i 1.49165e-12 0.000000-0.000000i 1.33641e-12 0.000000-0.000000i 3.02551e-12 0.000000-0.000000i 1.76448e-12 0.000000+0.000000i 2.03659e-12 0.000000-0.000000i 5.71796e-12 5.238680+0.000000i 3.67926e-11 5.238680+0.000000i 3.85213e-11 5.238680+0.000000i 3.3809e-11 12.047661+0.000000i 3.74891e-11 12.047661+0.000000i 3.36437e-11 12.047661+0.000000i 4.68278e-11 12.079279+0.000000i 4.27835e-11 12.079279+0.000000i 7.02149e-11 12.079279-0.000000i 1.14514e-10 12.954275+0.000000i 1.13398e-10 12.954275-0.000000i 1.0298e-10 12.955799-0.000000i 6.2543e-11 17.515033+0.000000i 1.25423e-10 17.515033-0.000000i 6.11696e-11 17.647490+0.000000i 2.56307e-10 19.770424-0.000000i 1.43313e-07 19.770424-0.000000i 4.1167e-11 19.770424-0.000000i 7.67294e-11 19.770424-0.000000i 5.92692e-11 19.770424+0.000000i 7.66231e-11 19.770424+0.000000i 9.85743e-11 20.199838+0.000000i 8.37107e-11 20.199838+0.000000i 3.56701e-11 20.199838-0.000000i 8.82029e-11 20.199838-0.000000i 8.52395e-11 20.199838-0.000000i 8.82165e-11 20.199838-0.000000i 4.32813e-07 24.112262+0.000000i 2.25578e-10 24.112262-0.000000i 1.27094e-10 24.112262+0.000000i 1.14679e-10 24.227579-0.000000i 9.77701e-11 24.227579+0.000000i 1.41993e-10 24.227579-0.000000i 1.84562e-10 29.035473-0.000000i 2.69505e-10 29.035473-0.000000i 1.72167e-10 29.035473-0.000000i 1.87434e-10 29.649484-0.000000i 2.44039e-10 29.649484-0.000000i 3.84759e-10 32.112699+0.000000i 1.35182e-10 32.112699+0.000000i 1.91162e-10 32.112699-0.000000i 3.58118e-10 32.115257+0.000000i 3.58743e-10 32.115257-0.000000i 3.46267e-10 32.115257+0.000000i 4.55903e-10 50.121754+0.000000i 6.04554e-10 50.121754-0.000000i 6.26139e-10 ---------------------- -------------------- Number of iterations of the method: 3 Number of linear iterations of the method: 168 Solution method: krylovschur Number of requested eigenvalues: 50 Stopping condition: tol=1e-08, maxit=100 Linear eigensolve converged (73 eigenpairs) due to CONVERGED_TOL; iterations 3 ---------------------- -------------------- k ||Ax-kBx||/||kx|| ---------------------- -------------------- 0.000000+0.000000i 31.3252 0.000000-0.000000i 18.2211 0.000000-0.000000i 44.631 0.000000+0.000000i 13.836 0.000000-0.000000i 30.8722 0.000000+0.000000i 26.1132 0.000000+0.000000i 28.6077 0.000000-0.000000i 74.5526 0.000000+0.000000i 23.1291 0.000000-0.000000i 74.0475 0.000000-0.000000i 74.7671 0.000000-0.000000i 90.664 0.000000+0.000000i 45.6476 0.000000-0.000000i 84.4024 0.000000+0.000000i 55.9734 0.000000-0.000000i 42.2662 0.000000+0.000000i 155.367 0.000000-0.000000i 82.083 0.000000-0.000000i 14.409 0.000000-0.000000i 163.883 0.000000+0.000000i 70.2693 0.000000-0.000000i 38.2544 0.000000-0.000000i 35.2561 0.000000-0.000000i 87.8327 0.000000-0.000000i 73.0972 0.000000+0.000000i 102.899 0.000000-0.000000i 841.374 5.238680+0.000000i 7.02326e-12 5.238680+0.000000i 7.35325e-12 5.238680+0.000000i 6.45373e-12 12.047661+0.000000i 3.11174e-12 12.047661+0.000000i 2.79255e-12 12.047661+0.000000i 3.88688e-12 12.079279+0.000000i 3.54189e-12 12.079279+0.000000i 5.81284e-12 12.079279-0.000000i 9.48017e-12 12.954275+0.000000i 8.75372e-12 12.954275-0.000000i 7.94948e-12 12.955799-0.000000i 4.82741e-12 17.515033+0.000000i 7.16088e-12 17.515033-0.000000i 3.49241e-12 17.647490+0.000000i 1.45237e-11 19.770424-0.000000i 7.24883e-09 19.770424-0.000000i 2.08225e-12 19.770424-0.000000i 3.88102e-12 19.770424-0.000000i 2.99787e-12 19.770424+0.000000i 3.87565e-12 19.770424+0.000000i 4.98595e-12 20.199838+0.000000i 4.14413e-12 20.199838+0.000000i 1.76586e-12 20.199838-0.000000i 4.36652e-12 20.199838-0.000000i 4.21981e-12 20.199838-0.000000i 4.36719e-12 20.199838-0.000000i 2.14265e-08 24.112262+0.000000i 9.35532e-12 24.112262-0.000000i 5.27091e-12 24.112262+0.000000i 4.75603e-12 24.227579-0.000000i 4.03549e-12 24.227579+0.000000i 5.8608e-12 24.227579-0.000000i 7.61787e-12 29.035473-0.000000i 9.28191e-12 29.035473-0.000000i 5.92953e-12 29.035473-0.000000i 6.45536e-12 29.649484-0.000000i 8.23079e-12 29.649484-0.000000i 1.29769e-11 32.112699+0.000000i 4.20961e-12 32.112699+0.000000i 5.95285e-12 32.112699-0.000000i 1.11519e-11 32.115257+0.000000i 1.11705e-11 32.115257-0.000000i 1.0782e-11 32.115257+0.000000i 1.41958e-11 50.121754+0.000000i 1.20617e-11 50.121754-0.000000i 1.24924e-11 ---------------------- -------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Fri Sep 15 01:04:12 2017 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 15 Sep 2017 08:04:12 +0200 Subject: [petsc-users] SLEPC - Hermitian matrix gives complex eigenvalues In-Reply-To: References: <14C3EF90-C404-45AD-9516-63D3F9CA4D10@wpi.edu> <071CC27A-4A0D-4529-8B62-67CA1A44042F@dsic.upv.es> Message-ID: <915D9580-3454-4C9B-81D5-0F8B6CB32DC8@dsic.upv.es> I computed it with Matlab. You could also count the zero eigenvalues computed by SLEPc. Jose > El 15 sept 2017, a las 0:09, Pandey, Siddhant escribi?: > > How do I calculate the rank of my A matrix? I couldn't find anything in PETSC documentation. It seems like a mumps functionality, but I cannot get any output regarding null pivots/rank deficiency. > > Sidd > CCNS, WPI > From: Jose E. Roman > Sent: Thursday, September 14, 2017 4:33:01 AM > To: Pandey, Siddhant > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] SLEPC - Hermitian matrix gives complex eigenvalues > > > > El 14 sept 2017, a las 0:27, Pandey, Siddhant escribi?: > > > > Hello all, > > > > I am diagonalizing a hermitian matrix (MatIsHermitian = 1 up to a tolerance of 1e-15) using krylovschur, with EPS_TARGET_MAGNITUDE = 0, but getting complex eigenvalues. Also, I am getting seemingly spurious eigenvalues of magnitude very close to 0, whose relative errors are much larger than my set tolerance. Can anyone indicate what might be the cause and possible solutions. > > > > I have attached the files eps_view, A_mat_binary, and B_mat_binary files, which show the settings I have used, and contain the A and B matrices respectively (in binary). > > > > The eigenvalues I get are: > > > > # > > > > Eigenvalue > > ? ---------------------------------------------------------------- > > 0 0.0000000000000077 + i -0.0000000000000000 > > 1 0.0000000000000265 + i -0.0000000000000008 > > 2 0.0000000000000340 + i 0.0000000000000002 > > 3 0.0000000000000373 + i 0.0000000000000001 > > 4 0.0000000000000444 + i -0.0000000000000007 > > 5 0.0000000000000448 + i -0.0000000000000017 > > 6 0.0000000000000470 + i 0.0000000000000002 > > 7 0.0000000000000489 + i 0.0000000000000030 > > 8 0.0000000000000548 + i 0.0000000000000006 > > 9 0.0000000000000585 + i 0.0000000000000001 > > 10 0.0000000000000643 + i 0.0000000000000005 > > 11 0.0000000000000714 + i -0.0000000000000008 > > 12 0.0000000000000750 + i -0.0000000000000019 > > 13 0.0000000000000752 + i 0.0000000000000031 > > 14 0.0000000000000769 + i -0.0000000000000002 > > 15 0.0000000000000784 + i -0.0000000000000037 > > 16 0.0000000000000860 + i 0.0000000000000002 > > 17 0.0000000000000880 + i -0.0000000000000004 > > 18 0.0000000000000968 + i -0.0000000000000001 > > 19 0.0000000000000979 + i 0.0000000000000013 > > 20 0.0000000000001045 + i -0.0000000000000001 > > 21 0.0000000000001150 + i -0.0000000000000011 > > 22 0.0000000000001348 + i -0.0000000000000012 > > 23 0.0000000000001446 + i -0.0000000000000019 > > 24 0.0000000000001454 + i 0.0000000000000011 > > 25 0.0000000000001555 + i 0.0000000000000003 > > 26 0.0000000000002513 + i -0.0000000000000009 > > 27 5.0908854514230413 + i -0.0022004178122762 > > 28 5.2768106039842175 + i 0.1043464906789375 > > 29 5.3003883062604187 + i -0.0757735907905433 > > 30 11.5143655883932929 + i 0.0049838692042474 > > 31 11.8821523259838653 + i -0.1475608751440501 > > 32 11.9515216995487101 + i 0.1857395729336506 > > 33 12.0909362158339384 + i 0.0049114285397287 > > 34 12.0905704159492675 + i -0.1522880348537981 > > 35 12.2205398661469111 + i -0.0781999802933937 > > 36 12.4807156964720480 + i -0.2850122604907908 > > 37 12.6082849940660289 + i 0.0560456679728079 > > 38 12.9384278480576125 + i 0.0238826907631012 > > 39 17.5230234731441357 + i -0.1824807274488794 > > 40 17.5503678395543901 + i 0.1785356473404145 > > 41 17.6933953160112409 + i -0.0626631055149425 > > 42 19.1692824404480930 + i -0.6351729266691462 > > 43 19.4158452684509797 + i 0.3151965488807310 > > 44 19.6020750507704591 + i -0.2559887580276014 > > 45 19.6443102906562181 + i 1.1005601705485646 > > 46 19.7948713379697452 + i 0.1230015140422697 > > 47 19.8791098474284347 + i -0.1943322911563744 > > 48 20.1268732265661860 + i -0.0340890856219265 > > 49 20.1368588182453898 + i 0.2673370459460956 > > > > > > Thanks in advance, > > Sidd > > Worcester Polytechnic, U.S.A. > > > > Your B matrix has full rank, but rank(A)=1002. This means your eigenproblem has 27 zero eigenvalues. So no wonder that you get tiny eigenvalues (they are numerically zero in the working precision). > > Since you problem is Hermitian, you should solve it as such, adding -eps_gen_hermitian. Then the accuracy will be a bit better. Still, the relative error for the tiny eigenvalues will be large (division by zero), but you can see that the *absolute* error shows that the eigenvector is computed correctly. See output below. > > Another thing is that since your A matrix is singular, you should not use target=0, because then the solver will invert matrix A. MUMPS is good at working with singular matrices, but still the error will propagate to the computed eigensolutions. You have to move away from zero, e.g. target=0.1 > > Finally, do not set ncv=1029. By doing this, the built subspace is equal to the whole space of the original problem, so the cost will be even higher than calling LAPACK directly on your matrices. Let SLEPc choose the ncv value. > > Jose > > > $ ./ex7 -f1 ~/tmp/pandey/A_mat_binary -f2 ~/tmp/pandey/B_mat_binary -st_pc_factor_mat_solver_package mumps -st_type sinvert -eps_target 0.1 -eps_nev 50 -eps_error_absolute ::ascii_info_detail > > Generalized eigenproblem stored in file. > > Reading COMPLEX matrices from binary files... > ---------------------- -------------------- > k ||Ax-kBx|| > ---------------------- -------------------- > 0.000000+0.000000i 7.65812e-12 > 0.000000-0.000000i 2.78585e-12 > 0.000000-0.000000i 6.30965e-12 > 0.000000+0.000000i 1.90477e-12 > 0.000000-0.000000i 4.18541e-12 > 0.000000+0.000000i 2.90717e-12 > 0.000000+0.000000i 2.90732e-12 > 0.000000-0.000000i 6.69097e-12 > 0.000000+0.000000i 2.01707e-12 > 0.000000-0.000000i 6.2849e-12 > 0.000000-0.000000i 6.30369e-12 > 0.000000-0.000000i 6.79209e-12 > 0.000000+0.000000i 3.3761e-12 > 0.000000-0.000000i 6.19772e-12 > 0.000000+0.000000i 3.87231e-12 > 0.000000-0.000000i 2.75338e-12 > 0.000000+0.000000i 9.50872e-12 > 0.000000-0.000000i 4.59981e-12 > 0.000000-0.000000i 7.62071e-13 > 0.000000-0.000000i 7.24146e-12 > 0.000000+0.000000i 2.98146e-12 > 0.000000-0.000000i 1.49165e-12 > 0.000000-0.000000i 1.33641e-12 > 0.000000-0.000000i 3.02551e-12 > 0.000000-0.000000i 1.76448e-12 > 0.000000+0.000000i 2.03659e-12 > 0.000000-0.000000i 5.71796e-12 > 5.238680+0.000000i 3.67926e-11 > 5.238680+0.000000i 3.85213e-11 > 5.238680+0.000000i 3.3809e-11 > 12.047661+0.000000i 3.74891e-11 > 12.047661+0.000000i 3.36437e-11 > 12.047661+0.000000i 4.68278e-11 > 12.079279+0.000000i 4.27835e-11 > 12.079279+0.000000i 7.02149e-11 > 12.079279-0.000000i 1.14514e-10 > 12.954275+0.000000i 1.13398e-10 > 12.954275-0.000000i 1.0298e-10 > 12.955799-0.000000i 6.2543e-11 > 17.515033+0.000000i 1.25423e-10 > 17.515033-0.000000i 6.11696e-11 > 17.647490+0.000000i 2.56307e-10 > 19.770424-0.000000i 1.43313e-07 > 19.770424-0.000000i 4.1167e-11 > 19.770424-0.000000i 7.67294e-11 > 19.770424-0.000000i 5.92692e-11 > 19.770424+0.000000i 7.66231e-11 > 19.770424+0.000000i 9.85743e-11 > 20.199838+0.000000i 8.37107e-11 > 20.199838+0.000000i 3.56701e-11 > 20.199838-0.000000i 8.82029e-11 > 20.199838-0.000000i 8.52395e-11 > 20.199838-0.000000i 8.82165e-11 > 20.199838-0.000000i 4.32813e-07 > 24.112262+0.000000i 2.25578e-10 > 24.112262-0.000000i 1.27094e-10 > 24.112262+0.000000i 1.14679e-10 > 24.227579-0.000000i 9.77701e-11 > 24.227579+0.000000i 1.41993e-10 > 24.227579-0.000000i 1.84562e-10 > 29.035473-0.000000i 2.69505e-10 > 29.035473-0.000000i 1.72167e-10 > 29.035473-0.000000i 1.87434e-10 > 29.649484-0.000000i 2.44039e-10 > 29.649484-0.000000i 3.84759e-10 > 32.112699+0.000000i 1.35182e-10 > 32.112699+0.000000i 1.91162e-10 > 32.112699-0.000000i 3.58118e-10 > 32.115257+0.000000i 3.58743e-10 > 32.115257-0.000000i 3.46267e-10 > 32.115257+0.000000i 4.55903e-10 > 50.121754+0.000000i 6.04554e-10 > 50.121754-0.000000i 6.26139e-10 > ---------------------- -------------------- > Number of iterations of the method: 3 > Number of linear iterations of the method: 168 > Solution method: krylovschur > > Number of requested eigenvalues: 50 > Stopping condition: tol=1e-08, maxit=100 > Linear eigensolve converged (73 eigenpairs) due to CONVERGED_TOL; iterations 3 > ---------------------- -------------------- > k ||Ax-kBx||/||kx|| > ---------------------- -------------------- > 0.000000+0.000000i 31.3252 > 0.000000-0.000000i 18.2211 > 0.000000-0.000000i 44.631 > 0.000000+0.000000i 13.836 > 0.000000-0.000000i 30.8722 > 0.000000+0.000000i 26.1132 > 0.000000+0.000000i 28.6077 > 0.000000-0.000000i 74.5526 > 0.000000+0.000000i 23.1291 > 0.000000-0.000000i 74.0475 > 0.000000-0.000000i 74.7671 > 0.000000-0.000000i 90.664 > 0.000000+0.000000i 45.6476 > 0.000000-0.000000i 84.4024 > 0.000000+0.000000i 55.9734 > 0.000000-0.000000i 42.2662 > 0.000000+0.000000i 155.367 > 0.000000-0.000000i 82.083 > 0.000000-0.000000i 14.409 > 0.000000-0.000000i 163.883 > 0.000000+0.000000i 70.2693 > 0.000000-0.000000i 38.2544 > 0.000000-0.000000i 35.2561 > 0.000000-0.000000i 87.8327 > 0.000000-0.000000i 73.0972 > 0.000000+0.000000i 102.899 > 0.000000-0.000000i 841.374 > 5.238680+0.000000i 7.02326e-12 > 5.238680+0.000000i 7.35325e-12 > 5.238680+0.000000i 6.45373e-12 > 12.047661+0.000000i 3.11174e-12 > 12.047661+0.000000i 2.79255e-12 > 12.047661+0.000000i 3.88688e-12 > 12.079279+0.000000i 3.54189e-12 > 12.079279+0.000000i 5.81284e-12 > 12.079279-0.000000i 9.48017e-12 > 12.954275+0.000000i 8.75372e-12 > 12.954275-0.000000i 7.94948e-12 > 12.955799-0.000000i 4.82741e-12 > 17.515033+0.000000i 7.16088e-12 > 17.515033-0.000000i 3.49241e-12 > 17.647490+0.000000i 1.45237e-11 > 19.770424-0.000000i 7.24883e-09 > 19.770424-0.000000i 2.08225e-12 > 19.770424-0.000000i 3.88102e-12 > 19.770424-0.000000i 2.99787e-12 > 19.770424+0.000000i 3.87565e-12 > 19.770424+0.000000i 4.98595e-12 > 20.199838+0.000000i 4.14413e-12 > 20.199838+0.000000i 1.76586e-12 > 20.199838-0.000000i 4.36652e-12 > 20.199838-0.000000i 4.21981e-12 > 20.199838-0.000000i 4.36719e-12 > 20.199838-0.000000i 2.14265e-08 > 24.112262+0.000000i 9.35532e-12 > 24.112262-0.000000i 5.27091e-12 > 24.112262+0.000000i 4.75603e-12 > 24.227579-0.000000i 4.03549e-12 > 24.227579+0.000000i 5.8608e-12 > 24.227579-0.000000i 7.61787e-12 > 29.035473-0.000000i 9.28191e-12 > 29.035473-0.000000i 5.92953e-12 > 29.035473-0.000000i 6.45536e-12 > 29.649484-0.000000i 8.23079e-12 > 29.649484-0.000000i 1.29769e-11 > 32.112699+0.000000i 4.20961e-12 > 32.112699+0.000000i 5.95285e-12 > 32.112699-0.000000i 1.11519e-11 > 32.115257+0.000000i 1.11705e-11 > 32.115257-0.000000i 1.0782e-11 > 32.115257+0.000000i 1.41958e-11 > 50.121754+0.000000i 1.20617e-11 > 50.121754-0.000000i 1.24924e-11 > ---------------------- -------------------- From afrah.nacib at gmail.com Fri Sep 15 06:00:36 2017 From: afrah.nacib at gmail.com (Afrah Najib) Date: Fri, 15 Sep 2017 14:00:36 +0300 Subject: [petsc-users] PDSLin 2.0.0 example memory corruption error Message-ID: Hi, I am trying to run the examples of PDSLin 2.0.0[ http://portal.nersc.gov/project/sparse/pdslin/] but I got the following error: :~/pdslin_2.0.0/examples$ mpiexec -n 8 ./dtest input matrices/U.dat input->num_doms = 4 Preconditioner parameters: input->nproc_dcomp = 4 input->nproc_schur = 4 input->free_local = PDSLin_NO input->dcomp_type = SCOTCH input->diag_tau = 0.000000e+00 input->tau_sub = 0.000000e+00 input->drop_tau0 = 1.000000e-02 input->drop_tau2 = 0.000000e+00 input->drop_tau1 = 1.000000e-03 input->drop_tau3 = 0.000000e+00 input->ilu_lvl = -1 input->equil_dom = 0 (no equil or row perm). input->perm_dom = 0 input->mat_type = UNSYMMETRIC input->mat_pattern = UNSYMMETRIC input->blk_size = 1 input->equil_schur = 5 input->relax_factor = 3.000000e-01 input->pperm_schur = PDSLin_YES input->psymb_schur = PDSLin_YES input->dom_solver = SLU_DIST Schur solution parameters: input->inner_outer = PDSLin_NO input->outer_tol = 1.000000e-12 input->outer_max = 50 input->schur_itrs = PDSLin_NO input->exact_schur = PDSLin_NO input->direct_schur = PDSLin_NO input->explicit_restart = PDSLin_NO input->inner_max = 500 input->inner_restart = 60 input->inner_tol = 1.000000e-12 input->inner_solver = PDSLin_FGMRES ==== dtest(): reading in matrix from matrices/U.dat took 1.785858e-01 seconds to read matrix. ********************************************************************** ==== dtest: factorizing the matrix. PARMETIS ERROR: Poor initial vertex distribution. Processor 2 has no vertices assigned to it! PARMETIS ERROR: Poor initial vertex distribution. Processor 3 has no vertices assigned to it! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 [0]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah Fri Sep 15 13:53:42 2017 [0]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs --download-scalapack=1 --download-mumps=1 --with-valgrind=1 --with-valgrind-dir="[/usr/local/bin]" [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Signal received [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 [1]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah Fri Sep 15 13:53:42 2017 [1]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs --download-scalapack=1 --download-mumps=1 --with-valgrind=1 --with-valgrind-dir="[/usr/local/bin]" [1]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [2]PETSC ERROR: likely location of problem given in stack below [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [2]PETSC ERROR: INSTEAD the line number of the start of the function [2]PETSC ERROR: is given. [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Signal received [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 [2]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah Fri Sep 15 13:53:42 2017 [2]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs --download-scalapack=1 --download-mumps=1 --with-valgrind=1 --with-valgrind-dir="[/usr/local/bin]" [2]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [3]PETSC ERROR: likely location of problem given in stack below [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [3]PETSC ERROR: INSTEAD the line number of the start of the function [3]PETSC ERROR: is given. [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Signal received [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 [3]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah Fri Sep 15 13:53:42 2017 [3]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs --download-scalapack=1 --download-mumps=1 --with-valgrind=1 --with-valgrind-dir="[/usr/local/bin]" [3]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [4]PETSC ERROR: likely location of problem given in stack below [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [4]PETSC ERROR: INSTEAD the line number of the start of the function [4]PETSC ERROR: is given. [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [4]PETSC ERROR: Signal received [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [4]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 [4]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah Fri Sep 15 13:53:42 2017 [4]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs --download-scalapack=1 --download-mumps=1 --with-valgrind=1 --with-valgrind-dir="[/usr/local/bin]" [4]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4 [5]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [5]PETSC ERROR: likely location of problem given in stack below [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [5]PETSC ERROR: INSTEAD the line number of the start of the function [5]PETSC ERROR: is given. [5]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [5]PETSC ERROR: Signal received [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [5]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 [5]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah Fri Sep 15 13:53:42 2017 [5]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs --download-scalapack=1 --download-mumps=1 --with-valgrind=1 --with-valgrind-dir="[/usr/local/bin]" [5]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5 [6]PETSC ERROR: [7]PETSC ERROR: ------------------------------------------------------------------------ [7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [7]PETSC ERROR: likely location of problem given in stack below [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [7]PETSC ERROR: INSTEAD the line number of the start of the function [7]PETSC ERROR: is given. [7]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [7]PETSC ERROR: Signal received [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [7]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 [7]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah Fri Sep 15 13:53:42 2017 [7]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs --download-scalapack=1 --download-mumps=1 --with-valgrind=1 --with-valgrind-dir="[/usr/local/bin]" [7]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 7 ------------------------------------------------------------------------ [6]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [6]PETSC ERROR: likely location of problem given in stack below [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [6]PETSC ERROR: INSTEAD the line number of the start of the function [6]PETSC ERROR: is given. [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [6]PETSC ERROR: Signal received [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [6]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 [6]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah Fri Sep 15 13:53:42 2017 [6]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs --download-scalapack=1 --download-mumps=1 --with-valgrind=1 --with-valgrind-dir="[/usr/local/bin]" [6]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6 any idea -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Sep 15 07:01:06 2017 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 15 Sep 2017 08:01:06 -0400 Subject: [petsc-users] PDSLin 2.0.0 example memory corruption error In-Reply-To: References: Message-ID: On Fri, Sep 15, 2017 at 7:00 AM, Afrah Najib wrote: > Hi, > I am trying to run the examples of PDSLin 2.0.0[http://portal.nersc.gov/ > project/sparse/pdslin/] > > but I got the following error: > This is not a PETSc error. PETSc is just catching the signal. I think you have to contact the PDSLin developers. Thanks, Matt > :~/pdslin_2.0.0/examples$ mpiexec -n 8 ./dtest input matrices/U.dat > input->num_doms = 4 > Preconditioner parameters: > input->nproc_dcomp = 4 > input->nproc_schur = 4 > input->free_local = PDSLin_NO > input->dcomp_type = SCOTCH > input->diag_tau = 0.000000e+00 > input->tau_sub = 0.000000e+00 > input->drop_tau0 = 1.000000e-02 > input->drop_tau2 = 0.000000e+00 > input->drop_tau1 = 1.000000e-03 > input->drop_tau3 = 0.000000e+00 > input->ilu_lvl = -1 > input->equil_dom = 0 (no equil or row perm). > input->perm_dom = 0 > input->mat_type = UNSYMMETRIC > input->mat_pattern = UNSYMMETRIC > input->blk_size = 1 > input->equil_schur = 5 > input->relax_factor = 3.000000e-01 > input->pperm_schur = PDSLin_YES > input->psymb_schur = PDSLin_YES > input->dom_solver = SLU_DIST > Schur solution parameters: > input->inner_outer = PDSLin_NO > input->outer_tol = 1.000000e-12 > input->outer_max = 50 > input->schur_itrs = PDSLin_NO > input->exact_schur = PDSLin_NO > input->direct_schur = PDSLin_NO > input->explicit_restart = PDSLin_NO > input->inner_max = 500 > input->inner_restart = 60 > input->inner_tol = 1.000000e-12 > input->inner_solver = PDSLin_FGMRES > ==== dtest(): reading in matrix from matrices/U.dat > took 1.785858e-01 seconds to read matrix. > > ********************************************************************** > > > ==== dtest: factorizing the matrix. > PARMETIS ERROR: Poor initial vertex distribution. Processor 2 has no > vertices assigned to it! > PARMETIS ERROR: Poor initial vertex distribution. Processor 3 has no > vertices assigned to it! > [0]PETSC ERROR: ------------------------------ > ------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 > [0]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah > Fri Sep 15 13:53:42 2017 > [0]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 > PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs > --download-scalapack=1 --download-mumps=1 --with-valgrind=1 > --with-valgrind-dir="[/usr/local/bin]" > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [1]PETSC ERROR: ------------------------------ > ------------------------------------------ > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [1]PETSC ERROR: likely location of problem given in stack below > [1]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [1]PETSC ERROR: INSTEAD the line number of the start of the function > [1]PETSC ERROR: is given. > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Signal received > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 > [1]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah > Fri Sep 15 13:53:42 2017 > [1]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 > PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs > --download-scalapack=1 --download-mumps=1 --with-valgrind=1 > --with-valgrind-dir="[/usr/local/bin]" > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 > [2]PETSC ERROR: ------------------------------ > ------------------------------------------ > [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [2]PETSC ERROR: likely location of problem given in stack below > [2]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [2]PETSC ERROR: INSTEAD the line number of the start of the function > [2]PETSC ERROR: is given. > [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: Signal received > [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [2]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 > [2]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah > Fri Sep 15 13:53:42 2017 > [2]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 > PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs > --download-scalapack=1 --download-mumps=1 --with-valgrind=1 > --with-valgrind-dir="[/usr/local/bin]" > [2]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 > [3]PETSC ERROR: ------------------------------ > ------------------------------------------ > [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [3]PETSC ERROR: likely location of problem given in stack below > [3]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [3]PETSC ERROR: INSTEAD the line number of the start of the function > [3]PETSC ERROR: is given. > [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [3]PETSC ERROR: Signal received > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 > [3]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah > Fri Sep 15 13:53:42 2017 > [3]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 > PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs > --download-scalapack=1 --download-mumps=1 --with-valgrind=1 > --with-valgrind-dir="[/usr/local/bin]" > [3]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 > [4]PETSC ERROR: ------------------------------ > ------------------------------------------ > [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [4]PETSC ERROR: likely location of problem given in stack below > [4]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [4]PETSC ERROR: INSTEAD the line number of the start of the function > [4]PETSC ERROR: is given. > [4]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [4]PETSC ERROR: Signal received > [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [4]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 > [4]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah > Fri Sep 15 13:53:42 2017 > [4]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 > PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs > --download-scalapack=1 --download-mumps=1 --with-valgrind=1 > --with-valgrind-dir="[/usr/local/bin]" > [4]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4 > [5]PETSC ERROR: ------------------------------ > ------------------------------------------ > [5]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [5]PETSC ERROR: likely location of problem given in stack below > [5]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [5]PETSC ERROR: INSTEAD the line number of the start of the function > [5]PETSC ERROR: is given. > [5]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [5]PETSC ERROR: Signal received > [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [5]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 > [5]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah > Fri Sep 15 13:53:42 2017 > [5]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 > PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs > --download-scalapack=1 --download-mumps=1 --with-valgrind=1 > --with-valgrind-dir="[/usr/local/bin]" > [5]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5 > [6]PETSC ERROR: [7]PETSC ERROR: ------------------------------ > ------------------------------------------ > [7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [7]PETSC ERROR: likely location of problem given in stack below > [7]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [7]PETSC ERROR: INSTEAD the line number of the start of the function > [7]PETSC ERROR: is given. > [7]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [7]PETSC ERROR: Signal received > [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [7]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 > [7]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah > Fri Sep 15 13:53:42 2017 > [7]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 > PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs > --download-scalapack=1 --download-mumps=1 --with-valgrind=1 > --with-valgrind-dir="[/usr/local/bin]" > [7]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 7 > ------------------------------------------------------------------------ > [6]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [6]PETSC ERROR: likely location of problem given in stack below > [6]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [6]PETSC ERROR: INSTEAD the line number of the start of the function > [6]PETSC ERROR: is given. > [6]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [6]PETSC ERROR: Signal received > [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [6]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 > [6]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah > Fri Sep 15 13:53:42 2017 > [6]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 > PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs > --download-scalapack=1 --download-mumps=1 --with-valgrind=1 > --with-valgrind-dir="[/usr/local/bin]" > [6]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6 > > any idea > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Sep 15 10:45:05 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 15 Sep 2017 10:45:05 -0500 Subject: [petsc-users] mg pre-conditioner default setup from PETSc-3.4 to PETSc-3.7 In-Reply-To: References: <3E0D16E2-BA8F-4792-BBC3-8C55611292CC@mcs.anl.gov> <6BDE8DAF-F70B-42D7-A92B-8CD682BAC928@mcs.anl.gov> Message-ID: <1A3CDFAF-5A93-4AA0-9222-DF43359C547B@mcs.anl.gov> $ python Python 2.7.13 (default, Dec 18 2016, 07:03:39) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. MatMult >>> 3.6272e+03 - 2.0894e+03 1537.7999999999997 KSP Solve >>> 3.6329e+03 - 2.0949e+03 1538.0 >>> You are right, all the extra time is within the MatMult() so for some reason your shell mat mult is much slower. I cannot guess why unless I can see inside your shell matmult at what it is doing. Make sure your configure options are identical and using same compiler. Barry > On Sep 15, 2017, at 5:08 AM, Federico Golfr? Andreasi wrote: > > Hi Barry, > > I have attached an extract of the our program output for both the versions: PETSc-3.4.4 and PETSc-3.7.3. > > In this program the KSP has a shell matrix as operator and a MPIAIJ matrix as pre-conditioner. > I was wondering if the slowing is related more on the operations done in the MatMult of the shell matrix; > because on a test program where I solve a similar system without shell matrix I do not see the performance degradation. > > Perhaps you could give me some hints, > Thank you and best regards, > Federico > > > > > On 13 September 2017 at 17:58, Barry Smith wrote: > > > On Sep 13, 2017, at 10:56 AM, Federico Golfr? Andreasi wrote: > > > > Hi Barry, > > > > I understand and perfectly agree with you that the behavior increase after the release due to better tuning. > > > > In my case, the difference in the solution is negligible, but the runtime increases up to +70% (with the same number of ksp_iterations). > > Ok this is an important (and bad) difference. > > > So I was wondering if maybe there were just some flags related to memory preallocation or re-usage of intermediate solution that before was defaulted. > > Note likely it is this. > > Are both compiled with the same level of compiler optimization? > > Please run both with -log_summary and send the output, this will tell us WHAT parts are now slower. > > Barry > > > > > Thank you, > > Federico > > > > > > > > On 13 September 2017 at 17:29, Barry Smith wrote: > > > > There will likely always be slight differences in convergence over that many releases. Lots of little defaults etc get changed over time as we learn from users and increase the robustness of the defaults. > > > > So in your case do the differences matter? > > > > 1) What is the time to solution in both cases, is it a few percent different or now much slower? > > > > 2) What about number of iterations? Almost identical (say 1 or 2 different) or does it now take 30 iterations when it use to take 5? > > > > Barry > > > > > On Sep 13, 2017, at 10:25 AM, Federico Golfr? Andreasi wrote: > > > > > > Dear PETSc users/developers, > > > > > > I recently switched from PETSc-3.4 to PETSc-3.7 and found that some default setup for the "mg" (mutigrid) preconditioner have changed. > > > > > > We were solving a linear system passing, throug command line, the following options: > > > -ksp_type fgmres > > > -ksp_max_it 100000 > > > -ksp_rtol 0.000001 > > > -pc_type mg > > > -ksp_view > > > > > > The output of the KSP view is as follow: > > > > > > KSP Object: 128 MPI processes > > > type: fgmres > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > > GMRES: happy breakdown tolerance 1e-30 > > > maximum iterations=100000, initial guess is zero > > > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > > > right preconditioning > > > using UNPRECONDITIONED norm type for convergence test > > > PC Object: 128 MPI processes > > > type: mg > > > MG: type is MULTIPLICATIVE, levels=1 cycles=v > > > Cycles per PCApply=1 > > > Not using Galerkin computed coarse grid matrices > > > Coarse grid solver -- level ------------------------------- > > > KSP Object: (mg_levels_0_) 128 MPI processes > > > type: chebyshev > > > Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903 > > > Chebyshev: estimated using: [0 0.1; 0 1.1] > > > KSP Object: (mg_levels_0_est_) 128 MPI processes > > > type: gmres > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > > GMRES: happy breakdown tolerance 1e-30 > > > maximum iterations=10, initial guess is zero > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > > left preconditioning > > > using NONE norm type for convergence test > > > PC Object: (mg_levels_0_) 128 MPI processes > > > type: sor > > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > > > linear system matrix followed by preconditioner matrix: > > > Matrix Object: 128 MPI processes > > > type: mpiaij > > > rows=279669, cols=279669 > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > total number of mallocs used during MatSetValues calls =0 > > > not using I-node (on process 0) routines > > > Matrix Object: 128 MPI processes > > > type: mpiaij > > > rows=279669, cols=279669 > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > total number of mallocs used during MatSetValues calls =0 > > > not using I-node (on process 0) routines > > > maximum iterations=1, initial guess is zero > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > > left preconditioning > > > using NONE norm type for convergence test > > > PC Object: (mg_levels_0_) 128 MPI processes > > > type: sor > > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > > > linear system matrix followed by preconditioner matrix: > > > Matrix Object: 128 MPI processes > > > type: mpiaij > > > rows=279669, cols=279669 > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > total number of mallocs used during MatSetValues calls =0 > > > not using I-node (on process 0) routines > > > Matrix Object: 128 MPI processes > > > type: mpiaij > > > rows=279669, cols=279669 > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > total number of mallocs used during MatSetValues calls =0 > > > not using I-node (on process 0) routines > > > linear system matrix followed by preconditioner matrix: > > > Matrix Object: 128 MPI processes > > > type: mpiaij > > > rows=279669, cols=279669 > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > total number of mallocs used during MatSetValues calls =0 > > > not using I-node (on process 0) routines > > > Matrix Object: 128 MPI processes > > > type: mpiaij > > > rows=279669, cols=279669 > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > total number of mallocs used during MatSetValues calls =0 > > > not using I-node (on process 0) routines > > > > > > When I build the same program using PETSc-3.7 and run it with the same options we observe that the runtime increases and the convergence is slightly different. The output of the KSP view is: > > > > > > KSP Object: 128 MPI processes > > > type: fgmres > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > > GMRES: happy breakdown tolerance 1e-30 > > > maximum iterations=100000, initial guess is zero > > > tolerances: relative=1e-06, absolute=1e-50, divergence=10000. > > > right preconditioning > > > using UNPRECONDITIONED norm type for convergence test > > > PC Object: 128 MPI processes > > > type: mg > > > MG: type is MULTIPLICATIVE, levels=1 cycles=v > > > Cycles per PCApply=1 > > > Not using Galerkin computed coarse grid matrices > > > Coarse grid solver -- level ------------------------------- > > > KSP Object: (mg_levels_0_) 128 MPI processes > > > type: chebyshev > > > Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903 > > > Chebyshev: eigenvalues estimated using gmres with translations [0. 0.1; 0. 1.1] > > > KSP Object: (mg_levels_0_esteig_) 128 MPI processes > > > type: gmres > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > > GMRES: happy breakdown tolerance 1e-30 > > > maximum iterations=10, initial guess is zero > > > tolerances: relative=1e-12, absolute=1e-50, divergence=10000. > > > left preconditioning > > > using PRECONDITIONED norm type for convergence test > > > maximum iterations=2, initial guess is zero > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > > left preconditioning > > > using NONE norm type for convergence test > > > PC Object: (mg_levels_0_) 128 MPI processes > > > type: sor > > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > > > linear system matrix followed by preconditioner matrix: > > > Mat Object: 128 MPI processes > > > type: mpiaij > > > rows=279669, cols=279669 > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > total number of mallocs used during MatSetValues calls =0 > > > not using I-node (on process 0) routines > > > Mat Object: 128 MPI processes > > > type: mpiaij > > > rows=279669, cols=279669 > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > total number of mallocs used during MatSetValues calls =0 > > > not using I-node (on process 0) routines > > > linear system matrix followed by preconditioner matrix: > > > Mat Object: 128 MPI processes > > > type: mpiaij > > > rows=279669, cols=279669 > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > total number of mallocs used during MatSetValues calls =0 > > > not using I-node (on process 0) routines > > > Mat Object: 128 MPI processes > > > type: mpiaij > > > rows=279669, cols=279669 > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > total number of mallocs used during MatSetValues calls =0 > > > not using I-node (on process 0) routines > > > > > > I was able to get a closer solution adding the following options: > > > -mg_levels_0_esteig_ksp_norm_type none > > > -mg_levels_0_esteig_ksp_rtol 1.0e-5 > > > -mg_levels_ksp_max_it 1 > > > > > > But I still can reach the same runtime we were observing with PETSc-3.4, could you please advice me if I should specify any other options? > > > > > > Thank you very much for your support, > > > Federico Golfre' Andreasi > > > > > > > > > > From hbcbh1999 at gmail.com Sat Sep 16 22:39:52 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Sat, 16 Sep 2017 23:39:52 -0400 Subject: [petsc-users] PETSc KSPBCGSL with HYPRE boomeramg preconditioner ksp_monitor in poisson function Message-ID: hi, I am using KSPBCGSL method and HYPRE boomeramg precondtioner passed into KSPsolve for poisson equation as part of incompressible NS. I have two questions Q1 is number of iterations for the same solver. max_iter = 20 for the same poisson function KSPGetIterationNumber() is used to set up max_iter = 20 0 KSP Residual norm 4.512245447770e-04 2 KSP Residual norm 6.396709069731e-07 4 KSP Residual norm 1.757784220489e-10 6 KSP Residual norm 1.501464709144e-14 8 KSP Residual norm 1.376381429122e-18 10 KSP Residual norm 1.072650548945e-19 In poisson_func(): num_iter = 10, rel_residual = 1.072651e-19 and the very same function at the next time step gives: remind you guys max_iter = 20. I assume the number of iterations should stop at num_iter = 2. but the solver seems to keep iterating util hitting the max_iter. 0 KSP Residual norm 5.107005838093e-16 2 KSP Residual norm 3.353634821198e-20 4 KSP Residual norm 1.835096039266e-23 6 KSP Residual norm 1.645496102409e-23 8 KSP Residual norm 1.645496099837e-23 10 KSP Residual norm 1.645496099836e-23 12 KSP Residual norm 1.645496099530e-23 14 KSP Residual norm 1.645496034473e-23 16 KSP Residual norm 1.645461130961e-23 18 KSP Residual norm 1.645460451075e-23 In poisson_func(): num_iter = 18, rel_residual = 1.645460e-23 Q2 is: it seems HYPRE with KSPsolve works with coarse mesh but when I double the mesh size. the solve halted indefinitely. ksp_monitor didn't produce any results. is there a way to understand what's happening? -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Sep 16 23:25:36 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 16 Sep 2017 23:25:36 -0500 Subject: [petsc-users] PETSc KSPBCGSL with HYPRE boomeramg preconditioner ksp_monitor in poisson function In-Reply-To: References: Message-ID: <392EB91C-0ED7-480C-8190-ADED7E7CDE24@mcs.anl.gov> > On Sep 16, 2017, at 10:39 PM, Hao Zhang wrote: > > hi, > > I am using KSPBCGSL method and HYPRE boomeramg precondtioner passed into KSPsolve for poisson equation as part of incompressible NS. > > I have two questions > Q1 is number of iterations for the same solver. max_iter = 20 for the same poisson function > > KSPGetIterationNumber() is used to set up max_iter = 20 What do you mean, KSPGetIterationNumber() tells you the current number of iterations. You cannot use it to set the maximum number of iterations. Use KSPSetTolerances() or -ksp_max_it to set the number > > > 0 KSP Residual norm 4.512245447770e-04 > 2 KSP Residual norm 6.396709069731e-07 > 4 KSP Residual norm 1.757784220489e-10 > 6 KSP Residual norm 1.501464709144e-14 > 8 KSP Residual norm 1.376381429122e-18 > 10 KSP Residual norm 1.072650548945e-19 > In poisson_func(): num_iter = 10, rel_residual = 1.072651e-19 > > and the very same function at the next time step gives: remind you guys max_iter = 20. I assume the number of iterations should stop at num_iter = 2. Why would it stop at num_iter 2. The number of iterations allowed is the same for each solve, it does not accumulate over all solves. > but the solver seems to keep iterating util hitting the max_iter. > 0 KSP Residual norm 5.107005838093e-16 > 2 KSP Residual norm 3.353634821198e-20 > 4 KSP Residual norm 1.835096039266e-23 > 6 KSP Residual norm 1.645496102409e-23 > 8 KSP Residual norm 1.645496099837e-23 > 10 KSP Residual norm 1.645496099836e-23 > 12 KSP Residual norm 1.645496099530e-23 > 14 KSP Residual norm 1.645496034473e-23 > 16 KSP Residual norm 1.645461130961e-23 > 18 KSP Residual norm 1.645460451075e-23 > In poisson_func(): num_iter = 18, rel_residual = 1.645460e-23 But the way it is almost never realistic to try to get residual norms to be this small. You might consider using something like -ksp_atol 1.e-16 or something > > Q2 is: it seems HYPRE with KSPsolve works with coarse mesh but when I double the mesh size. the solve halted indefinitely. ksp_monitor didn't produce any results. > is there a way to understand what's happening? You could run in the debugger and use control c to interrupt the code when it is "hanging" and type bt to see the stack trace for the hanging. > > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 From hbcbh1999 at gmail.com Sat Sep 16 23:33:09 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Sun, 17 Sep 2017 00:33:09 -0400 Subject: [petsc-users] PETSc KSPBCGSL with HYPRE boomeramg preconditioner ksp_monitor in poisson function In-Reply-To: <392EB91C-0ED7-480C-8190-ADED7E7CDE24@mcs.anl.gov> References: <392EB91C-0ED7-480C-8190-ADED7E7CDE24@mcs.anl.gov> Message-ID: thanks. Yes. KSPSetTolerances is used. tolerance is set to be 1.e-14 I was thinking if there is a debug option I could use from PETSc to get some insight? GDB was used. On Sun, Sep 17, 2017 at 12:25 AM, Barry Smith wrote: > > > On Sep 16, 2017, at 10:39 PM, Hao Zhang wrote: > > > > hi, > > > > I am using KSPBCGSL method and HYPRE boomeramg precondtioner passed into > KSPsolve for poisson equation as part of incompressible NS. > > > > I have two questions > > Q1 is number of iterations for the same solver. max_iter = 20 for the > same poisson function > > > > KSPGetIterationNumber() is used to set up max_iter = 20 > > What do you mean, KSPGetIterationNumber() tells you the current number > of iterations. You cannot use it to set the maximum number of iterations. > Use KSPSetTolerances() or -ksp_max_it to set the number > > > > > > 0 KSP Residual norm 4.512245447770e-04 > > 2 KSP Residual norm 6.396709069731e-07 > > 4 KSP Residual norm 1.757784220489e-10 > > 6 KSP Residual norm 1.501464709144e-14 > > 8 KSP Residual norm 1.376381429122e-18 > > 10 KSP Residual norm 1.072650548945e-19 > > In poisson_func(): num_iter = 10, rel_residual = 1.072651e-19 > > > > and the very same function at the next time step gives: remind you guys > max_iter = 20. I assume the number of iterations should stop at num_iter = > 2. > > Why would it stop at num_iter 2. The number of iterations allowed is > the same for each solve, it does not accumulate over all solves. > > > but the solver seems to keep iterating util hitting the max_iter. > > 0 KSP Residual norm 5.107005838093e-16 > > 2 KSP Residual norm 3.353634821198e-20 > > 4 KSP Residual norm 1.835096039266e-23 > > 6 KSP Residual norm 1.645496102409e-23 > > 8 KSP Residual norm 1.645496099837e-23 > > 10 KSP Residual norm 1.645496099836e-23 > > 12 KSP Residual norm 1.645496099530e-23 > > 14 KSP Residual norm 1.645496034473e-23 > > 16 KSP Residual norm 1.645461130961e-23 > > 18 KSP Residual norm 1.645460451075e-23 > > In poisson_func(): num_iter = 18, rel_residual = 1.645460e-23 > > But the way it is almost never realistic to try to get residual norms > to be this small. You might consider using something like -ksp_atol 1.e-16 > or something > > > > > Q2 is: it seems HYPRE with KSPsolve works with coarse mesh but when I > double the mesh size. the solve halted indefinitely. ksp_monitor didn't > produce any results. > > is there a way to understand what's happening? > > You could run in the debugger and use control c to interrupt the code > when it is "hanging" and type bt to see the stack trace for the hanging. > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Sep 16 23:44:14 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 16 Sep 2017 23:44:14 -0500 Subject: [petsc-users] PETSc KSPBCGSL with HYPRE boomeramg preconditioner ksp_monitor in poisson function In-Reply-To: References: <392EB91C-0ED7-480C-8190-ADED7E7CDE24@mcs.anl.gov> Message-ID: > On Sep 16, 2017, at 11:33 PM, Hao Zhang wrote: > > thanks. > > Yes. KSPSetTolerances is used. > > tolerance is set to be 1.e-14 There is the rtol and atol are you sure you set the atol to 1e-14? > > I was thinking if there is a debug option I could use from PETSc to get some insight? GDB was used. You can use the command line option -start_in_debugger > > On Sun, Sep 17, 2017 at 12:25 AM, Barry Smith wrote: > > > On Sep 16, 2017, at 10:39 PM, Hao Zhang wrote: > > > > hi, > > > > I am using KSPBCGSL method and HYPRE boomeramg precondtioner passed into KSPsolve for poisson equation as part of incompressible NS. > > > > I have two questions > > Q1 is number of iterations for the same solver. max_iter = 20 for the same poisson function > > > > KSPGetIterationNumber() is used to set up max_iter = 20 > > What do you mean, KSPGetIterationNumber() tells you the current number of iterations. You cannot use it to set the maximum number of iterations. Use KSPSetTolerances() or -ksp_max_it to set the number > > > > > > 0 KSP Residual norm 4.512245447770e-04 > > 2 KSP Residual norm 6.396709069731e-07 > > 4 KSP Residual norm 1.757784220489e-10 > > 6 KSP Residual norm 1.501464709144e-14 > > 8 KSP Residual norm 1.376381429122e-18 > > 10 KSP Residual norm 1.072650548945e-19 > > In poisson_func(): num_iter = 10, rel_residual = 1.072651e-19 > > > > and the very same function at the next time step gives: remind you guys max_iter = 20. I assume the number of iterations should stop at num_iter = 2. > > Why would it stop at num_iter 2. The number of iterations allowed is the same for each solve, it does not accumulate over all solves. > > > but the solver seems to keep iterating util hitting the max_iter. > > 0 KSP Residual norm 5.107005838093e-16 > > 2 KSP Residual norm 3.353634821198e-20 > > 4 KSP Residual norm 1.835096039266e-23 > > 6 KSP Residual norm 1.645496102409e-23 > > 8 KSP Residual norm 1.645496099837e-23 > > 10 KSP Residual norm 1.645496099836e-23 > > 12 KSP Residual norm 1.645496099530e-23 > > 14 KSP Residual norm 1.645496034473e-23 > > 16 KSP Residual norm 1.645461130961e-23 > > 18 KSP Residual norm 1.645460451075e-23 > > In poisson_func(): num_iter = 18, rel_residual = 1.645460e-23 > > But the way it is almost never realistic to try to get residual norms to be this small. You might consider using something like -ksp_atol 1.e-16 or something > > > > > Q2 is: it seems HYPRE with KSPsolve works with coarse mesh but when I double the mesh size. the solve halted indefinitely. ksp_monitor didn't produce any results. > > is there a way to understand what's happening? > > You could run in the debugger and use control c to interrupt the code when it is "hanging" and type bt to see the stack trace for the hanging. > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > > > > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 From hbcbh1999 at gmail.com Sun Sep 17 00:11:25 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Sun, 17 Sep 2017 01:11:25 -0400 Subject: [petsc-users] PETSc KSPBCGSL with HYPRE boomeramg preconditioner ksp_monitor in poisson function In-Reply-To: References: <392EB91C-0ED7-480C-8190-ADED7E7CDE24@mcs.anl.gov> Message-ID: Yes. but only works via command line option.before was set up in the code. On Sun, Sep 17, 2017 at 12:44 AM, Barry Smith wrote: > > > On Sep 16, 2017, at 11:33 PM, Hao Zhang wrote: > > > > thanks. > > > > Yes. KSPSetTolerances is used. > > > > tolerance is set to be 1.e-14 > > There is the rtol and atol are you sure you set the atol to 1e-14? > > > > > I was thinking if there is a debug option I could use from PETSc to get > some insight? GDB was used. > > You can use the command line option -start_in_debugger > > > > > > On Sun, Sep 17, 2017 at 12:25 AM, Barry Smith > wrote: > > > > > On Sep 16, 2017, at 10:39 PM, Hao Zhang wrote: > > > > > > hi, > > > > > > I am using KSPBCGSL method and HYPRE boomeramg precondtioner passed > into KSPsolve for poisson equation as part of incompressible NS. > > > > > > I have two questions > > > Q1 is number of iterations for the same solver. max_iter = 20 for the > same poisson function > > > > > > KSPGetIterationNumber() is used to set up max_iter = 20 > > > > What do you mean, KSPGetIterationNumber() tells you the current number > of iterations. You cannot use it to set the maximum number of iterations. > Use KSPSetTolerances() or -ksp_max_it to set the number > > > > > > > > > 0 KSP Residual norm 4.512245447770e-04 > > > 2 KSP Residual norm 6.396709069731e-07 > > > 4 KSP Residual norm 1.757784220489e-10 > > > 6 KSP Residual norm 1.501464709144e-14 > > > 8 KSP Residual norm 1.376381429122e-18 > > > 10 KSP Residual norm 1.072650548945e-19 > > > In poisson_func(): num_iter = 10, rel_residual = 1.072651e-19 > > > > > > and the very same function at the next time step gives: remind you > guys max_iter = 20. I assume the number of iterations should stop at > num_iter = 2. > > > > Why would it stop at num_iter 2. The number of iterations allowed is > the same for each solve, it does not accumulate over all solves. > > > > > but the solver seems to keep iterating util hitting the max_iter. > > > 0 KSP Residual norm 5.107005838093e-16 > > > 2 KSP Residual norm 3.353634821198e-20 > > > 4 KSP Residual norm 1.835096039266e-23 > > > 6 KSP Residual norm 1.645496102409e-23 > > > 8 KSP Residual norm 1.645496099837e-23 > > > 10 KSP Residual norm 1.645496099836e-23 > > > 12 KSP Residual norm 1.645496099530e-23 > > > 14 KSP Residual norm 1.645496034473e-23 > > > 16 KSP Residual norm 1.645461130961e-23 > > > 18 KSP Residual norm 1.645460451075e-23 > > > In poisson_func(): num_iter = 18, rel_residual = 1.645460e-23 > > > > But the way it is almost never realistic to try to get residual norms > to be this small. You might consider using something like -ksp_atol 1.e-16 > or something > > > > > > > > Q2 is: it seems HYPRE with KSPsolve works with coarse mesh but when I > double the mesh size. the solve halted indefinitely. ksp_monitor didn't > produce any results. > > > is there a way to understand what's happening? > > > > You could run in the debugger and use control c to interrupt the code > when it is "hanging" and type bt to see the stack trace for the hanging. > > > > > > > > > > -- > > > Hao Zhang > > > Dept. of Applid Mathematics and Statistics, > > > Stony Brook University, > > > Stony Brook, New York, 11790 > > > > > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Sep 17 00:12:28 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 17 Sep 2017 00:12:28 -0500 Subject: [petsc-users] PETSc KSPBCGSL with HYPRE boomeramg preconditioner ksp_monitor in poisson function In-Reply-To: References: <392EB91C-0ED7-480C-8190-ADED7E7CDE24@mcs.anl.gov> Message-ID: Should work in the code also. Call KSPSetTolerances() and then call KSPView() to see what it is set to > On Sep 17, 2017, at 12:11 AM, Hao Zhang wrote: > > Yes. but only works via command line option.before was set up in the code. > > On Sun, Sep 17, 2017 at 12:44 AM, Barry Smith wrote: > > > On Sep 16, 2017, at 11:33 PM, Hao Zhang wrote: > > > > thanks. > > > > Yes. KSPSetTolerances is used. > > > > tolerance is set to be 1.e-14 > > There is the rtol and atol are you sure you set the atol to 1e-14? > > > > > I was thinking if there is a debug option I could use from PETSc to get some insight? GDB was used. > > You can use the command line option -start_in_debugger > > > > > > On Sun, Sep 17, 2017 at 12:25 AM, Barry Smith wrote: > > > > > On Sep 16, 2017, at 10:39 PM, Hao Zhang wrote: > > > > > > hi, > > > > > > I am using KSPBCGSL method and HYPRE boomeramg precondtioner passed into KSPsolve for poisson equation as part of incompressible NS. > > > > > > I have two questions > > > Q1 is number of iterations for the same solver. max_iter = 20 for the same poisson function > > > > > > KSPGetIterationNumber() is used to set up max_iter = 20 > > > > What do you mean, KSPGetIterationNumber() tells you the current number of iterations. You cannot use it to set the maximum number of iterations. Use KSPSetTolerances() or -ksp_max_it to set the number > > > > > > > > > 0 KSP Residual norm 4.512245447770e-04 > > > 2 KSP Residual norm 6.396709069731e-07 > > > 4 KSP Residual norm 1.757784220489e-10 > > > 6 KSP Residual norm 1.501464709144e-14 > > > 8 KSP Residual norm 1.376381429122e-18 > > > 10 KSP Residual norm 1.072650548945e-19 > > > In poisson_func(): num_iter = 10, rel_residual = 1.072651e-19 > > > > > > and the very same function at the next time step gives: remind you guys max_iter = 20. I assume the number of iterations should stop at num_iter = 2. > > > > Why would it stop at num_iter 2. The number of iterations allowed is the same for each solve, it does not accumulate over all solves. > > > > > but the solver seems to keep iterating util hitting the max_iter. > > > 0 KSP Residual norm 5.107005838093e-16 > > > 2 KSP Residual norm 3.353634821198e-20 > > > 4 KSP Residual norm 1.835096039266e-23 > > > 6 KSP Residual norm 1.645496102409e-23 > > > 8 KSP Residual norm 1.645496099837e-23 > > > 10 KSP Residual norm 1.645496099836e-23 > > > 12 KSP Residual norm 1.645496099530e-23 > > > 14 KSP Residual norm 1.645496034473e-23 > > > 16 KSP Residual norm 1.645461130961e-23 > > > 18 KSP Residual norm 1.645460451075e-23 > > > In poisson_func(): num_iter = 18, rel_residual = 1.645460e-23 > > > > But the way it is almost never realistic to try to get residual norms to be this small. You might consider using something like -ksp_atol 1.e-16 or something > > > > > > > > Q2 is: it seems HYPRE with KSPsolve works with coarse mesh but when I double the mesh size. the solve halted indefinitely. ksp_monitor didn't produce any results. > > > is there a way to understand what's happening? > > > > You could run in the debugger and use control c to interrupt the code when it is "hanging" and type bt to see the stack trace for the hanging. > > > > > > > > > > -- > > > Hao Zhang > > > Dept. of Applid Mathematics and Statistics, > > > Stony Brook University, > > > Stony Brook, New York, 11790 > > > > > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > > > > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 From hbcbh1999 at gmail.com Sun Sep 17 00:17:35 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Sun, 17 Sep 2017 01:17:35 -0400 Subject: [petsc-users] PETSc KSPBCGSL with HYPRE boomeramg preconditioner ksp_monitor in poisson function In-Reply-To: References: <392EB91C-0ED7-480C-8190-ADED7E7CDE24@mcs.anl.gov> Message-ID: OK. I will check the output Calling KSPView() I am experimenting with -start_in_debugger. it's for serial not for parallel, right? On Sun, Sep 17, 2017 at 1:12 AM, Barry Smith wrote: > > Should work in the code also. Call KSPSetTolerances() and then call > KSPView() to see what it is set to > > > On Sep 17, 2017, at 12:11 AM, Hao Zhang wrote: > > > > Yes. but only works via command line option.before was set up in the > code. > > > > On Sun, Sep 17, 2017 at 12:44 AM, Barry Smith > wrote: > > > > > On Sep 16, 2017, at 11:33 PM, Hao Zhang wrote: > > > > > > thanks. > > > > > > Yes. KSPSetTolerances is used. > > > > > > tolerance is set to be 1.e-14 > > > > There is the rtol and atol are you sure you set the atol to 1e-14? > > > > > > > > I was thinking if there is a debug option I could use from PETSc to > get some insight? GDB was used. > > > > You can use the command line option -start_in_debugger > > > > > > > > > > On Sun, Sep 17, 2017 at 12:25 AM, Barry Smith > wrote: > > > > > > > On Sep 16, 2017, at 10:39 PM, Hao Zhang wrote: > > > > > > > > hi, > > > > > > > > I am using KSPBCGSL method and HYPRE boomeramg precondtioner passed > into KSPsolve for poisson equation as part of incompressible NS. > > > > > > > > I have two questions > > > > Q1 is number of iterations for the same solver. max_iter = 20 for > the same poisson function > > > > > > > > KSPGetIterationNumber() is used to set up max_iter = 20 > > > > > > What do you mean, KSPGetIterationNumber() tells you the current > number of iterations. You cannot use it to set the maximum number of > iterations. Use KSPSetTolerances() or -ksp_max_it to set the number > > > > > > > > > > > > 0 KSP Residual norm 4.512245447770e-04 > > > > 2 KSP Residual norm 6.396709069731e-07 > > > > 4 KSP Residual norm 1.757784220489e-10 > > > > 6 KSP Residual norm 1.501464709144e-14 > > > > 8 KSP Residual norm 1.376381429122e-18 > > > > 10 KSP Residual norm 1.072650548945e-19 > > > > In poisson_func(): num_iter = 10, rel_residual = 1.072651e-19 > > > > > > > > and the very same function at the next time step gives: remind you > guys max_iter = 20. I assume the number of iterations should stop at > num_iter = 2. > > > > > > Why would it stop at num_iter 2. The number of iterations allowed > is the same for each solve, it does not accumulate over all solves. > > > > > > > but the solver seems to keep iterating util hitting the max_iter. > > > > 0 KSP Residual norm 5.107005838093e-16 > > > > 2 KSP Residual norm 3.353634821198e-20 > > > > 4 KSP Residual norm 1.835096039266e-23 > > > > 6 KSP Residual norm 1.645496102409e-23 > > > > 8 KSP Residual norm 1.645496099837e-23 > > > > 10 KSP Residual norm 1.645496099836e-23 > > > > 12 KSP Residual norm 1.645496099530e-23 > > > > 14 KSP Residual norm 1.645496034473e-23 > > > > 16 KSP Residual norm 1.645461130961e-23 > > > > 18 KSP Residual norm 1.645460451075e-23 > > > > In poisson_func(): num_iter = 18, rel_residual = 1.645460e-23 > > > > > > But the way it is almost never realistic to try to get residual > norms to be this small. You might consider using something like -ksp_atol > 1.e-16 or something > > > > > > > > > > > Q2 is: it seems HYPRE with KSPsolve works with coarse mesh but when > I double the mesh size. the solve halted indefinitely. ksp_monitor didn't > produce any results. > > > > is there a way to understand what's happening? > > > > > > You could run in the debugger and use control c to interrupt the > code when it is "hanging" and type bt to see the stack trace for the > hanging. > > > > > > > > > > > > > > -- > > > > Hao Zhang > > > > Dept. of Applid Mathematics and Statistics, > > > > Stony Brook University, > > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > > > -- > > > Hao Zhang > > > Dept. of Applid Mathematics and Statistics, > > > Stony Brook University, > > > Stony Brook, New York, 11790 > > > > > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Sep 17 00:20:46 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 17 Sep 2017 00:20:46 -0500 Subject: [petsc-users] PETSc KSPBCGSL with HYPRE boomeramg preconditioner ksp_monitor in poisson function In-Reply-To: References: <392EB91C-0ED7-480C-8190-ADED7E7CDE24@mcs.anl.gov> Message-ID: <9617D011-9AAA-4D52-9FED-63879C90B9B8@mcs.anl.gov> > On Sep 17, 2017, at 12:17 AM, Hao Zhang wrote: > > OK. I will check the output Calling KSPView() > > I am experimenting with -start_in_debugger. it's for serial not for parallel, right? It can be for parallel, opens an xterm with the debugger for each MPI process. You need to type cont in all the windows > > On Sun, Sep 17, 2017 at 1:12 AM, Barry Smith wrote: > > Should work in the code also. Call KSPSetTolerances() and then call KSPView() to see what it is set to > > > On Sep 17, 2017, at 12:11 AM, Hao Zhang wrote: > > > > Yes. but only works via command line option.before was set up in the code. > > > > On Sun, Sep 17, 2017 at 12:44 AM, Barry Smith wrote: > > > > > On Sep 16, 2017, at 11:33 PM, Hao Zhang wrote: > > > > > > thanks. > > > > > > Yes. KSPSetTolerances is used. > > > > > > tolerance is set to be 1.e-14 > > > > There is the rtol and atol are you sure you set the atol to 1e-14? > > > > > > > > I was thinking if there is a debug option I could use from PETSc to get some insight? GDB was used. > > > > You can use the command line option -start_in_debugger > > > > > > > > > > On Sun, Sep 17, 2017 at 12:25 AM, Barry Smith wrote: > > > > > > > On Sep 16, 2017, at 10:39 PM, Hao Zhang wrote: > > > > > > > > hi, > > > > > > > > I am using KSPBCGSL method and HYPRE boomeramg precondtioner passed into KSPsolve for poisson equation as part of incompressible NS. > > > > > > > > I have two questions > > > > Q1 is number of iterations for the same solver. max_iter = 20 for the same poisson function > > > > > > > > KSPGetIterationNumber() is used to set up max_iter = 20 > > > > > > What do you mean, KSPGetIterationNumber() tells you the current number of iterations. You cannot use it to set the maximum number of iterations. Use KSPSetTolerances() or -ksp_max_it to set the number > > > > > > > > > > > > 0 KSP Residual norm 4.512245447770e-04 > > > > 2 KSP Residual norm 6.396709069731e-07 > > > > 4 KSP Residual norm 1.757784220489e-10 > > > > 6 KSP Residual norm 1.501464709144e-14 > > > > 8 KSP Residual norm 1.376381429122e-18 > > > > 10 KSP Residual norm 1.072650548945e-19 > > > > In poisson_func(): num_iter = 10, rel_residual = 1.072651e-19 > > > > > > > > and the very same function at the next time step gives: remind you guys max_iter = 20. I assume the number of iterations should stop at num_iter = 2. > > > > > > Why would it stop at num_iter 2. The number of iterations allowed is the same for each solve, it does not accumulate over all solves. > > > > > > > but the solver seems to keep iterating util hitting the max_iter. > > > > 0 KSP Residual norm 5.107005838093e-16 > > > > 2 KSP Residual norm 3.353634821198e-20 > > > > 4 KSP Residual norm 1.835096039266e-23 > > > > 6 KSP Residual norm 1.645496102409e-23 > > > > 8 KSP Residual norm 1.645496099837e-23 > > > > 10 KSP Residual norm 1.645496099836e-23 > > > > 12 KSP Residual norm 1.645496099530e-23 > > > > 14 KSP Residual norm 1.645496034473e-23 > > > > 16 KSP Residual norm 1.645461130961e-23 > > > > 18 KSP Residual norm 1.645460451075e-23 > > > > In poisson_func(): num_iter = 18, rel_residual = 1.645460e-23 > > > > > > But the way it is almost never realistic to try to get residual norms to be this small. You might consider using something like -ksp_atol 1.e-16 or something > > > > > > > > > > > Q2 is: it seems HYPRE with KSPsolve works with coarse mesh but when I double the mesh size. the solve halted indefinitely. ksp_monitor didn't produce any results. > > > > is there a way to understand what's happening? > > > > > > You could run in the debugger and use control c to interrupt the code when it is "hanging" and type bt to see the stack trace for the hanging. > > > > > > > > > > > > > > -- > > > > Hao Zhang > > > > Dept. of Applid Mathematics and Statistics, > > > > Stony Brook University, > > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > > > -- > > > Hao Zhang > > > Dept. of Applid Mathematics and Statistics, > > > Stony Brook University, > > > Stony Brook, New York, 11790 > > > > > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > > > > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 From hbcbh1999 at gmail.com Sun Sep 17 00:22:04 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Sun, 17 Sep 2017 01:22:04 -0400 Subject: [petsc-users] PETSc KSPBCGSL with HYPRE boomeramg preconditioner ksp_monitor in poisson function In-Reply-To: <9617D011-9AAA-4D52-9FED-63879C90B9B8@mcs.anl.gov> References: <392EB91C-0ED7-480C-8190-ADED7E7CDE24@mcs.anl.gov> <9617D011-9AAA-4D52-9FED-63879C90B9B8@mcs.anl.gov> Message-ID: Thanks! On Sun, Sep 17, 2017 at 1:20 AM, Barry Smith wrote: > > > On Sep 17, 2017, at 12:17 AM, Hao Zhang wrote: > > > > OK. I will check the output Calling KSPView() > > > > I am experimenting with -start_in_debugger. it's for serial not for > parallel, right? > > It can be for parallel, opens an xterm with the debugger for each MPI > process. You need to type cont in all the windows > > > > > > On Sun, Sep 17, 2017 at 1:12 AM, Barry Smith wrote: > > > > Should work in the code also. Call KSPSetTolerances() and then call > KSPView() to see what it is set to > > > > > On Sep 17, 2017, at 12:11 AM, Hao Zhang wrote: > > > > > > Yes. but only works via command line option.before was set up in the > code. > > > > > > On Sun, Sep 17, 2017 at 12:44 AM, Barry Smith > wrote: > > > > > > > On Sep 16, 2017, at 11:33 PM, Hao Zhang wrote: > > > > > > > > thanks. > > > > > > > > Yes. KSPSetTolerances is used. > > > > > > > > tolerance is set to be 1.e-14 > > > > > > There is the rtol and atol are you sure you set the atol to 1e-14? > > > > > > > > > > > I was thinking if there is a debug option I could use from PETSc to > get some insight? GDB was used. > > > > > > You can use the command line option -start_in_debugger > > > > > > > > > > > > > > On Sun, Sep 17, 2017 at 12:25 AM, Barry Smith > wrote: > > > > > > > > > On Sep 16, 2017, at 10:39 PM, Hao Zhang > wrote: > > > > > > > > > > hi, > > > > > > > > > > I am using KSPBCGSL method and HYPRE boomeramg precondtioner > passed into KSPsolve for poisson equation as part of incompressible NS. > > > > > > > > > > I have two questions > > > > > Q1 is number of iterations for the same solver. max_iter = 20 for > the same poisson function > > > > > > > > > > KSPGetIterationNumber() is used to set up max_iter = 20 > > > > > > > > What do you mean, KSPGetIterationNumber() tells you the current > number of iterations. You cannot use it to set the maximum number of > iterations. Use KSPSetTolerances() or -ksp_max_it to set the number > > > > > > > > > > > > > > > 0 KSP Residual norm 4.512245447770e-04 > > > > > 2 KSP Residual norm 6.396709069731e-07 > > > > > 4 KSP Residual norm 1.757784220489e-10 > > > > > 6 KSP Residual norm 1.501464709144e-14 > > > > > 8 KSP Residual norm 1.376381429122e-18 > > > > > 10 KSP Residual norm 1.072650548945e-19 > > > > > In poisson_func(): num_iter = 10, rel_residual = 1.072651e-19 > > > > > > > > > > and the very same function at the next time step gives: remind you > guys max_iter = 20. I assume the number of iterations should stop at > num_iter = 2. > > > > > > > > Why would it stop at num_iter 2. The number of iterations allowed > is the same for each solve, it does not accumulate over all solves. > > > > > > > > > but the solver seems to keep iterating util hitting the max_iter. > > > > > 0 KSP Residual norm 5.107005838093e-16 > > > > > 2 KSP Residual norm 3.353634821198e-20 > > > > > 4 KSP Residual norm 1.835096039266e-23 > > > > > 6 KSP Residual norm 1.645496102409e-23 > > > > > 8 KSP Residual norm 1.645496099837e-23 > > > > > 10 KSP Residual norm 1.645496099836e-23 > > > > > 12 KSP Residual norm 1.645496099530e-23 > > > > > 14 KSP Residual norm 1.645496034473e-23 > > > > > 16 KSP Residual norm 1.645461130961e-23 > > > > > 18 KSP Residual norm 1.645460451075e-23 > > > > > In poisson_func(): num_iter = 18, rel_residual = 1.645460e-23 > > > > > > > > But the way it is almost never realistic to try to get residual > norms to be this small. You might consider using something like -ksp_atol > 1.e-16 or something > > > > > > > > > > > > > > Q2 is: it seems HYPRE with KSPsolve works with coarse mesh but > when I double the mesh size. the solve halted indefinitely. ksp_monitor > didn't produce any results. > > > > > is there a way to understand what's happening? > > > > > > > > You could run in the debugger and use control c to interrupt the > code when it is "hanging" and type bt to see the stack trace for the > hanging. > > > > > > > > > > > > > > > > > > -- > > > > > Hao Zhang > > > > > Dept. of Applid Mathematics and Statistics, > > > > > Stony Brook University, > > > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > > > > > > > > -- > > > > Hao Zhang > > > > Dept. of Applid Mathematics and Statistics, > > > > Stony Brook University, > > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > > > -- > > > Hao Zhang > > > Dept. of Applid Mathematics and Statistics, > > > Stony Brook University, > > > Stony Brook, New York, 11790 > > > > > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbcbh1999 at gmail.com Sun Sep 17 00:33:50 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Sun, 17 Sep 2017 01:33:50 -0400 Subject: [petsc-users] PETSc KSPBCGSL with HYPRE boomeramg preconditioner ksp_monitor in poisson function In-Reply-To: References: <392EB91C-0ED7-480C-8190-ADED7E7CDE24@mcs.anl.gov> <9617D011-9AAA-4D52-9FED-63879C90B9B8@mcs.anl.gov> Message-ID: Call KSPView(). this is I have from the output for my COARSE grid: imin = 4 imax = 41 jmin = 4 jmax = 8 kmin = 4 kmax = 20 KSP Object: 24 MPI processes type not yet set maximum iterations=20, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using DEFAULT norm type for convergence test PC Object: 24 MPI processes type not yet set PC has not been set up so information may be incomplete KSP Object: 24 MPI processes type not yet set maximum iterations=20, initial guess is zero tolerances: relative=1e-14, absolute=1e-50, divergence=10000 left preconditioning using DEFAULT norm type for convergence test PC Object: 24 MPI processes type not yet set PC has not been set up so information may be incomplete Using Neumann Solver 0 KSP Residual norm 4.933753594518e+00 2 KSP Residual norm 1.028883218492e-03 4 KSP Residual norm 2.980337307701e-06 6 KSP Residual norm 2.976110274418e-06 8 KSP Residual norm 2.976110265786e-06 10 KSP Residual norm 2.976110265696e-06 12 KSP Residual norm 2.976158755617e-06 14 KSP Residual norm 2.976038363683e-06 16 KSP Residual norm 2.976102606899e-06 18 KSP Residual norm 2.976429345213e-06 Linear solve did not converge due to DIVERGED_ITS iterations 18 The solution diverges! The residual is 2.976429e-06. Solve again using GMRES! 0 KSP Residual norm 4.932103470452e+00 1 KSP Residual norm 4.586081825060e-01 2 KSP Residual norm 4.702606826126e-02 3 KSP Residual norm 5.717152414461e-03 4 KSP Residual norm 5.122354290806e-04 5 KSP Residual norm 4.682702445806e-05 6 KSP Residual norm 4.368929895884e-06 7 KSP Residual norm 4.908451624092e-07 8 KSP Residual norm 4.758895647974e-08 9 KSP Residual norm 3.831683374269e-09 10 KSP Residual norm 2.844848230467e-10 11 KSP Residual norm 2.252748969152e-11 12 KSP Residual norm 1.432700892770e-12 13 KSP Residual norm 9.569163228927e-14 14 KSP Residual norm 1.129667714748e-14 Linear solve converged due to CONVERGED_RTOL iterations 14 In poisson_func(): num_iter = 14, rel_residual = 1.129668e-14 for my REFINE grid: imin = 4 imax = 28 jmin = 4 jmax = 10 kmin = 4 kmax = 28 KSP Object: 96 MPI processes type not yet set maximum iterations=20, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using DEFAULT norm type for convergence test PC Object: 96 MPI processes type not yet set PC has not been set up so information may be incomplete KSP Object: 96 MPI processes type not yet set maximum iterations=20, initial guess is zero tolerances: relative=1e-14, absolute=1e-50, divergence=10000 left preconditioning using DEFAULT norm type for convergence test PC Object: 96 MPI processes type not yet set PC has not been set up so information may be incomplete On Sun, Sep 17, 2017 at 1:22 AM, Hao Zhang wrote: > Thanks! > > On Sun, Sep 17, 2017 at 1:20 AM, Barry Smith wrote: > >> >> > On Sep 17, 2017, at 12:17 AM, Hao Zhang wrote: >> > >> > OK. I will check the output Calling KSPView() >> > >> > I am experimenting with -start_in_debugger. it's for serial not for >> parallel, right? >> >> It can be for parallel, opens an xterm with the debugger for each MPI >> process. You need to type cont in all the windows >> >> >> > >> > On Sun, Sep 17, 2017 at 1:12 AM, Barry Smith >> wrote: >> > >> > Should work in the code also. Call KSPSetTolerances() and then call >> KSPView() to see what it is set to >> > >> > > On Sep 17, 2017, at 12:11 AM, Hao Zhang wrote: >> > > >> > > Yes. but only works via command line option.before was set up in the >> code. >> > > >> > > On Sun, Sep 17, 2017 at 12:44 AM, Barry Smith >> wrote: >> > > >> > > > On Sep 16, 2017, at 11:33 PM, Hao Zhang >> wrote: >> > > > >> > > > thanks. >> > > > >> > > > Yes. KSPSetTolerances is used. >> > > > >> > > > tolerance is set to be 1.e-14 >> > > >> > > There is the rtol and atol are you sure you set the atol to 1e-14? >> > > >> > > > >> > > > I was thinking if there is a debug option I could use from PETSc to >> get some insight? GDB was used. >> > > >> > > You can use the command line option -start_in_debugger >> > > >> > > >> > > > >> > > > On Sun, Sep 17, 2017 at 12:25 AM, Barry Smith >> wrote: >> > > > >> > > > > On Sep 16, 2017, at 10:39 PM, Hao Zhang >> wrote: >> > > > > >> > > > > hi, >> > > > > >> > > > > I am using KSPBCGSL method and HYPRE boomeramg precondtioner >> passed into KSPsolve for poisson equation as part of incompressible NS. >> > > > > >> > > > > I have two questions >> > > > > Q1 is number of iterations for the same solver. max_iter = 20 for >> the same poisson function >> > > > > >> > > > > KSPGetIterationNumber() is used to set up max_iter = 20 >> > > > >> > > > What do you mean, KSPGetIterationNumber() tells you the current >> number of iterations. You cannot use it to set the maximum number of >> iterations. Use KSPSetTolerances() or -ksp_max_it to set the number >> > > > > >> > > > > >> > > > > 0 KSP Residual norm 4.512245447770e-04 >> > > > > 2 KSP Residual norm 6.396709069731e-07 >> > > > > 4 KSP Residual norm 1.757784220489e-10 >> > > > > 6 KSP Residual norm 1.501464709144e-14 >> > > > > 8 KSP Residual norm 1.376381429122e-18 >> > > > > 10 KSP Residual norm 1.072650548945e-19 >> > > > > In poisson_func(): num_iter = 10, rel_residual = 1.072651e-19 >> > > > > >> > > > > and the very same function at the next time step gives: remind >> you guys max_iter = 20. I assume the number of iterations should stop at >> num_iter = 2. >> > > > >> > > > Why would it stop at num_iter 2. The number of iterations >> allowed is the same for each solve, it does not accumulate over all solves. >> > > > >> > > > > but the solver seems to keep iterating util hitting the max_iter. >> > > > > 0 KSP Residual norm 5.107005838093e-16 >> > > > > 2 KSP Residual norm 3.353634821198e-20 >> > > > > 4 KSP Residual norm 1.835096039266e-23 >> > > > > 6 KSP Residual norm 1.645496102409e-23 >> > > > > 8 KSP Residual norm 1.645496099837e-23 >> > > > > 10 KSP Residual norm 1.645496099836e-23 >> > > > > 12 KSP Residual norm 1.645496099530e-23 >> > > > > 14 KSP Residual norm 1.645496034473e-23 >> > > > > 16 KSP Residual norm 1.645461130961e-23 >> > > > > 18 KSP Residual norm 1.645460451075e-23 >> > > > > In poisson_func(): num_iter = 18, rel_residual = 1.645460e-23 >> > > > >> > > > But the way it is almost never realistic to try to get residual >> norms to be this small. You might consider using something like -ksp_atol >> 1.e-16 or something >> > > > >> > > > > >> > > > > Q2 is: it seems HYPRE with KSPsolve works with coarse mesh but >> when I double the mesh size. the solve halted indefinitely. ksp_monitor >> didn't produce any results. >> > > > > is there a way to understand what's happening? >> > > > >> > > > You could run in the debugger and use control c to interrupt the >> code when it is "hanging" and type bt to see the stack trace for the >> hanging. >> > > > >> > > > >> > > > > >> > > > > -- >> > > > > Hao Zhang >> > > > > Dept. of Applid Mathematics and Statistics, >> > > > > Stony Brook University, >> > > > > Stony Brook, New York, 11790 >> > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > Hao Zhang >> > > > Dept. of Applid Mathematics and Statistics, >> > > > Stony Brook University, >> > > > Stony Brook, New York, 11790 >> > > >> > > >> > > >> > > >> > > -- >> > > Hao Zhang >> > > Dept. of Applid Mathematics and Statistics, >> > > Stony Brook University, >> > > Stony Brook, New York, 11790 >> > >> > >> > >> > >> > -- >> > Hao Zhang >> > Dept. of Applid Mathematics and Statistics, >> > Stony Brook University, >> > Stony Brook, New York, 11790 >> >> > > > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Sep 17 00:38:36 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 17 Sep 2017 00:38:36 -0500 Subject: [petsc-users] PETSc KSPBCGSL with HYPRE boomeramg preconditioner ksp_monitor in poisson function In-Reply-To: References: <392EB91C-0ED7-480C-8190-ADED7E7CDE24@mcs.anl.gov> <9617D011-9AAA-4D52-9FED-63879C90B9B8@mcs.anl.gov> Message-ID: <7AC41DBE-9F07-4F8E-88E7-DAEE0D2712C6@mcs.anl.gov> Send a code fragment that reproduces the problem; you must be calling on the wrong KSP or at the wrong place in the code > On Sep 17, 2017, at 12:33 AM, Hao Zhang wrote: > > Call KSPView(). this is I have from the output > > for my COARSE grid: > > imin = 4 imax = 41 > jmin = 4 jmax = 8 > kmin = 4 kmax = 20 > KSP Object: 24 MPI processes > type not yet set > maximum iterations=20, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: 24 MPI processes > type not yet set > PC has not been set up so information may be incomplete > KSP Object: 24 MPI processes > type not yet set > maximum iterations=20, initial guess is zero > tolerances: relative=1e-14, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: 24 MPI processes > type not yet set > PC has not been set up so information may be incomplete > > Using Neumann Solver > 0 KSP Residual norm 4.933753594518e+00 > 2 KSP Residual norm 1.028883218492e-03 > 4 KSP Residual norm 2.980337307701e-06 > 6 KSP Residual norm 2.976110274418e-06 > 8 KSP Residual norm 2.976110265786e-06 > 10 KSP Residual norm 2.976110265696e-06 > 12 KSP Residual norm 2.976158755617e-06 > 14 KSP Residual norm 2.976038363683e-06 > 16 KSP Residual norm 2.976102606899e-06 > 18 KSP Residual norm 2.976429345213e-06 > Linear solve did not converge due to DIVERGED_ITS iterations 18 > > The solution diverges! The residual is 2.976429e-06. Solve again using GMRES! > 0 KSP Residual norm 4.932103470452e+00 > 1 KSP Residual norm 4.586081825060e-01 > 2 KSP Residual norm 4.702606826126e-02 > 3 KSP Residual norm 5.717152414461e-03 > 4 KSP Residual norm 5.122354290806e-04 > 5 KSP Residual norm 4.682702445806e-05 > 6 KSP Residual norm 4.368929895884e-06 > 7 KSP Residual norm 4.908451624092e-07 > 8 KSP Residual norm 4.758895647974e-08 > 9 KSP Residual norm 3.831683374269e-09 > 10 KSP Residual norm 2.844848230467e-10 > 11 KSP Residual norm 2.252748969152e-11 > 12 KSP Residual norm 1.432700892770e-12 > 13 KSP Residual norm 9.569163228927e-14 > 14 KSP Residual norm 1.129667714748e-14 > Linear solve converged due to CONVERGED_RTOL iterations 14 > > In poisson_func(): num_iter = 14, rel_residual = 1.129668e-14 > > for my REFINE grid: > > imin = 4 imax = 28 > jmin = 4 jmax = 10 > kmin = 4 kmax = 28 > KSP Object: 96 MPI processes > type not yet set > maximum iterations=20, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: 96 MPI processes > type not yet set > PC has not been set up so information may be incomplete > KSP Object: 96 MPI processes > type not yet set > maximum iterations=20, initial guess is zero > tolerances: relative=1e-14, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: 96 MPI processes > type not yet set > PC has not been set up so information may be incomplete > > On Sun, Sep 17, 2017 at 1:22 AM, Hao Zhang wrote: > Thanks! > > On Sun, Sep 17, 2017 at 1:20 AM, Barry Smith wrote: > > > On Sep 17, 2017, at 12:17 AM, Hao Zhang wrote: > > > > OK. I will check the output Calling KSPView() > > > > I am experimenting with -start_in_debugger. it's for serial not for parallel, right? > > It can be for parallel, opens an xterm with the debugger for each MPI process. You need to type cont in all the windows > > > > > > On Sun, Sep 17, 2017 at 1:12 AM, Barry Smith wrote: > > > > Should work in the code also. Call KSPSetTolerances() and then call KSPView() to see what it is set to > > > > > On Sep 17, 2017, at 12:11 AM, Hao Zhang wrote: > > > > > > Yes. but only works via command line option.before was set up in the code. > > > > > > On Sun, Sep 17, 2017 at 12:44 AM, Barry Smith wrote: > > > > > > > On Sep 16, 2017, at 11:33 PM, Hao Zhang wrote: > > > > > > > > thanks. > > > > > > > > Yes. KSPSetTolerances is used. > > > > > > > > tolerance is set to be 1.e-14 > > > > > > There is the rtol and atol are you sure you set the atol to 1e-14? > > > > > > > > > > > I was thinking if there is a debug option I could use from PETSc to get some insight? GDB was used. > > > > > > You can use the command line option -start_in_debugger > > > > > > > > > > > > > > On Sun, Sep 17, 2017 at 12:25 AM, Barry Smith wrote: > > > > > > > > > On Sep 16, 2017, at 10:39 PM, Hao Zhang wrote: > > > > > > > > > > hi, > > > > > > > > > > I am using KSPBCGSL method and HYPRE boomeramg precondtioner passed into KSPsolve for poisson equation as part of incompressible NS. > > > > > > > > > > I have two questions > > > > > Q1 is number of iterations for the same solver. max_iter = 20 for the same poisson function > > > > > > > > > > KSPGetIterationNumber() is used to set up max_iter = 20 > > > > > > > > What do you mean, KSPGetIterationNumber() tells you the current number of iterations. You cannot use it to set the maximum number of iterations. Use KSPSetTolerances() or -ksp_max_it to set the number > > > > > > > > > > > > > > > 0 KSP Residual norm 4.512245447770e-04 > > > > > 2 KSP Residual norm 6.396709069731e-07 > > > > > 4 KSP Residual norm 1.757784220489e-10 > > > > > 6 KSP Residual norm 1.501464709144e-14 > > > > > 8 KSP Residual norm 1.376381429122e-18 > > > > > 10 KSP Residual norm 1.072650548945e-19 > > > > > In poisson_func(): num_iter = 10, rel_residual = 1.072651e-19 > > > > > > > > > > and the very same function at the next time step gives: remind you guys max_iter = 20. I assume the number of iterations should stop at num_iter = 2. > > > > > > > > Why would it stop at num_iter 2. The number of iterations allowed is the same for each solve, it does not accumulate over all solves. > > > > > > > > > but the solver seems to keep iterating util hitting the max_iter. > > > > > 0 KSP Residual norm 5.107005838093e-16 > > > > > 2 KSP Residual norm 3.353634821198e-20 > > > > > 4 KSP Residual norm 1.835096039266e-23 > > > > > 6 KSP Residual norm 1.645496102409e-23 > > > > > 8 KSP Residual norm 1.645496099837e-23 > > > > > 10 KSP Residual norm 1.645496099836e-23 > > > > > 12 KSP Residual norm 1.645496099530e-23 > > > > > 14 KSP Residual norm 1.645496034473e-23 > > > > > 16 KSP Residual norm 1.645461130961e-23 > > > > > 18 KSP Residual norm 1.645460451075e-23 > > > > > In poisson_func(): num_iter = 18, rel_residual = 1.645460e-23 > > > > > > > > But the way it is almost never realistic to try to get residual norms to be this small. You might consider using something like -ksp_atol 1.e-16 or something > > > > > > > > > > > > > > Q2 is: it seems HYPRE with KSPsolve works with coarse mesh but when I double the mesh size. the solve halted indefinitely. ksp_monitor didn't produce any results. > > > > > is there a way to understand what's happening? > > > > > > > > You could run in the debugger and use control c to interrupt the code when it is "hanging" and type bt to see the stack trace for the hanging. > > > > > > > > > > > > > > > > > > -- > > > > > Hao Zhang > > > > > Dept. of Applid Mathematics and Statistics, > > > > > Stony Brook University, > > > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > > > > > > > > -- > > > > Hao Zhang > > > > Dept. of Applid Mathematics and Statistics, > > > > Stony Brook University, > > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > > > -- > > > Hao Zhang > > > Dept. of Applid Mathematics and Statistics, > > > Stony Brook University, > > > Stony Brook, New York, 11790 > > > > > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > > > > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 > > > > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 From hbcbh1999 at gmail.com Sun Sep 17 09:29:42 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Sun, 17 Sep 2017 10:29:42 -0400 Subject: [petsc-users] PETSc KSPBCGSL with HYPRE boomeramg preconditioner ksp_monitor in poisson function In-Reply-To: <7AC41DBE-9F07-4F8E-88E7-DAEE0D2712C6@mcs.anl.gov> References: <392EB91C-0ED7-480C-8190-ADED7E7CDE24@mcs.anl.gov> <9617D011-9AAA-4D52-9FED-63879C90B9B8@mcs.anl.gov> <7AC41DBE-9F07-4F8E-88E7-DAEE0D2712C6@mcs.anl.gov> Message-ID: Let me get more debugging information first. code were implemented in a bunch of different files. simply copy-and-paste will make it very messy. On Sun, Sep 17, 2017 at 1:38 AM, Barry Smith wrote: > > Send a code fragment that reproduces the problem; you must be calling on > the wrong KSP or at the wrong place in the code > > > On Sep 17, 2017, at 12:33 AM, Hao Zhang wrote: > > > > Call KSPView(). this is I have from the output > > > > for my COARSE grid: > > > > imin = 4 imax = 41 > > jmin = 4 jmax = 8 > > kmin = 4 kmax = 20 > > KSP Object: 24 MPI processes > > type not yet set > > maximum iterations=20, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: 24 MPI processes > > type not yet set > > PC has not been set up so information may be incomplete > > KSP Object: 24 MPI processes > > type not yet set > > maximum iterations=20, initial guess is zero > > tolerances: relative=1e-14, absolute=1e-50, divergence=10000 > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: 24 MPI processes > > type not yet set > > PC has not been set up so information may be incomplete > > > > Using Neumann Solver > > 0 KSP Residual norm 4.933753594518e+00 > > 2 KSP Residual norm 1.028883218492e-03 > > 4 KSP Residual norm 2.980337307701e-06 > > 6 KSP Residual norm 2.976110274418e-06 > > 8 KSP Residual norm 2.976110265786e-06 > > 10 KSP Residual norm 2.976110265696e-06 > > 12 KSP Residual norm 2.976158755617e-06 > > 14 KSP Residual norm 2.976038363683e-06 > > 16 KSP Residual norm 2.976102606899e-06 > > 18 KSP Residual norm 2.976429345213e-06 > > Linear solve did not converge due to DIVERGED_ITS iterations 18 > > > > The solution diverges! The residual is 2.976429e-06. Solve again using > GMRES! > > 0 KSP Residual norm 4.932103470452e+00 > > 1 KSP Residual norm 4.586081825060e-01 > > 2 KSP Residual norm 4.702606826126e-02 > > 3 KSP Residual norm 5.717152414461e-03 > > 4 KSP Residual norm 5.122354290806e-04 > > 5 KSP Residual norm 4.682702445806e-05 > > 6 KSP Residual norm 4.368929895884e-06 > > 7 KSP Residual norm 4.908451624092e-07 > > 8 KSP Residual norm 4.758895647974e-08 > > 9 KSP Residual norm 3.831683374269e-09 > > 10 KSP Residual norm 2.844848230467e-10 > > 11 KSP Residual norm 2.252748969152e-11 > > 12 KSP Residual norm 1.432700892770e-12 > > 13 KSP Residual norm 9.569163228927e-14 > > 14 KSP Residual norm 1.129667714748e-14 > > Linear solve converged due to CONVERGED_RTOL iterations 14 > > > > In poisson_func(): num_iter = 14, rel_residual = 1.129668e-14 > > > > for my REFINE grid: > > > > imin = 4 imax = 28 > > jmin = 4 jmax = 10 > > kmin = 4 kmax = 28 > > KSP Object: 96 MPI processes > > type not yet set > > maximum iterations=20, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: 96 MPI processes > > type not yet set > > PC has not been set up so information may be incomplete > > KSP Object: 96 MPI processes > > type not yet set > > maximum iterations=20, initial guess is zero > > tolerances: relative=1e-14, absolute=1e-50, divergence=10000 > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: 96 MPI processes > > type not yet set > > PC has not been set up so information may be incomplete > > > > On Sun, Sep 17, 2017 at 1:22 AM, Hao Zhang wrote: > > Thanks! > > > > On Sun, Sep 17, 2017 at 1:20 AM, Barry Smith wrote: > > > > > On Sep 17, 2017, at 12:17 AM, Hao Zhang wrote: > > > > > > OK. I will check the output Calling KSPView() > > > > > > I am experimenting with -start_in_debugger. it's for serial not for > parallel, right? > > > > It can be for parallel, opens an xterm with the debugger for each MPI > process. You need to type cont in all the windows > > > > > > > > > > On Sun, Sep 17, 2017 at 1:12 AM, Barry Smith > wrote: > > > > > > Should work in the code also. Call KSPSetTolerances() and then call > KSPView() to see what it is set to > > > > > > > On Sep 17, 2017, at 12:11 AM, Hao Zhang wrote: > > > > > > > > Yes. but only works via command line option.before was set up in the > code. > > > > > > > > On Sun, Sep 17, 2017 at 12:44 AM, Barry Smith > wrote: > > > > > > > > > On Sep 16, 2017, at 11:33 PM, Hao Zhang > wrote: > > > > > > > > > > thanks. > > > > > > > > > > Yes. KSPSetTolerances is used. > > > > > > > > > > tolerance is set to be 1.e-14 > > > > > > > > There is the rtol and atol are you sure you set the atol to 1e-14? > > > > > > > > > > > > > > I was thinking if there is a debug option I could use from PETSc > to get some insight? GDB was used. > > > > > > > > You can use the command line option -start_in_debugger > > > > > > > > > > > > > > > > > > On Sun, Sep 17, 2017 at 12:25 AM, Barry Smith > wrote: > > > > > > > > > > > On Sep 16, 2017, at 10:39 PM, Hao Zhang > wrote: > > > > > > > > > > > > hi, > > > > > > > > > > > > I am using KSPBCGSL method and HYPRE boomeramg precondtioner > passed into KSPsolve for poisson equation as part of incompressible NS. > > > > > > > > > > > > I have two questions > > > > > > Q1 is number of iterations for the same solver. max_iter = 20 > for the same poisson function > > > > > > > > > > > > KSPGetIterationNumber() is used to set up max_iter = 20 > > > > > > > > > > What do you mean, KSPGetIterationNumber() tells you the current > number of iterations. You cannot use it to set the maximum number of > iterations. Use KSPSetTolerances() or -ksp_max_it to set the number > > > > > > > > > > > > > > > > > > 0 KSP Residual norm 4.512245447770e-04 > > > > > > 2 KSP Residual norm 6.396709069731e-07 > > > > > > 4 KSP Residual norm 1.757784220489e-10 > > > > > > 6 KSP Residual norm 1.501464709144e-14 > > > > > > 8 KSP Residual norm 1.376381429122e-18 > > > > > > 10 KSP Residual norm 1.072650548945e-19 > > > > > > In poisson_func(): num_iter = 10, rel_residual = 1.072651e-19 > > > > > > > > > > > > and the very same function at the next time step gives: remind > you guys max_iter = 20. I assume the number of iterations should stop at > num_iter = 2. > > > > > > > > > > Why would it stop at num_iter 2. The number of iterations > allowed is the same for each solve, it does not accumulate over all solves. > > > > > > > > > > > but the solver seems to keep iterating util hitting the > max_iter. > > > > > > 0 KSP Residual norm 5.107005838093e-16 > > > > > > 2 KSP Residual norm 3.353634821198e-20 > > > > > > 4 KSP Residual norm 1.835096039266e-23 > > > > > > 6 KSP Residual norm 1.645496102409e-23 > > > > > > 8 KSP Residual norm 1.645496099837e-23 > > > > > > 10 KSP Residual norm 1.645496099836e-23 > > > > > > 12 KSP Residual norm 1.645496099530e-23 > > > > > > 14 KSP Residual norm 1.645496034473e-23 > > > > > > 16 KSP Residual norm 1.645461130961e-23 > > > > > > 18 KSP Residual norm 1.645460451075e-23 > > > > > > In poisson_func(): num_iter = 18, rel_residual = 1.645460e-23 > > > > > > > > > > But the way it is almost never realistic to try to get residual > norms to be this small. You might consider using something like -ksp_atol > 1.e-16 or something > > > > > > > > > > > > > > > > > Q2 is: it seems HYPRE with KSPsolve works with coarse mesh but > when I double the mesh size. the solve halted indefinitely. ksp_monitor > didn't produce any results. > > > > > > is there a way to understand what's happening? > > > > > > > > > > You could run in the debugger and use control c to interrupt > the code when it is "hanging" and type bt to see the stack trace for the > hanging. > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Hao Zhang > > > > > > Dept. of Applid Mathematics and Statistics, > > > > > > Stony Brook University, > > > > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Hao Zhang > > > > > Dept. of Applid Mathematics and Statistics, > > > > > Stony Brook University, > > > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > > > > > > > > -- > > > > Hao Zhang > > > > Dept. of Applid Mathematics and Statistics, > > > > Stony Brook University, > > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > > > -- > > > Hao Zhang > > > Dept. of Applid Mathematics and Statistics, > > > Stony Brook University, > > > Stony Brook, New York, 11790 > > > > > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From k_burkart at yahoo.com Sun Sep 17 09:25:51 2017 From: k_burkart at yahoo.com (Klaus Burkart) Date: Sun, 17 Sep 2017 14:25:51 +0000 (UTC) Subject: [petsc-users] Loading a PETsc matrix with matrix data in CSR format? References: <876053565.1090563.1505658351610.ref@mail.yahoo.com> Message-ID: <876053565.1090563.1505658351610@mail.yahoo.com> The matrix import function looks like this: void csr2pet ( ??? const Foam::lduMatrix & matrix, ??? petsc_declaration & petsc_matrix? // How to declare the PETsc matrix to be filled? ) { ??? int n = matrix.diag().size(); // small case n = 40800 ??? int nnz = matrix.lower().size() + matrix.upper().size() + matrix.diag().size(); // small case nnz = 203800 ??? // allocate memory for CSR sparse matrix using calloc ??? ScalarType * vals = (ScalarType *)calloc(nnz, sizeof(ScalarType)); ??? uint * cols = (uint *)calloc(nnz, sizeof(uint)); ??? uint * rows = (uint *)calloc(n, sizeof(uint)); ??? // call function to convert original LDU matrix to CSR format ??? exPet::ldu2csr(matrix,rows,cols,vals); ??? // fill PETsc matrix ??? MatSetValues(petsc_matrix, ?, ?, ?, ?, ?, INSERT_VALUES); ??? // free and release the matrix memory ??? free(rows); free(cols); free(vals);? // calloc() } Questions: 1: How to declare the petsc_matrix to be filled by the function with the content of the original matrix? 2: MatSetValues(petsc_matrix, ?, ?, ?, ?, ?, INSERT_VALUES); is used to actually fill the petsc_matrix and I was of the opinion that PETsc uses the CSR format but I can't work out how CSR format is described by: ??? v ??? ??? - a logically two-dimensional array of values ??? m, idxm ??? - the number of rows and their global indices ??? n, idxn ??? - the number of columns and their global indices My original matrix is converted to CSR format, i.e. three arrays cols (column_indices), rows (row_start_indices) and vals (values). How can I load my matrix into a PETsc matrix for parallel processing? MatSetValues(petsc_matrix, ?, ?, ?, ?, ?, INSERT_VALUES); Klaus -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Sep 17 09:38:26 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 17 Sep 2017 09:38:26 -0500 Subject: [petsc-users] Loading a PETsc matrix with matrix data in CSR format? In-Reply-To: <876053565.1090563.1505658351610@mail.yahoo.com> References: <876053565.1090563.1505658351610.ref@mail.yahoo.com> <876053565.1090563.1505658351610@mail.yahoo.com> Message-ID: 1) MatSetValues() has absolutely nothing to do with the format used internally by PETSc to store the matrix. MatSetValues() is used to convey blocks of values into the matrix. 2) If you want to load a matrix onto multiple processes from a file NEVER write your own parallel ASCII matrix reader. Instead in either C, C++, Python, or Matlab write a routine that reads in the matrix SEQUENTIALLY and then saves it with MatView() and a binary viewer. You can then load the matrix easily and efficiently in PETSc in parallel with MatLoad 3) If you have matrix already in CSR format you can use MatCreateSeqAIJWithArrays() Barry > On Sep 17, 2017, at 9:25 AM, Klaus Burkart wrote: > > The matrix import function looks like this: > > void csr2pet > ( > const Foam::lduMatrix & matrix, > petsc_declaration & petsc_matrix // How to declare the PETsc matrix to be filled? > ) > { > int n = matrix.diag().size(); // small case n = 40800 > int nnz = matrix.lower().size() + matrix.upper().size() + matrix.diag().size(); // small case nnz = 203800 > > // allocate memory for CSR sparse matrix using calloc > ScalarType * vals = (ScalarType *)calloc(nnz, sizeof(ScalarType)); > uint * cols = (uint *)calloc(nnz, sizeof(uint)); > uint * rows = (uint *)calloc(n, sizeof(uint)); > > // call function to convert original LDU matrix to CSR format > exPet::ldu2csr(matrix,rows,cols,vals); > > // fill PETsc matrix > MatSetValues(petsc_matrix, ?, ?, ?, ?, ?, INSERT_VALUES); > > // free and release the matrix memory > free(rows); free(cols); free(vals); // calloc() > } > > > Questions: > > 1: How to declare the petsc_matrix to be filled by the function with the content of the original matrix? > > 2: MatSetValues(petsc_matrix, ?, ?, ?, ?, ?, INSERT_VALUES); is used to actually fill the petsc_matrix and I was of the opinion that PETsc uses the CSR format but I can't work out how CSR format is described by: > > v - a logically two-dimensional array of values > m, idxm - the number of rows and their global indices > n, idxn - the number of columns and their global indices > > My original matrix is converted to CSR format, i.e. three arrays cols (column_indices), rows (row_start_indices) and vals (values). > > How can I load my matrix into a PETsc matrix for parallel processing? MatSetValues(petsc_matrix, ?, ?, ?, ?, ?, INSERT_VALUES); > > Klaus From xsli at lbl.gov Sun Sep 17 19:42:26 2017 From: xsli at lbl.gov (Xiaoye S. Li) Date: Sun, 17 Sep 2017 17:42:26 -0700 Subject: [petsc-users] PDSLin 2.0.0 example memory corruption error In-Reply-To: References: Message-ID: This looks like error from ParMETIS -- some processes do not have initial vertices. Which version of PerMETIS are you using? There may be error in the way you run the test: $ mpiexec -n 8 ./dtest input matrices/U.dat Try to remove "input" in the command line. Can also try using smaller number of processes, e.g.: $ mpiexec -n 2 ./dtest matrices/U.dat Sherry Li On Fri, Sep 15, 2017 at 5:01 AM, Matthew Knepley wrote: > On Fri, Sep 15, 2017 at 7:00 AM, Afrah Najib > wrote: > >> Hi, >> I am trying to run the examples of PDSLin 2.0.0[http://portal.nersc.gov/ >> project/sparse/pdslin/] >> >> but I got the following error: >> > > This is not a PETSc error. PETSc is just catching the signal. I think you > have to contact > the PDSLin developers. > > Thanks, > > Matt > > >> :~/pdslin_2.0.0/examples$ mpiexec -n 8 ./dtest input matrices/U.dat >> input->num_doms = 4 >> Preconditioner parameters: >> input->nproc_dcomp = 4 >> input->nproc_schur = 4 >> input->free_local = PDSLin_NO >> input->dcomp_type = SCOTCH >> input->diag_tau = 0.000000e+00 >> input->tau_sub = 0.000000e+00 >> input->drop_tau0 = 1.000000e-02 >> input->drop_tau2 = 0.000000e+00 >> input->drop_tau1 = 1.000000e-03 >> input->drop_tau3 = 0.000000e+00 >> input->ilu_lvl = -1 >> input->equil_dom = 0 (no equil or row perm). >> input->perm_dom = 0 >> input->mat_type = UNSYMMETRIC >> input->mat_pattern = UNSYMMETRIC >> input->blk_size = 1 >> input->equil_schur = 5 >> input->relax_factor = 3.000000e-01 >> input->pperm_schur = PDSLin_YES >> input->psymb_schur = PDSLin_YES >> input->dom_solver = SLU_DIST >> Schur solution parameters: >> input->inner_outer = PDSLin_NO >> input->outer_tol = 1.000000e-12 >> input->outer_max = 50 >> input->schur_itrs = PDSLin_NO >> input->exact_schur = PDSLin_NO >> input->direct_schur = PDSLin_NO >> input->explicit_restart = PDSLin_NO >> input->inner_max = 500 >> input->inner_restart = 60 >> input->inner_tol = 1.000000e-12 >> input->inner_solver = PDSLin_FGMRES >> ==== dtest(): reading in matrix from matrices/U.dat >> took 1.785858e-01 seconds to read matrix. >> >> ********************************************************************** >> >> >> ==== dtest: factorizing the matrix. >> PARMETIS ERROR: Poor initial vertex distribution. Processor 2 has no >> vertices assigned to it! >> PARMETIS ERROR: Poor initial vertex distribution. Processor 3 has no >> vertices assigned to it! >> [0]PETSC ERROR: ------------------------------ >> ------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >> ocumentation/faq.html#valgrind >> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS >> X to find memory corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Signal received >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 >> [0]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah >> Fri Sep 15 13:53:42 2017 >> [0]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 >> PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs >> --download-scalapack=1 --download-mumps=1 --with-valgrind=1 >> --with-valgrind-dir="[/usr/local/bin]" >> [0]PETSC ERROR: #1 User provided function() line 0 in unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> [1]PETSC ERROR: ------------------------------ >> ------------------------------------------ >> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >> ocumentation/faq.html#valgrind >> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS >> X to find memory corruption errors >> [1]PETSC ERROR: likely location of problem given in stack below >> [1]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [1]PETSC ERROR: INSTEAD the line number of the start of the function >> [1]PETSC ERROR: is given. >> [1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [1]PETSC ERROR: Signal received >> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [1]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 >> [1]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah >> Fri Sep 15 13:53:42 2017 >> [1]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 >> PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs >> --download-scalapack=1 --download-mumps=1 --with-valgrind=1 >> --with-valgrind-dir="[/usr/local/bin]" >> [1]PETSC ERROR: #1 User provided function() line 0 in unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 >> [2]PETSC ERROR: ------------------------------ >> ------------------------------------------ >> [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >> ocumentation/faq.html#valgrind >> [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS >> X to find memory corruption errors >> [2]PETSC ERROR: likely location of problem given in stack below >> [2]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [2]PETSC ERROR: INSTEAD the line number of the start of the function >> [2]PETSC ERROR: is given. >> [2]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [2]PETSC ERROR: Signal received >> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [2]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 >> [2]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah >> Fri Sep 15 13:53:42 2017 >> [2]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 >> PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs >> --download-scalapack=1 --download-mumps=1 --with-valgrind=1 >> --with-valgrind-dir="[/usr/local/bin]" >> [2]PETSC ERROR: #1 User provided function() line 0 in unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 >> [3]PETSC ERROR: ------------------------------ >> ------------------------------------------ >> [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >> ocumentation/faq.html#valgrind >> [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS >> X to find memory corruption errors >> [3]PETSC ERROR: likely location of problem given in stack below >> [3]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [3]PETSC ERROR: INSTEAD the line number of the start of the function >> [3]PETSC ERROR: is given. >> [3]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [3]PETSC ERROR: Signal received >> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [3]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 >> [3]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah >> Fri Sep 15 13:53:42 2017 >> [3]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 >> PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs >> --download-scalapack=1 --download-mumps=1 --with-valgrind=1 >> --with-valgrind-dir="[/usr/local/bin]" >> [3]PETSC ERROR: #1 User provided function() line 0 in unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 >> [4]PETSC ERROR: ------------------------------ >> ------------------------------------------ >> [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >> ocumentation/faq.html#valgrind >> [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS >> X to find memory corruption errors >> [4]PETSC ERROR: likely location of problem given in stack below >> [4]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [4]PETSC ERROR: INSTEAD the line number of the start of the function >> [4]PETSC ERROR: is given. >> [4]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [4]PETSC ERROR: Signal received >> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [4]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 >> [4]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah >> Fri Sep 15 13:53:42 2017 >> [4]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 >> PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs >> --download-scalapack=1 --download-mumps=1 --with-valgrind=1 >> --with-valgrind-dir="[/usr/local/bin]" >> [4]PETSC ERROR: #1 User provided function() line 0 in unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4 >> [5]PETSC ERROR: ------------------------------ >> ------------------------------------------ >> [5]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >> ocumentation/faq.html#valgrind >> [5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS >> X to find memory corruption errors >> [5]PETSC ERROR: likely location of problem given in stack below >> [5]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [5]PETSC ERROR: INSTEAD the line number of the start of the function >> [5]PETSC ERROR: is given. >> [5]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [5]PETSC ERROR: Signal received >> [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [5]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 >> [5]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah >> Fri Sep 15 13:53:42 2017 >> [5]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 >> PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs >> --download-scalapack=1 --download-mumps=1 --with-valgrind=1 >> --with-valgrind-dir="[/usr/local/bin]" >> [5]PETSC ERROR: #1 User provided function() line 0 in unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5 >> [6]PETSC ERROR: [7]PETSC ERROR: ------------------------------ >> ------------------------------------------ >> [7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >> ocumentation/faq.html#valgrind >> [7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS >> X to find memory corruption errors >> [7]PETSC ERROR: likely location of problem given in stack below >> [7]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [7]PETSC ERROR: INSTEAD the line number of the start of the function >> [7]PETSC ERROR: is given. >> [7]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [7]PETSC ERROR: Signal received >> [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [7]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 >> [7]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah >> Fri Sep 15 13:53:42 2017 >> [7]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 >> PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs >> --download-scalapack=1 --download-mumps=1 --with-valgrind=1 >> --with-valgrind-dir="[/usr/local/bin]" >> [7]PETSC ERROR: #1 User provided function() line 0 in unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 7 >> ------------------------------------------------------------------------ >> [6]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >> ocumentation/faq.html#valgrind >> [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS >> X to find memory corruption errors >> [6]PETSC ERROR: likely location of problem given in stack below >> [6]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [6]PETSC ERROR: INSTEAD the line number of the start of the function >> [6]PETSC ERROR: is given. >> [6]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [6]PETSC ERROR: Signal received >> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [6]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017 >> [6]PETSC ERROR: ./dtest on a arch-linux2-c-debug named afrah-pc by afrah >> Fri Sep 15 13:53:42 2017 >> [6]PETSC ERROR: Configure options PETSC_DIR=/home/afrah/PETSc-library2/petsc-3.7.6 >> PETSC_ARCH=arch-linux2-c-debug prefix=/home/afrah/PETSc-library2/petsc-3.7.6/source >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-blacs >> --download-scalapack=1 --download-mumps=1 --with-valgrind=1 >> --with-valgrind-dir="[/usr/local/bin]" >> [6]PETSC ERROR: #1 User provided function() line 0 in unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6 >> >> any idea >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davegwebb10 at gmail.com Mon Sep 18 17:16:15 2017 From: davegwebb10 at gmail.com (David Gross) Date: Tue, 19 Sep 2017 00:16:15 +0200 Subject: [petsc-users] Matrix dot product In-Reply-To: References: Message-ID: Hi Matt, I took a deeper look into the dev documentation and the src/mat code. From what I understand there are multiple places that the code needs to be added or referenced in. I created a copy of AXPY in axpy.c and compiled without issue and could even reference it in my code without compiler error, but found that it wasn't actually doing anything as best I can tell. Am I correct in that one must make implementations for each matrix type in mat/impls? Also I assume that it needs to be added to matipl.h, include/petscmat.h, and finclude/petscmat.h . Is there somewhere that documents the process of adding functions to PETSc classes? I couldn't find anything in the developers manual. Regards, Dave On Thu, Sep 14, 2017 at 10:07 PM, David Gross wrote: > Hi Matt, > Noted for the naming convention. > > Yes, it is a Newton method (see Pan, V. and Reif, J., Fast and Efficient > Parallel Solution of Dense Linear Systems, Computers Math. Applications. > Vol. 17, No. 11, pp. 1481-1491, 1989) > > The dense matrix I have is repeatedly inverted while slowly changing such > that the previous inverse is a near perfect guess for the new inverse. > > Regards, > Dave > > On Thu, Sep 14, 2017 at 2:49 AM, Matthew Knepley > wrote: > >> On Wed, Sep 13, 2017 at 6:14 PM, David Gross >> wrote: >> >>> Hi Matt, >>> Thank you for getting back to me. Your answer confirms what I thought in >>> terms of existing functionality. I think it shouldn't be too hard to make a >>> copy of MatAXPY to MatAXY where it performs Xij = A*Xij*Yij (or without the >>> A). I could then do the MatNorm of the resulting matrix to get what I need. >>> >>> Is a MatAXY function desirable as a source contribution? >>> >> >> Yes. I would prefer calling it MatPointwiseMult, since you can see it as >> VecPointwiseMult on a Vec obtained >> from forgetting the linear operator structure of the matrix (forgetful >> functor). >> >> >>> I am hoping to use PETSc for performing basic vector and matrix >>> operations on dense matrices and 1D vectors. The main uses are matmult, >>> matmatmult and matrix additions and scaling. The application is for >>> implementing a parallel version on an existing Pan-Reif matrix inversion >>> algorithm. >>> >> >> Is this Newton's method on the vector space of matrices? >> >> Thanks, >> >> Matt >> >> >>> The choice of using PETSc is mostly due to us already using it in the >>> same program to solve sparse matrices (with MUMPS) with the goal of >>> avoiding adding yet another package (ex ScaLAPACK/PBLAS) into the code even >>> if other packages may be more directly oriented towards my application. >>> >>> Regards, >>> Dave >>> >>> >>> >>> On Mon, Sep 11, 2017 at 2:19 AM, Matthew Knepley >>> wrote: >>> >>>> On Sun, Sep 10, 2017 at 5:51 PM, David Gross >>>> wrote: >>>> >>>>> Hello, >>>>> I was wondering if there was a matrix equivalent to the vecDot >>>>> function (Frobenius inner product)? As far as I can tell the closest thing >>>>> is MatNorm with NORM_FROBENIUS, but obviously this is acting on only one >>>>> matrix. >>>>> >>>>> If there is not a built in function, what is the best way to compute >>>>> this? I am working fortran90. >>>>> >>>> >>>> We do not have this. However, it would be trivial to add since we have >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpage >>>> s/Mat/MatAXPY.html >>>> >>>> since you just replace + with * in our code. You could argue that we >>>> should have written for >>>> a general ring, but C makes this cumbersome. Do you think you could >>>> make the change? >>>> >>>> What are you using this for? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Regards, >>>>> Dave >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> http://www.caam.rice.edu/~mk51/ >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elbueler at alaska.edu Mon Sep 18 18:00:33 2017 From: elbueler at alaska.edu (Ed Bueler) Date: Mon, 18 Sep 2017 15:00:33 -0800 Subject: [petsc-users] what is the restriction for PCMG, and how to control it? Message-ID: Dear PETSc -- This is most embarassing, but somehow I can't see it in the docs. Regarding defaults for DMDA and PCMG, for the fine-to-coarse restriction, does -pc_type mg use the adjoint of interpolation? (I.e. "full weighting," at least up to a constant.) Or is an injection used? I see that in src/dm/impls/da/dainterp.c, both (Q1,Q0) interpolation and some injection are _implemented_ for DMDA. But I don't see what is used by default, or how to control it at the command line. Asking for options related to my question is a bit of a dead end. For example, in src/snes/examples/tutorials/, ./ex5 -da_refine 3 -pc_type mg -help | grep XX returns nothing at all for XX=injection,restriction,interpolation. Would a new option -pc_mg_restriction_type transpose|injection make sense? Thanks, Ed -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 18 19:31:56 2017 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 18 Sep 2017 20:31:56 -0400 Subject: [petsc-users] Matrix dot product In-Reply-To: References: Message-ID: On Mon, Sep 18, 2017 at 6:16 PM, David Gross wrote: > Hi Matt, > I took a deeper look into the dev documentation and the src/mat code. > All the source is online, so I think that is a good way to talk about it in email. > From what I understand there are multiple places that the code needs to be > added or referenced in. I created a copy of AXPY in axpy.c > Lets look at MatAXPY(): https://bitbucket.org/petsc/petsc/src/a2d7f5eaef7c3b636ab943ac50d4f7c86905df62/src/mat/utils/axpy.c?at=master&fileviewer=file-view-default#axpy.c-22 You just need to make a copy of this function. You want Z = X * Y, right?, (or maybe Y = X * Y) so its MatPointwiseMult(Mat Z, Mat X, Mat Y) After the initial checks for compatibility, it dispatches https://bitbucket.org/petsc/petsc/src/a2d7f5eaef7c3b636ab943ac50d4f7c86905df62/src/mat/utils/axpy.c?at=master&fileviewer=file-view-default#axpy.c-36 Here lets ignore the version optimized for a particular storage format, and just use the generic version https://bitbucket.org/petsc/petsc/src/a2d7f5eaef7c3b636ab943ac50d4f7c86905df62/src/mat/utils/axpy.c?at=master&fileviewer=file-view-default#axpy.c-58 Here we take advantage of the built-in ADD_VALUES, but there is no MULT_VALUES option. Thus, we would need to get the row of X and Y, compare columns (which come out sorted) and then insert the result. However, the alternative is have it error if MatStructure is DIFFERENT_NONZERO_PATTERN, so you can assume that both rows are identical and dispense with the column checking. > and compiled without issue and could even reference it in my code without > compiler error, but found that it wasn't actually doing anything as best I > can tell. Am I correct in that one must make implementations for each > matrix type in mat/impls? Also I assume that it needs to be added to > matipl.h, include/petscmat.h, and finclude/petscmat.h . > > Is there somewhere that documents the process of adding functions to PETSc > classes? I couldn't find anything in the developers manual. > There is a discussion of this in the tutorials, and in here http://www.mcs.anl.gov/petsc/developers/index.html there is a discussion of adding different types of things. THanks, Matt > Regards, > Dave > > On Thu, Sep 14, 2017 at 10:07 PM, David Gross > wrote: > >> Hi Matt, >> Noted for the naming convention. >> >> Yes, it is a Newton method (see Pan, V. and Reif, J., Fast and Efficient >> Parallel Solution of Dense Linear Systems, Computers Math. Applications. >> Vol. 17, No. 11, pp. 1481-1491, 1989) >> >> The dense matrix I have is repeatedly inverted while slowly changing such >> that the previous inverse is a near perfect guess for the new inverse. >> >> Regards, >> Dave >> >> On Thu, Sep 14, 2017 at 2:49 AM, Matthew Knepley >> wrote: >> >>> On Wed, Sep 13, 2017 at 6:14 PM, David Gross >>> wrote: >>> >>>> Hi Matt, >>>> Thank you for getting back to me. Your answer confirms what I thought >>>> in terms of existing functionality. I think it shouldn't be too hard to >>>> make a copy of MatAXPY to MatAXY where it performs Xij = A*Xij*Yij (or >>>> without the A). I could then do the MatNorm of the resulting matrix to get >>>> what I need. >>>> >>>> Is a MatAXY function desirable as a source contribution? >>>> >>> >>> Yes. I would prefer calling it MatPointwiseMult, since you can see it as >>> VecPointwiseMult on a Vec obtained >>> from forgetting the linear operator structure of the matrix (forgetful >>> functor). >>> >>> >>>> I am hoping to use PETSc for performing basic vector and matrix >>>> operations on dense matrices and 1D vectors. The main uses are matmult, >>>> matmatmult and matrix additions and scaling. The application is for >>>> implementing a parallel version on an existing Pan-Reif matrix inversion >>>> algorithm. >>>> >>> >>> Is this Newton's method on the vector space of matrices? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> The choice of using PETSc is mostly due to us already using it in the >>>> same program to solve sparse matrices (with MUMPS) with the goal of >>>> avoiding adding yet another package (ex ScaLAPACK/PBLAS) into the code even >>>> if other packages may be more directly oriented towards my application. >>>> >>>> Regards, >>>> Dave >>>> >>>> >>>> >>>> On Mon, Sep 11, 2017 at 2:19 AM, Matthew Knepley >>>> wrote: >>>> >>>>> On Sun, Sep 10, 2017 at 5:51 PM, David Gross >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> I was wondering if there was a matrix equivalent to the vecDot >>>>>> function (Frobenius inner product)? As far as I can tell the closest thing >>>>>> is MatNorm with NORM_FROBENIUS, but obviously this is acting on only one >>>>>> matrix. >>>>>> >>>>>> If there is not a built in function, what is the best way to compute >>>>>> this? I am working fortran90. >>>>>> >>>>> >>>>> We do not have this. However, it would be trivial to add since we have >>>>> >>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpage >>>>> s/Mat/MatAXPY.html >>>>> >>>>> since you just replace + with * in our code. You could argue that we >>>>> should have written for >>>>> a general ring, but C makes this cumbersome. Do you think you could >>>>> make the change? >>>>> >>>>> What are you using this for? >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Regards, >>>>>> Dave >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> http://www.caam.rice.edu/~mk51/ >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> http://www.caam.rice.edu/~mk51/ >>> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Sep 18 20:17:21 2017 From: jed at jedbrown.org (Jed Brown) Date: Mon, 18 Sep 2017 19:17:21 -0600 Subject: [petsc-users] what is the restriction for PCMG, and how to control it? In-Reply-To: References: Message-ID: <87wp4vv65a.fsf@jedbrown.org> Ed Bueler writes: > Dear PETSc -- > > This is most embarassing, but somehow I can't see it in the docs. > > Regarding defaults for DMDA and PCMG, for the fine-to-coarse restriction, > does -pc_type mg use the adjoint of interpolation? (I.e. "full > weighting," at least up to a constant.) Or is an injection used? It's "full weighting" -- transpose of interpolation [*] which you can select (for DMDA) using DMDASetInterpolationType. Injection is terrible for PCMG because it has order zero -- high frequency harmonics are aliased at full amplitude onto the coarse grid (see Brandt's MG guide, ?4.3). Injection can (and should, when relevant) be used for *state* restriction in FAS because it is more accurate for low-frequency components ("infinite secondary order") and aliasing is irrelevant in that context. So there isn't an "automatic" way to use injection in PCMG, though I suppose it could be added for educational purposes. [*] The scaling for restriction is managed by the "rscale" vector returned by DMCreateInterpolation. > I see that in src/dm/impls/da/dainterp.c, both (Q1,Q0) interpolation and > some injection are _implemented_ for DMDA. But I don't see what is used by > default, or how to control it at the command line. Asking for options > related to my question is a bit of a dead end. For example, in > src/snes/examples/tutorials/, > ./ex5 -da_refine 3 -pc_type mg -help | grep XX > returns nothing at all for XX=injection,restriction,interpolation. > > Would a new option > -pc_mg_restriction_type transpose|injection > make sense? > > Thanks, > > Ed > > > -- > Ed Bueler > Dept of Math and Stat and Geophysical Institute > University of Alaska Fairbanks > Fairbanks, AK 99775-6660 > 301C Chapman -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From t.appel17 at imperial.ac.uk Tue Sep 19 11:02:41 2017 From: t.appel17 at imperial.ac.uk (Appel, Thibaut) Date: Tue, 19 Sep 2017 16:02:41 +0000 Subject: [petsc-users] SLEPc - control of MUMPS in Fortran Message-ID: Good afternoon, I am using SLEPc in shift-and-invert mode to solve linear generalized EVPs. I am also using MUMPS for solving the linear systems that stem from the problem. After browsing a lot of pages, I am quite surprised by the lack of documentation available regarding the MUMPS interface with SLEPc/PETSc. - I would like, in my code, to set and print MUMPS control and output parameters (CNTL,ICNTL,INFO) to monitor the process but it seems like a tedious task in Fortran. I found this discussion https://lists.mcs.anl.gov/pipermail/petsc-users/2012-May/013437.html that suggests hard-coded statements in c++ to set MUMPS parameters. Is there a clean solution in Fortran? - When using MUMPS together with SLEPc, can you perform iterative refinement? I tried to set ICNTL(10) to do so but nothing happens. Can I trust SLEPc for automatically setting MUMPS parameters that will ensure best scalability? If you have further advice for using optimally SLEPc together with MUMPS I would really appreciate it and enjoy discussing it with you. Thank you in advance, Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Tue Sep 19 11:23:24 2017 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 19 Sep 2017 18:23:24 +0200 Subject: [petsc-users] SLEPc - control of MUMPS in Fortran In-Reply-To: References: Message-ID: > El 19 sept 2017, a las 18:02, Appel, Thibaut escribi?: > > Good afternoon, > > I am using SLEPc in shift-and-invert mode to solve linear generalized EVPs. I am also using MUMPS for solving the linear systems that stem from the problem. After browsing a lot of pages, I am quite surprised by the lack of documentation available regarding the MUMPS interface with SLEPc/PETSc. > > - I would like, in my code, to set and print MUMPS control and output parameters (CNTL,ICNTL,INFO) to monitor the process but it seems like a tedious task in Fortran. I found this discussion https://lists.mcs.anl.gov/pipermail/petsc-users/2012-May/013437.html that suggests hard-coded statements in c++ to set MUMPS parameters. Is there a clean solution in Fortran? An easy way to set MUMPS options is by inserting options in the PETSc options database with PetscOptionsInsertString(), see ex25.c http://slepc.upv.es/documentation/current/src/eps/examples/tutorials/ex25.c.html PetscOptionsInsertString() should work also in Fortran. > > - When using MUMPS together with SLEPc, can you perform iterative refinement? I tried to set ICNTL(10) to do so but nothing happens. Can I trust SLEPc for automatically setting MUMPS parameters that will ensure best scalability? If you have further advice for using optimally SLEPc together with MUMPS I would really appreciate it and enjoy discussing it with you. With -eps_view you can check the actual options (including MUMPS options) that are being used. You can activate iterative refinement if you want, but I don't think it is necessary for shift-and-invert computations. Regarding scalability, you can try changing between sequential and parallel symbolic factorization, but I think the default is usually the best setting. SLEPc does not set any MUMPS options other than the default. Jose > > Thank you in advance, > > > Thibaut From bsmith at mcs.anl.gov Tue Sep 19 17:34:29 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 19 Sep 2017 17:34:29 -0500 Subject: [petsc-users] SLEPc - control of MUMPS in Fortran In-Reply-To: References: Message-ID: Please send a Fortran code that demonstrates the difficulty. In theory one can do this in Fortran as simply as C but since it is far less used there could be problems. Barry > On Sep 19, 2017, at 11:02 AM, Appel, Thibaut wrote: > > Good afternoon, > > I am using SLEPc in shift-and-invert mode to solve linear generalized EVPs. I am also using MUMPS for solving the linear systems that stem from the problem. After browsing a lot of pages, I am quite surprised by the lack of documentation available regarding the MUMPS interface with SLEPc/PETSc. > > - I would like, in my code, to set and print MUMPS control and output parameters (CNTL,ICNTL,INFO) to monitor the process but it seems like a tedious task in Fortran. I found this discussion https://lists.mcs.anl.gov/pipermail/petsc-users/2012-May/013437.html that suggests hard-coded statements in c++ to set MUMPS parameters. Is there a clean solution in Fortran? > > - When using MUMPS together with SLEPc, can you perform iterative refinement? I tried to set ICNTL(10) to do so but nothing happens. Can I trust SLEPc for automatically setting MUMPS parameters that will ensure best scalability? If you have further advice for using optimally SLEPc together with MUMPS I would really appreciate it and enjoy discussing it with you. > > Thank you in advance, > > > Thibaut From imilian.hartig at gmail.com Wed Sep 20 10:46:57 2017 From: imilian.hartig at gmail.com (Maximilian Hartig) Date: Wed, 20 Sep 2017 17:46:57 +0200 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? Message-ID: Hello, I?m trying to implement plasticity using petscFE but I am quite stuck since a while. Here?s what I?m trying to do: I have a TS which solves the following equation: gradient(stress) +Forces = density*acceleration where at the moment stress is a linear function of the strain and hence the gradient of the displacement. This works fine. Now I want to compare the stress to a reference value and if it lies above this yield stress, I have to reevaluate the stress at the respective location. Then I need to update the plastic strain / yield stress at this location. I tried doing that first by solving three fields at the same time: displacements, stresses and yield stress. This failed. Then, I tried solving only for displacement increments, storing the displacements, stresses and yield stress from the past time step in an auxiliary field. The auxiliary fields are updated after each time step with a second SNES, using the displacement increments from the current, converged time step. This also failed. In both cases the code had problems converging and when it did, I ended up with negative plastic strain. This is not possible and I don?t know how it happens because I explicitly only increment the plastic strain when the increment is positive. I am sure there is an easy solution to how I can update the internal variables and determine the correct stress for the residual but I just cannot figure it out. I?d be thankful for any hints. Thanks, Max From knepley at gmail.com Wed Sep 20 11:17:47 2017 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 20 Sep 2017 12:17:47 -0400 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: Message-ID: On Wed, Sep 20, 2017 at 11:46 AM, Maximilian Hartig < imilian.hartig at gmail.com> wrote: > Hello, > > I?m trying to implement plasticity using petscFE but I am quite stuck > since a while. Here?s what I?m trying to do: > > I have a TS which solves the following equation: > gradient(stress) +Forces = density*acceleration > where at the moment stress is a linear function of the strain and hence > the gradient of the displacement. This works fine. Now I want to compare > the stress to a reference value and if it lies above this yield stress, I > have to reevaluate the stress at the respective location. Then I need to > update the plastic strain / yield stress at this location. > I tried doing that first by solving three fields at the same time: > displacements, stresses and yield stress. This failed. > Then, I tried solving only for displacement increments, storing the > displacements, stresses and yield stress from the past time step in an > auxiliary field. The auxiliary fields are updated after each time step with > a second SNES, using the displacement increments from the current, > converged time step. This also failed. > In both cases the code had problems converging and when it did, I ended up > with negative plastic strain. This is not possible and I don?t know how it > happens because I explicitly only increment the plastic strain when the > increment is positive. > > I am sure there is an easy solution to how I can update the internal > variables and determine the correct stress for the residual but I just > cannot figure it out. I?d be thankful for any hints. > It looks like there are two problems above: 1) Convergence For any convergence question, we at minimum need to see the output of -snes_view -snes_converged_reason -snes_monitor -ksp_monitor_true_residual -snes_linesearch_monitor However, this does not seem to be the main issue. 2) Negative plastic strain If the system really converged (I cannot tell without other information), then the system formulation is wrong. Of course, its really easy to check by just plugging your solution into the residual function too. I do not understand your explanation above completely however. Do you solve for the plastic strain or the increment? Thanks, Matt > Thanks, > Max -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From imilian.hartig at gmail.com Wed Sep 20 11:57:33 2017 From: imilian.hartig at gmail.com (Maximilian Hartig) Date: Wed, 20 Sep 2017 18:57:33 +0200 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: Message-ID: > On 20. Sep 2017, at 18:17, Matthew Knepley wrote: > > On Wed, Sep 20, 2017 at 11:46 AM, Maximilian Hartig > wrote: > Hello, > > I?m trying to implement plasticity using petscFE but I am quite stuck since a while. Here?s what I?m trying to do: > > I have a TS which solves the following equation: > gradient(stress) +Forces = density*acceleration > where at the moment stress is a linear function of the strain and hence the gradient of the displacement. This works fine. Now I want to compare the stress to a reference value and if it lies above this yield stress, I have to reevaluate the stress at the respective location. Then I need to update the plastic strain / yield stress at this location. > I tried doing that first by solving three fields at the same time: displacements, stresses and yield stress. This failed. > Then, I tried solving only for displacement increments, storing the displacements, stresses and yield stress from the past time step in an auxiliary field. The auxiliary fields are updated after each time step with a second SNES, using the displacement increments from the current, converged time step. This also failed. > In both cases the code had problems converging and when it did, I ended up with negative plastic strain. This is not possible and I don?t know how it happens because I explicitly only increment the plastic strain when the increment is positive. > > I am sure there is an easy solution to how I can update the internal variables and determine the correct stress for the residual but I just cannot figure it out. I?d be thankful for any hints. > > It looks like there are two problems above: > > 1) Convergence > > For any convergence question, we at minimum need to see the output of > > -snes_view -snes_converged_reason -snes_monitor -ksp_monitor_true_residual -snes_linesearch_monitor > > However, this does not seem to be the main issue. > > 2) Negative plastic strain This is what I?m mainly concerned with. > > If the system really converged (I cannot tell without other information), then the system formulation is wrong. Of course, its > really easy to check by just plugging your solution into the residual function too. I do not understand your explanation above > completely however. Do you solve for the plastic strain or the increment? I am trying to find a formulation that works and I think there is a core concept I am just not ?getting?. I want to solve for the displacements. This works fine in an elastic case. When plasticity is involved, I need to determine the actual stress for my residual evaluation and I have not found a way to do that. All formulations for stress I found in literature use strain increments so I tried to just solve for increments each timestep and then add them together in tspoststep. But I still need to somehow evaluate the stress for my displacement increment residuals. So currently, I have auxiliary fields with the stress and the plastic strain. I evaluate the current trial stress by adding a stress increment assuming elastic behaviour. If the trial stress lies beyond the yield stress I calculate the corrected stress to evaluate my residual for the displacements. But now I somehow need to update my plastic strain and the stress in the auxiliary fields. So in tspoststep I created another SNES to now calculate the stress and plastic strain while the displacement is the auxiliary field. I?m sure there?s an elegant solution on how to update internal variables but I have not found it. Thanks, Max > > Thanks, > > Matt > > Thanks, > Max > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 20 12:05:27 2017 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 20 Sep 2017 13:05:27 -0400 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: Message-ID: On Wed, Sep 20, 2017 at 12:57 PM, Maximilian Hartig < imilian.hartig at gmail.com> wrote: > On 20. Sep 2017, at 18:17, Matthew Knepley wrote: > > On Wed, Sep 20, 2017 at 11:46 AM, Maximilian Hartig com> wrote: > >> Hello, >> >> I?m trying to implement plasticity using petscFE but I am quite stuck >> since a while. Here?s what I?m trying to do: >> >> I have a TS which solves the following equation: >> gradient(stress) +Forces = density*acceleration >> where at the moment stress is a linear function of the strain and hence >> the gradient of the displacement. This works fine. Now I want to compare >> the stress to a reference value and if it lies above this yield stress, I >> have to reevaluate the stress at the respective location. Then I need to >> update the plastic strain / yield stress at this location. >> I tried doing that first by solving three fields at the same time: >> displacements, stresses and yield stress. This failed. >> Then, I tried solving only for displacement increments, storing the >> displacements, stresses and yield stress from the past time step in an >> auxiliary field. The auxiliary fields are updated after each time step with >> a second SNES, using the displacement increments from the current, >> converged time step. This also failed. >> In both cases the code had problems converging and when it did, I ended >> up with negative plastic strain. This is not possible and I don?t know how >> it happens because I explicitly only increment the plastic strain when the >> increment is positive. >> >> I am sure there is an easy solution to how I can update the internal >> variables and determine the correct stress for the residual but I just >> cannot figure it out. I?d be thankful for any hints. >> > > It looks like there are two problems above: > > 1) Convergence > > For any convergence question, we at minimum need to see the output of > > -snes_view -snes_converged_reason -snes_monitor > -ksp_monitor_true_residual -snes_linesearch_monitor > > However, this does not seem to be the main issue. > > 2) Negative plastic strain > > > This is what I?m mainly concerned with. > > > If the system really converged (I cannot tell without other information), > then the system formulation is wrong. Of course, its > really easy to check by just plugging your solution into the residual > function too. I do not understand your explanation above > completely however. Do you solve for the plastic strain or the increment? > > > I am trying to find a formulation that works and I think there is a core > concept I am just not ?getting?. > I want to solve for the displacements. > This works fine in an elastic case. When plasticity is involved, I need to > determine the actual stress for my residual evaluation and I have not found > a way to do that. > All formulations for stress I found in literature use strain increments so > I tried to just solve for increments each timestep and then add them > together in tspoststep. But I still need to somehow evaluate the stress for > my displacement increment residuals. So currently, I have auxiliary fields > with the stress and the plastic strain. > First question: Don't you get stress by just applying a local operator, rather than a solve? Thanks, Matt > I evaluate the current trial stress by adding a stress increment assuming > elastic behaviour. If the trial stress lies beyond the yield stress I > calculate the corrected stress to evaluate my residual for the > displacements. But now I somehow need to update my plastic strain and the > stress in the auxiliary fields. So in tspoststep I created another SNES to > now calculate the stress and plastic strain while the displacement is the > auxiliary field. > > I?m sure there?s an elegant solution on how to update internal variables > but I have not found it. > > Thanks, > Max > > > Thanks, > > Matt > > >> Thanks, >> Max > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Wed Sep 20 12:37:55 2017 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Wed, 20 Sep 2017 13:37:55 -0400 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: Message-ID: <143111bc-4a58-e6de-1cdd-f5708826fbdf@berkeley.edu> The standard methodology for this problem is to solve only for the displacements (globally).? The stresses are recomputed at the Gauss point level.? The needed history that is usually kept at the Gauss point level is the plastic strain.? Convergence is strongly tied to having a good tangent operator; in this case you need the so-called consistent tangent operator (see Simo and Taylor, Computer Methods in Applied Mechanics and Engineering, 1985). Have a look also at the comprehensive and tutorial books by Simo and Hughes (Computational Inelasticity) and the 1st and 2nd Volumes of Zienkeiwicz and Taylor (The Finite Element Method) now in the 7th edition.? These texts provide virtually all of the implementation details that you need. -sg On 9/20/17 1:05 PM, Matthew Knepley wrote: > On Wed, Sep 20, 2017 at 12:57 PM, Maximilian Hartig > > wrote: > >> On 20. Sep 2017, at 18:17, Matthew Knepley > > wrote: >> >> On Wed, Sep 20, 2017 at 11:46 AM, Maximilian >> Hartig> >wrote: >> >> Hello, >> >> I?m trying to implement plasticity using petscFE but I am >> quite stuck since a while. Here?s what I?m trying to do: >> >> I have a TS which solves the following equation: >> gradient(stress) +Forces = density*acceleration >> where at the moment stress is a linear function of the strain >> and hence the gradient of the displacement. This works fine. >> Now I want to compare the stress to a reference value and if >> it lies above this yield stress, I have to reevaluate the >> stress at the respective location. Then I need to update the >> plastic strain / yield stress? at this location. >> I tried doing that first by solving three fields at the same >> time: displacements, stresses and yield stress. This failed. >> Then, I tried solving only for displacement increments, >> storing the displacements, stresses and yield stress from the >> past time step in an auxiliary field. The auxiliary fields >> are updated after each time step with a second SNES, using >> the displacement increments from the current, converged time >> step. This also failed. >> In both cases the code had problems converging and when it >> did, I ended up with negative plastic strain. This is not >> possible and I don?t know how it happens because I explicitly >> only increment the plastic strain when the increment is positive. >> >> I am sure there is an easy solution to how I can update the >> internal variables and determine the correct stress for the >> residual but I just cannot figure it out. I?d be thankful for >> any hints. >> >> >> It looks like there are two problems above: >> >> 1) Convergence >> >> For any convergence question, we at minimum need to see the output of >> >> ? -snes_view -snes_converged_reason -snes_monitor >> -ksp_monitor_true_residual -snes_linesearch_monitor >> >> However, this does not seem to be the main issue. >> >> 2) Negative plastic strain > > This is what I?m mainly concerned with. >> >> If the system really converged (I cannot tell without other >> information), then the system formulation is wrong. Of course, its >> really easy to check by just plugging your solution into the >> residual function too. I do not understand your explanation above >> completely however. Do you solve for the plastic strain or the >> increment? > > I am trying to find a formulation that works and I think there is > a core concept I am just not ?getting?. > I want to solve for the displacements. > This works fine in an elastic case. When plasticity is involved, I > need to determine the actual stress for my residual evaluation and > I have not found a way to do that. > All formulations for stress I found in literature use strain > increments so I tried to just solve for increments each timestep > and then add them together in tspoststep. But I still need to > somehow evaluate the stress for my displacement increment > residuals. So currently, I have auxiliary fields with the stress > and the plastic strain. > > > First question: Don't you get stress by just applying a local > operator, rather than a solve? > > ? Thanks, > > ? ? ?Matt > > I evaluate the current trial stress by adding a stress increment > assuming elastic behaviour. If the trial stress lies beyond the > yield stress I calculate the corrected stress to evaluate my > residual for the displacements. But now I somehow need to update > my plastic strain and the stress in the auxiliary fields. So in > tspoststep I created another SNES to now calculate the stress and > plastic strain while the displacement is the auxiliary field. > > I?m sure there?s an elegant solution on how to update internal > variables but I have not found it. > > Thanks, > Max >> >> Thanks, >> >> ? ? ?Matt >> >> Thanks, >> Max >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ -- ------------------------------------------------------------------- Sanjay Govindjee, PhD, PE Horace, Dorothy, and Katherine Johnson Professor in Engineering 779 Davis Hall University of California Berkeley, CA 94720-1710 Voice: +1 510 642 6060 FAX: +1 510 643 5264 s_g at berkeley.edu http://faculty.ce.berkeley.edu/sanjay ------------------------------------------------------------------- Books: Engineering Mechanics of Deformable Solids: A Presentation with Exercises http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641 http://ukcatalogue.oup.com/product/9780199651641.do http://amzn.com/0199651647 Engineering Mechanics 3 (Dynamics) 2nd Edition http://www.springer.com/978-3-642-53711-0 http://amzn.com/3642537111 Engineering Mechanics 3, Supplementary Problems: Dynamics http://www.amzn.com/B00SOXN8JU ----------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From imilian.hartig at gmail.com Wed Sep 20 14:46:41 2017 From: imilian.hartig at gmail.com (Maximilian Hartig) Date: Wed, 20 Sep 2017 21:46:41 +0200 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: Message-ID: > On 20. Sep 2017, at 19:05, Matthew Knepley wrote: > > On Wed, Sep 20, 2017 at 12:57 PM, Maximilian Hartig > wrote: >> On 20. Sep 2017, at 18:17, Matthew Knepley > wrote: >> >> On Wed, Sep 20, 2017 at 11:46 AM, Maximilian Hartig > wrote: >> Hello, >> >> I?m trying to implement plasticity using petscFE but I am quite stuck since a while. Here?s what I?m trying to do: >> >> I have a TS which solves the following equation: >> gradient(stress) +Forces = density*acceleration >> where at the moment stress is a linear function of the strain and hence the gradient of the displacement. This works fine. Now I want to compare the stress to a reference value and if it lies above this yield stress, I have to reevaluate the stress at the respective location. Then I need to update the plastic strain / yield stress at this location. >> I tried doing that first by solving three fields at the same time: displacements, stresses and yield stress. This failed. >> Then, I tried solving only for displacement increments, storing the displacements, stresses and yield stress from the past time step in an auxiliary field. The auxiliary fields are updated after each time step with a second SNES, using the displacement increments from the current, converged time step. This also failed. >> In both cases the code had problems converging and when it did, I ended up with negative plastic strain. This is not possible and I don?t know how it happens because I explicitly only increment the plastic strain when the increment is positive. >> >> I am sure there is an easy solution to how I can update the internal variables and determine the correct stress for the residual but I just cannot figure it out. I?d be thankful for any hints. >> >> It looks like there are two problems above: >> >> 1) Convergence >> >> For any convergence question, we at minimum need to see the output of >> >> -snes_view -snes_converged_reason -snes_monitor -ksp_monitor_true_residual -snes_linesearch_monitor >> >> However, this does not seem to be the main issue. >> >> 2) Negative plastic strain > > This is what I?m mainly concerned with. >> >> If the system really converged (I cannot tell without other information), then the system formulation is wrong. Of course, its >> really easy to check by just plugging your solution into the residual function too. I do not understand your explanation above >> completely however. Do you solve for the plastic strain or the increment? > > I am trying to find a formulation that works and I think there is a core concept I am just not ?getting?. > I want to solve for the displacements. > This works fine in an elastic case. When plasticity is involved, I need to determine the actual stress for my residual evaluation and I have not found a way to do that. > All formulations for stress I found in literature use strain increments so I tried to just solve for increments each timestep and then add them together in tspoststep. But I still need to somehow evaluate the stress for my displacement increment residuals. So currently, I have auxiliary fields with the stress and the plastic strain. > > First question: Don't you get stress by just applying a local operator, rather than a solve? That depends on the type of plasticity. For a linear hardening formulation it is correct that I could just apply a local operator. I?d be happy with that for now. But I?d still need to save stress state and plastic strain to determine whether or not I?m dealing with a plasticity. I don?t know how to do that inside the residual evaluation. Plus DMProjectField seems to have problems evaluating the gradient when boundary conditions are imposed. Thanks, Max > > Thanks, > > Matt > > I evaluate the current trial stress by adding a stress increment assuming elastic behaviour. If the trial stress lies beyond the yield stress I calculate the corrected stress to evaluate my residual for the displacements. But now I somehow need to update my plastic strain and the stress in the auxiliary fields. So in tspoststep I created another SNES to now calculate the stress and plastic strain while the displacement is the auxiliary field. > > I?m sure there?s an elegant solution on how to update internal variables but I have not found it. > > Thanks, > Max >> >> Thanks, >> >> Matt >> >> Thanks, >> Max >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 20 15:51:00 2017 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 20 Sep 2017 16:51:00 -0400 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: Message-ID: On Wed, Sep 20, 2017 at 3:46 PM, Maximilian Hartig wrote: > > On 20. Sep 2017, at 19:05, Matthew Knepley wrote: > > On Wed, Sep 20, 2017 at 12:57 PM, Maximilian Hartig com> wrote: > >> On 20. Sep 2017, at 18:17, Matthew Knepley wrote: >> >> On Wed, Sep 20, 2017 at 11:46 AM, Maximilian Hartig < >> imilian.hartig at gmail.com> wrote: >> >>> Hello, >>> >>> I?m trying to implement plasticity using petscFE but I am quite stuck >>> since a while. Here?s what I?m trying to do: >>> >>> I have a TS which solves the following equation: >>> gradient(stress) +Forces = density*acceleration >>> where at the moment stress is a linear function of the strain and hence >>> the gradient of the displacement. This works fine. Now I want to compare >>> the stress to a reference value and if it lies above this yield stress, I >>> have to reevaluate the stress at the respective location. Then I need to >>> update the plastic strain / yield stress at this location. >>> I tried doing that first by solving three fields at the same time: >>> displacements, stresses and yield stress. This failed. >>> Then, I tried solving only for displacement increments, storing the >>> displacements, stresses and yield stress from the past time step in an >>> auxiliary field. The auxiliary fields are updated after each time step with >>> a second SNES, using the displacement increments from the current, >>> converged time step. This also failed. >>> In both cases the code had problems converging and when it did, I ended >>> up with negative plastic strain. This is not possible and I don?t know how >>> it happens because I explicitly only increment the plastic strain when the >>> increment is positive. >>> >>> I am sure there is an easy solution to how I can update the internal >>> variables and determine the correct stress for the residual but I just >>> cannot figure it out. I?d be thankful for any hints. >>> >> >> It looks like there are two problems above: >> >> 1) Convergence >> >> For any convergence question, we at minimum need to see the output of >> >> -snes_view -snes_converged_reason -snes_monitor >> -ksp_monitor_true_residual -snes_linesearch_monitor >> >> However, this does not seem to be the main issue. >> >> 2) Negative plastic strain >> >> >> This is what I?m mainly concerned with. >> >> >> If the system really converged (I cannot tell without other information), >> then the system formulation is wrong. Of course, its >> really easy to check by just plugging your solution into the residual >> function too. I do not understand your explanation above >> completely however. Do you solve for the plastic strain or the increment? >> >> >> I am trying to find a formulation that works and I think there is a core >> concept I am just not ?getting?. >> I want to solve for the displacements. >> This works fine in an elastic case. When plasticity is involved, I need >> to determine the actual stress for my residual evaluation and I have not >> found a way to do that. >> All formulations for stress I found in literature use strain increments >> so I tried to just solve for increments each timestep and then add them >> together in tspoststep. But I still need to somehow evaluate the stress for >> my displacement increment residuals. So currently, I have auxiliary fields >> with the stress and the plastic strain. >> > > First question: Don't you get stress by just applying a local operator, > rather than a solve? > > That depends on the type of plasticity. > What type of plasticity is not local? > For a linear hardening formulation it is correct that I could just apply a > local operator. I?d be happy with that for now. But I?d still need to save > stress state and plastic strain to determine whether or not I?m dealing > with a plasticity. I don?t know how to do that inside the residual > evaluation. > I do not know what you mean by this, meaning why you can't just save these as auxiliary fields. Also, it would seem to be enough to have the old displacement and the plastic strain. > Plus DMProjectField seems to have problems evaluating the gradient when > boundary conditions are imposed. > There are several examples where we do exactly this. Can you show me what you mean by this? Thanks, Matt > Thanks, > Max > > > Thanks, > > Matt > > >> I evaluate the current trial stress by adding a stress increment assuming >> elastic behaviour. If the trial stress lies beyond the yield stress I >> calculate the corrected stress to evaluate my residual for the >> displacements. But now I somehow need to update my plastic strain and the >> stress in the auxiliary fields. So in tspoststep I created another SNES to >> now calculate the stress and plastic strain while the displacement is the >> auxiliary field. >> >> I?m sure there?s an elegant solution on how to update internal variables >> but I have not found it. >> >> Thanks, >> Max >> >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Max >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Wed Sep 20 17:19:34 2017 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Wed, 20 Sep 2017 18:19:34 -0400 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: Message-ID: Matt, ? There is such a thing as non-local plasticity, one where there is a separate field equation to solve for the plasticity (beyond the balance of momentum). -sanjay On 9/20/17 4:51 PM, Matthew Knepley wrote: > On Wed, Sep 20, 2017 at 3:46 PM, Maximilian Hartig > > wrote: > > >> On 20. Sep 2017, at 19:05, Matthew Knepley > > wrote: >> >> On Wed, Sep 20, 2017 at 12:57 PM, Maximilian >> Hartig> >wrote: >> >>> On 20. Sep 2017, at 18:17, Matthew Knepley >>> > wrote: >>> >>> On Wed, Sep 20, 2017 at 11:46 AM, Maximilian >>> Hartig>> >wrote: >>> >>> Hello, >>> >>> I?m trying to implement plasticity using petscFE but I >>> am quite stuck since a while. Here?s what I?m trying to do: >>> >>> I have a TS which solves the following equation: >>> gradient(stress) +Forces = density*acceleration >>> where at the moment stress is a linear function of the >>> strain and hence the gradient of the displacement. This >>> works fine. Now I want to compare the stress to a >>> reference value and if it lies above this yield stress, >>> I have to reevaluate the stress at the respective >>> location. Then I need to update the plastic strain / >>> yield stress? at this location. >>> I tried doing that first by solving three fields at the >>> same time: displacements, stresses and yield stress. >>> This failed. >>> Then, I tried solving only for displacement increments, >>> storing the displacements, stresses and yield stress >>> from the past time step in an auxiliary field. The >>> auxiliary fields are updated after each time step with a >>> second SNES, using the displacement increments from the >>> current, converged time step. This also failed. >>> In both cases the code had problems converging and when >>> it did, I ended up with negative plastic strain. This is >>> not possible and I don?t know how it happens because I >>> explicitly only increment the plastic strain when the >>> increment is positive. >>> >>> I am sure there is an easy solution to how I can update >>> the internal variables and determine the correct stress >>> for the residual but I just cannot figure it out. I?d be >>> thankful for any hints. >>> >>> >>> It looks like there are two problems above: >>> >>> 1) Convergence >>> >>> For any convergence question, we at minimum need to see the >>> output of >>> >>> -snes_view -snes_converged_reason -snes_monitor >>> -ksp_monitor_true_residual -snes_linesearch_monitor >>> >>> However, this does not seem to be the main issue. >>> >>> 2) Negative plastic strain >> >> This is what I?m mainly concerned with. >>> >>> If the system really converged (I cannot tell without other >>> information), then the system formulation is wrong. Of >>> course, its >>> really easy to check by just plugging your solution into the >>> residual function too. I do not understand your explanation >>> above >>> completely however. Do you solve for the plastic strain or >>> the increment? >> >> I am trying to find a formulation that works and I think >> there is a core concept I am just not ?getting?. >> I want to solve for the displacements. >> This works fine in an elastic case. When plasticity is >> involved, I need to determine the actual stress for my >> residual evaluation and I have not found a way to do that. >> All formulations for stress I found in literature use strain >> increments so I tried to just solve for increments each >> timestep and then add them together in tspoststep. But I >> still need to somehow evaluate the stress for my displacement >> increment residuals. So currently, I have auxiliary fields >> with the stress and the plastic strain. >> >> >> First question: Don't you get stress by just applying a local >> operator, rather than a solve? > That depends on the type of plasticity. > > > What type of plasticity is not local? > > For a linear hardening formulation it is correct that I could just > apply a local operator. I?d be happy with that for now. But I?d > still need to save stress state and plastic strain to determine > whether or not I?m dealing with a plasticity. I don?t know how to > do that inside the residual evaluation. > > > I do not know what you mean by this, meaning why you can't just save > these as auxiliary fields. Also, it would seem to be enough to have > the old displacement and the plastic strain. > > Plus DMProjectField seems to have problems evaluating the gradient > when boundary conditions are imposed. > > > There are several examples where we do exactly this. Can you show me > what you mean by this? > > ? Thanks, > > ? ? Matt > > Thanks, > Max >> >> ? Thanks, >> >> ? ? ?Matt >> >> I evaluate the current trial stress by adding a stress >> increment assuming elastic behaviour. If the trial stress >> lies beyond the yield stress I calculate the corrected stress >> to evaluate my residual for the displacements. But now I >> somehow need to update my plastic strain and the stress in >> the auxiliary fields. So in tspoststep I created another SNES >> to now calculate the stress and plastic strain while the >> displacement is the auxiliary field. >> >> I?m sure there?s an elegant solution on how to update >> internal variables but I have not found it. >> >> Thanks, >> Max >>> >>> Thanks, >>> >>> ? ? ?Matt >>> >>> Thanks, >>> Max >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >>> >>> http://www.caam.rice.edu/~mk51/ >>> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ -- ------------------------------------------------------------------- Sanjay Govindjee, PhD, PE Horace, Dorothy, and Katherine Johnson Professor in Engineering 779 Davis Hall University of California Berkeley, CA 94720-1710 Voice: +1 510 642 6060 FAX: +1 510 643 5264 s_g at berkeley.edu http://faculty.ce.berkeley.edu/sanjay ------------------------------------------------------------------- Books: Engineering Mechanics of Deformable Solids: A Presentation with Exercises http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641 http://ukcatalogue.oup.com/product/9780199651641.do http://amzn.com/0199651647 Engineering Mechanics 3 (Dynamics) 2nd Edition http://www.springer.com/978-3-642-53711-0 http://amzn.com/3642537111 Engineering Mechanics 3, Supplementary Problems: Dynamics http://www.amzn.com/B00SOXN8JU ----------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From federico.golfre at gmail.com Thu Sep 21 07:07:25 2017 From: federico.golfre at gmail.com (=?UTF-8?Q?Federico_Golfr=C3=A8_Andreasi?=) Date: Thu, 21 Sep 2017 14:07:25 +0200 Subject: [petsc-users] mg pre-conditioner default setup from PETSc-3.4 to PETSc-3.7 In-Reply-To: <1A3CDFAF-5A93-4AA0-9222-DF43359C547B@mcs.anl.gov> References: <3E0D16E2-BA8F-4792-BBC3-8C55611292CC@mcs.anl.gov> <6BDE8DAF-F70B-42D7-A92B-8CD682BAC928@mcs.anl.gov> <1A3CDFAF-5A93-4AA0-9222-DF43359C547B@mcs.anl.gov> Message-ID: Barry, I solved the issue, it was related to a wrong change I made in the creation of the IS for the VecScatter (used in the shell matrix). Fixing that, I reached the same performance. Thank your support. Mark, Hong, Thank you for your replies; I am also evaluating your suggestion on the optimal parametrization for the MG pre-conditioner. Best regards, Federico On 15 September 2017 at 17:45, Barry Smith wrote: > > $ python > Python 2.7.13 (default, Dec 18 2016, 07:03:39) > [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > MatMult >>> 3.6272e+03 - 2.0894e+03 > 1537.7999999999997 > KSP Solve >>> 3.6329e+03 - 2.0949e+03 > 1538.0 > >>> > > You are right, all the extra time is within the MatMult() so for some > reason your shell mat mult is much slower. I cannot guess why unless I can > see inside your shell matmult at what it is doing. > > > Make sure your configure options are identical and using same compiler. > > Barry > > > > > > > On Sep 15, 2017, at 5:08 AM, Federico Golfr? Andreasi < > federico.golfre at gmail.com> wrote: > > > > Hi Barry, > > > > I have attached an extract of the our program output for both the > versions: PETSc-3.4.4 and PETSc-3.7.3. > > > > In this program the KSP has a shell matrix as operator and a MPIAIJ > matrix as pre-conditioner. > > I was wondering if the slowing is related more on the operations done in > the MatMult of the shell matrix; > > because on a test program where I solve a similar system without shell > matrix I do not see the performance degradation. > > > > Perhaps you could give me some hints, > > Thank you and best regards, > > Federico > > > > > > > > > > On 13 September 2017 at 17:58, Barry Smith wrote: > > > > > On Sep 13, 2017, at 10:56 AM, Federico Golfr? Andreasi < > federico.golfre at gmail.com> wrote: > > > > > > Hi Barry, > > > > > > I understand and perfectly agree with you that the behavior increase > after the release due to better tuning. > > > > > > In my case, the difference in the solution is negligible, but the > runtime increases up to +70% (with the same number of ksp_iterations). > > > > Ok this is an important (and bad) difference. > > > > > So I was wondering if maybe there were just some flags related to > memory preallocation or re-usage of intermediate solution that before was > defaulted. > > > > Note likely it is this. > > > > Are both compiled with the same level of compiler optimization? > > > > Please run both with -log_summary and send the output, this will tell > us WHAT parts are now slower. > > > > Barry > > > > > > > > Thank you, > > > Federico > > > > > > > > > > > > On 13 September 2017 at 17:29, Barry Smith wrote: > > > > > > There will likely always be slight differences in convergence over > that many releases. Lots of little defaults etc get changed over time as we > learn from users and increase the robustness of the defaults. > > > > > > So in your case do the differences matter? > > > > > > 1) What is the time to solution in both cases, is it a few percent > different or now much slower? > > > > > > 2) What about number of iterations? Almost identical (say 1 or 2 > different) or does it now take 30 iterations when it use to take 5? > > > > > > Barry > > > > > > > On Sep 13, 2017, at 10:25 AM, Federico Golfr? Andreasi < > federico.golfre at gmail.com> wrote: > > > > > > > > Dear PETSc users/developers, > > > > > > > > I recently switched from PETSc-3.4 to PETSc-3.7 and found that some > default setup for the "mg" (mutigrid) preconditioner have changed. > > > > > > > > We were solving a linear system passing, throug command line, the > following options: > > > > -ksp_type fgmres > > > > -ksp_max_it 100000 > > > > -ksp_rtol 0.000001 > > > > -pc_type mg > > > > -ksp_view > > > > > > > > The output of the KSP view is as follow: > > > > > > > > KSP Object: 128 MPI processes > > > > type: fgmres > > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > > > > GMRES: happy breakdown tolerance 1e-30 > > > > maximum iterations=100000, initial guess is zero > > > > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > > > > right preconditioning > > > > using UNPRECONDITIONED norm type for convergence test > > > > PC Object: 128 MPI processes > > > > type: mg > > > > MG: type is MULTIPLICATIVE, levels=1 cycles=v > > > > Cycles per PCApply=1 > > > > Not using Galerkin computed coarse grid matrices > > > > Coarse grid solver -- level ------------------------------- > > > > KSP Object: (mg_levels_0_) 128 MPI processes > > > > type: chebyshev > > > > Chebyshev: eigenvalue estimates: min = 0.223549, max = > 2.45903 > > > > Chebyshev: estimated using: [0 0.1; 0 1.1] > > > > KSP Object: (mg_levels_0_est_) 128 MPI > processes > > > > type: gmres > > > > GMRES: restart=30, using Classical (unmodified) > Gram-Schmidt Orthogonalization with no iterative refinement > > > > GMRES: happy breakdown tolerance 1e-30 > > > > maximum iterations=10, initial guess is zero > > > > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > > > > left preconditioning > > > > using NONE norm type for convergence test > > > > PC Object: (mg_levels_0_) 128 MPI processes > > > > type: sor > > > > SOR: type = local_symmetric, iterations = 1, local > iterations = 1, omega = 1 > > > > linear system matrix followed by preconditioner matrix: > > > > Matrix Object: 128 MPI processes > > > > type: mpiaij > > > > rows=279669, cols=279669 > > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > > total number of mallocs used during MatSetValues calls =0 > > > > not using I-node (on process 0) routines > > > > Matrix Object: 128 MPI processes > > > > type: mpiaij > > > > rows=279669, cols=279669 > > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > > total number of mallocs used during MatSetValues calls =0 > > > > not using I-node (on process 0) routines > > > > maximum iterations=1, initial guess is zero > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > > > left preconditioning > > > > using NONE norm type for convergence test > > > > PC Object: (mg_levels_0_) 128 MPI processes > > > > type: sor > > > > SOR: type = local_symmetric, iterations = 1, local > iterations = 1, omega = 1 > > > > linear system matrix followed by preconditioner matrix: > > > > Matrix Object: 128 MPI processes > > > > type: mpiaij > > > > rows=279669, cols=279669 > > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > > total number of mallocs used during MatSetValues calls =0 > > > > not using I-node (on process 0) routines > > > > Matrix Object: 128 MPI processes > > > > type: mpiaij > > > > rows=279669, cols=279669 > > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > > total number of mallocs used during MatSetValues calls =0 > > > > not using I-node (on process 0) routines > > > > linear system matrix followed by preconditioner matrix: > > > > Matrix Object: 128 MPI processes > > > > type: mpiaij > > > > rows=279669, cols=279669 > > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > > total number of mallocs used during MatSetValues calls =0 > > > > not using I-node (on process 0) routines > > > > Matrix Object: 128 MPI processes > > > > type: mpiaij > > > > rows=279669, cols=279669 > > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > > total number of mallocs used during MatSetValues calls =0 > > > > not using I-node (on process 0) routines > > > > > > > > When I build the same program using PETSc-3.7 and run it with the > same options we observe that the runtime increases and the convergence is > slightly different. The output of the KSP view is: > > > > > > > > KSP Object: 128 MPI processes > > > > type: fgmres > > > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > > > > GMRES: happy breakdown tolerance 1e-30 > > > > maximum iterations=100000, initial guess is zero > > > > tolerances: relative=1e-06, absolute=1e-50, divergence=10000. > > > > right preconditioning > > > > using UNPRECONDITIONED norm type for convergence test > > > > PC Object: 128 MPI processes > > > > type: mg > > > > MG: type is MULTIPLICATIVE, levels=1 cycles=v > > > > Cycles per PCApply=1 > > > > Not using Galerkin computed coarse grid matrices > > > > Coarse grid solver -- level ------------------------------- > > > > KSP Object: (mg_levels_0_) 128 MPI processes > > > > type: chebyshev > > > > Chebyshev: eigenvalue estimates: min = 0.223549, max = > 2.45903 > > > > Chebyshev: eigenvalues estimated using gmres with > translations [0. 0.1; 0. 1.1] > > > > KSP Object: (mg_levels_0_esteig_) 128 MPI > processes > > > > type: gmres > > > > GMRES: restart=30, using Classical (unmodified) > Gram-Schmidt Orthogonalization with no iterative refinement > > > > GMRES: happy breakdown tolerance 1e-30 > > > > maximum iterations=10, initial guess is zero > > > > tolerances: relative=1e-12, absolute=1e-50, > divergence=10000. > > > > left preconditioning > > > > using PRECONDITIONED norm type for convergence test > > > > maximum iterations=2, initial guess is zero > > > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > > > left preconditioning > > > > using NONE norm type for convergence test > > > > PC Object: (mg_levels_0_) 128 MPI processes > > > > type: sor > > > > SOR: type = local_symmetric, iterations = 1, local > iterations = 1, omega = 1. > > > > linear system matrix followed by preconditioner matrix: > > > > Mat Object: 128 MPI processes > > > > type: mpiaij > > > > rows=279669, cols=279669 > > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > > total number of mallocs used during MatSetValues calls =0 > > > > not using I-node (on process 0) routines > > > > Mat Object: 128 MPI processes > > > > type: mpiaij > > > > rows=279669, cols=279669 > > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > > total number of mallocs used during MatSetValues calls =0 > > > > not using I-node (on process 0) routines > > > > linear system matrix followed by preconditioner matrix: > > > > Mat Object: 128 MPI processes > > > > type: mpiaij > > > > rows=279669, cols=279669 > > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > > total number of mallocs used during MatSetValues calls =0 > > > > not using I-node (on process 0) routines > > > > Mat Object: 128 MPI processes > > > > type: mpiaij > > > > rows=279669, cols=279669 > > > > total: nonzeros=6427943, allocated nonzeros=6427943 > > > > total number of mallocs used during MatSetValues calls =0 > > > > not using I-node (on process 0) routines > > > > > > > > I was able to get a closer solution adding the following options: > > > > -mg_levels_0_esteig_ksp_norm_type none > > > > -mg_levels_0_esteig_ksp_rtol 1.0e-5 > > > > -mg_levels_ksp_max_it 1 > > > > > > > > But I still can reach the same runtime we were observing with > PETSc-3.4, could you please advice me if I should specify any other options? > > > > > > > > Thank you very much for your support, > > > > Federico Golfre' Andreasi > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 21 07:16:39 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 21 Sep 2017 08:16:39 -0400 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: Message-ID: On Wed, Sep 20, 2017 at 6:19 PM, Sanjay Govindjee wrote: > Matt, > There is such a thing as non-local plasticity, one where there is a > separate field equation to > solve for the plasticity (beyond the balance of momentum). > I had not seen that. We use non-locality in electrostatics to account for solvent screening which cannot be modeling well in the continuum. Thanks, Matt > -sanjay > > On 9/20/17 4:51 PM, Matthew Knepley wrote: > > On Wed, Sep 20, 2017 at 3:46 PM, Maximilian Hartig < > imilian.hartig at gmail.com> wrote: > >> >> On 20. Sep 2017, at 19:05, Matthew Knepley wrote: >> >> On Wed, Sep 20, 2017 at 12:57 PM, Maximilian Hartig < >> imilian.hartig at gmail.com> wrote: >> >>> On 20. Sep 2017, at 18:17, Matthew Knepley wrote: >>> >>> On Wed, Sep 20, 2017 at 11:46 AM, Maximilian Hartig < >>> imilian.hartig at gmail.com> wrote: >>> >>>> Hello, >>>> >>>> I?m trying to implement plasticity using petscFE but I am quite stuck >>>> since a while. Here?s what I?m trying to do: >>>> >>>> I have a TS which solves the following equation: >>>> gradient(stress) +Forces = density*acceleration >>>> where at the moment stress is a linear function of the strain and hence >>>> the gradient of the displacement. This works fine. Now I want to compare >>>> the stress to a reference value and if it lies above this yield stress, I >>>> have to reevaluate the stress at the respective location. Then I need to >>>> update the plastic strain / yield stress at this location. >>>> I tried doing that first by solving three fields at the same time: >>>> displacements, stresses and yield stress. This failed. >>>> Then, I tried solving only for displacement increments, storing the >>>> displacements, stresses and yield stress from the past time step in an >>>> auxiliary field. The auxiliary fields are updated after each time step with >>>> a second SNES, using the displacement increments from the current, >>>> converged time step. This also failed. >>>> In both cases the code had problems converging and when it did, I ended >>>> up with negative plastic strain. This is not possible and I don?t know how >>>> it happens because I explicitly only increment the plastic strain when the >>>> increment is positive. >>>> >>>> I am sure there is an easy solution to how I can update the internal >>>> variables and determine the correct stress for the residual but I just >>>> cannot figure it out. I?d be thankful for any hints. >>>> >>> >>> It looks like there are two problems above: >>> >>> 1) Convergence >>> >>> For any convergence question, we at minimum need to see the output of >>> >>> -snes_view -snes_converged_reason -snes_monitor >>> -ksp_monitor_true_residual -snes_linesearch_monitor >>> >>> However, this does not seem to be the main issue. >>> >>> 2) Negative plastic strain >>> >>> >>> This is what I?m mainly concerned with. >>> >>> >>> If the system really converged (I cannot tell without other >>> information), then the system formulation is wrong. Of course, its >>> really easy to check by just plugging your solution into the residual >>> function too. I do not understand your explanation above >>> completely however. Do you solve for the plastic strain or the increment? >>> >>> >>> I am trying to find a formulation that works and I think there is a core >>> concept I am just not ?getting?. >>> I want to solve for the displacements. >>> This works fine in an elastic case. When plasticity is involved, I need >>> to determine the actual stress for my residual evaluation and I have not >>> found a way to do that. >>> All formulations for stress I found in literature use strain increments >>> so I tried to just solve for increments each timestep and then add them >>> together in tspoststep. But I still need to somehow evaluate the stress for >>> my displacement increment residuals. So currently, I have auxiliary fields >>> with the stress and the plastic strain. >>> >> >> First question: Don't you get stress by just applying a local operator, >> rather than a solve? >> >> That depends on the type of plasticity. >> > > What type of plasticity is not local? > > >> For a linear hardening formulation it is correct that I could just apply >> a local operator. I?d be happy with that for now. But I?d still need to >> save stress state and plastic strain to determine whether or not I?m >> dealing with a plasticity. I don?t know how to do that inside the residual >> evaluation. >> > > I do not know what you mean by this, meaning why you can't just save these > as auxiliary fields. Also, it would seem to be enough to have the old > displacement and the plastic strain. > > >> Plus DMProjectField seems to have problems evaluating the gradient when >> boundary conditions are imposed. >> > > There are several examples where we do exactly this. Can you show me what > you mean by this? > > Thanks, > > Matt > > >> Thanks, >> Max >> >> >> Thanks, >> >> Matt >> >> >>> I evaluate the current trial stress by adding a stress increment >>> assuming elastic behaviour. If the trial stress lies beyond the yield >>> stress I calculate the corrected stress to evaluate my residual for the >>> displacements. But now I somehow need to update my plastic strain and the >>> stress in the auxiliary fields. So in tspoststep I created another SNES to >>> now calculate the stress and plastic strain while the displacement is the >>> auxiliary field. >>> >>> I?m sure there?s an elegant solution on how to update internal variables >>> but I have not found it. >>> >>> Thanks, >>> Max >>> >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Max >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> http://www.caam.rice.edu/~mk51/ >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > > > -- > ------------------------------------------------------------------- > Sanjay Govindjee, PhD, PE > Horace, Dorothy, and Katherine Johnson Professor in Engineering > > 779 Davis Hall > University of California > Berkeley, CA 94720-1710 > > Voice: +1 510 642 6060 <(510)%20642-6060> > FAX: +1 510 643 5264 <(510)%20643-5264>s_g at berkeley.eduhttp://faculty.ce.berkeley.edu/sanjay > ------------------------------------------------------------------- > > Books: > > Engineering Mechanics of Deformable > Solids: A Presentation with Exerciseshttp://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641http://ukcatalogue.oup.com/product/9780199651641.dohttp://amzn.com/0199651647 > > Engineering Mechanics 3 (Dynamics) 2nd Editionhttp://www.springer.com/978-3-642-53711-0http://amzn.com/3642537111 > > Engineering Mechanics 3, Supplementary Problems: Dynamics http://www.amzn.com/B00SOXN8JU > > ----------------------------------------------- > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Sep 21 08:55:54 2017 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 21 Sep 2017 09:55:54 -0400 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: Message-ID: > > In both cases the code had problems converging and when it did, I ended up > with negative plastic strain. This is not possible and I don?t know how it > happens because I explicitly only increment the plastic strain when the > increment is positive. > > This sounds like a logic bug. You are updating the auxilary variable for plastic strain with all positive numbers but as the system evolves you get negative numbers? PETSc does does not change these number other than computing the gradients for you. > I am sure there is an easy solution to how I can update the internal > variables and determine the correct stress for the residual but I just > cannot figure it out. I?d be thankful for any hints. > > Thanks, > Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Sep 21 09:02:50 2017 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 21 Sep 2017 10:02:50 -0400 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: Message-ID: > > >> That depends on the type of plasticity. For a linear hardening > formulation it is correct that I could just apply a local operator. I?d be > happy with that for now. > I would just do this for now and get it working. > But I?d still need to save stress state and plastic strain to determine > whether or not I?m dealing with a plasticity. I don?t know how to do that > inside the residual evaluation. Plus DMProjectField seems to have problems > evaluating the gradient when boundary conditions are imposed. > After you get the local version working try your nonlocal thing without BCs. You may find bugs while you do this that are causing your problem, and if not we can think more about it. > > Thanks, > Max > > > Thanks, > > Matt > > >> I evaluate the current trial stress by adding a stress increment assuming >> elastic behaviour. If the trial stress lies beyond the yield stress I >> calculate the corrected stress to evaluate my residual for the >> displacements. But now I somehow need to update my plastic strain and the >> stress in the auxiliary fields. So in tspoststep I created another SNES to >> now calculate the stress and plastic strain while the displacement is the >> auxiliary field. >> >> I?m sure there?s an elegant solution on how to update internal variables >> but I have not found it. >> >> Thanks, >> Max >> >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Max >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imilian.hartig at gmail.com Thu Sep 21 10:28:03 2017 From: imilian.hartig at gmail.com (Maximilian Hartig) Date: Thu, 21 Sep 2017 17:28:03 +0200 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: Message-ID: <6BE7C5AF-4D2B-4192-81A7-D940CF8D0B0E@gmail.com> > On 20. Sep 2017, at 22:51, Matthew Knepley wrote: > > On Wed, Sep 20, 2017 at 3:46 PM, Maximilian Hartig > wrote: > >> On 20. Sep 2017, at 19:05, Matthew Knepley > wrote: >> >> On Wed, Sep 20, 2017 at 12:57 PM, Maximilian Hartig > wrote: >>> On 20. Sep 2017, at 18:17, Matthew Knepley > wrote: >>> >>> On Wed, Sep 20, 2017 at 11:46 AM, Maximilian Hartig > wrote: >>> Hello, >>> >>> I?m trying to implement plasticity using petscFE but I am quite stuck since a while. Here?s what I?m trying to do: >>> >>> I have a TS which solves the following equation: >>> gradient(stress) +Forces = density*acceleration >>> where at the moment stress is a linear function of the strain and hence the gradient of the displacement. This works fine. Now I want to compare the stress to a reference value and if it lies above this yield stress, I have to reevaluate the stress at the respective location. Then I need to update the plastic strain / yield stress at this location. >>> I tried doing that first by solving three fields at the same time: displacements, stresses and yield stress. This failed. >>> Then, I tried solving only for displacement increments, storing the displacements, stresses and yield stress from the past time step in an auxiliary field. The auxiliary fields are updated after each time step with a second SNES, using the displacement increments from the current, converged time step. This also failed. >>> In both cases the code had problems converging and when it did, I ended up with negative plastic strain. This is not possible and I don?t know how it happens because I explicitly only increment the plastic strain when the increment is positive. >>> >>> I am sure there is an easy solution to how I can update the internal variables and determine the correct stress for the residual but I just cannot figure it out. I?d be thankful for any hints. >>> >>> It looks like there are two problems above: >>> >>> 1) Convergence >>> >>> For any convergence question, we at minimum need to see the output of >>> >>> -snes_view -snes_converged_reason -snes_monitor -ksp_monitor_true_residual -snes_linesearch_monitor >>> >>> However, this does not seem to be the main issue. >>> >>> 2) Negative plastic strain >> >> This is what I?m mainly concerned with. >>> >>> If the system really converged (I cannot tell without other information), then the system formulation is wrong. Of course, its >>> really easy to check by just plugging your solution into the residual function too. I do not understand your explanation above >>> completely however. Do you solve for the plastic strain or the increment? >> >> I am trying to find a formulation that works and I think there is a core concept I am just not ?getting?. >> I want to solve for the displacements. >> This works fine in an elastic case. When plasticity is involved, I need to determine the actual stress for my residual evaluation and I have not found a way to do that. >> All formulations for stress I found in literature use strain increments so I tried to just solve for increments each timestep and then add them together in tspoststep. But I still need to somehow evaluate the stress for my displacement increment residuals. So currently, I have auxiliary fields with the stress and the plastic strain. >> >> First question: Don't you get stress by just applying a local operator, rather than a solve? > That depends on the type of plasticity. > > What type of plasticity is not local? I did not express myself correctly. The plasticity is local, but for nonlinear hardening I would have to iteratively solve to get the correct stress and plastic strain from the displacement increments. > > For a linear hardening formulation it is correct that I could just apply a local operator. I?d be happy with that for now. But I?d still need to save stress state and plastic strain to determine whether or not I?m dealing with a plasticity. I don?t know how to do that inside the residual evaluation. > > I do not know what you mean by this, meaning why you can't just save these as auxiliary fields. Also, it would seem to be enough to have the old displacement and the plastic strain. Yes, I can update the auxiliary fields but only after I solved for the displacements in my understanding. I still need the stress though as I have to determine whether I am in the plastic or the elastic domain. > > Plus DMProjectField seems to have problems evaluating the gradient when boundary conditions are imposed. > > There are several examples where we do exactly this. Can you show me what you mean by this? Yes, of course. I sent an example a week ago but maybe there was a problem with the attached files. I?ll copy and paste the code and the gmsh file as text below. Thanks, Max #include #include #include #include #include #include #include #include #include /* define the pointwise functions we'd like to project */ void projectstress(PetscInt dim, PetscInt Nf, PetscInt NfAux, const PetscInt uOff[], const PetscInt uOff_x[], const PetscScalar u[], const PetscScalar u_t[], const PetscScalar u_x[], const PetscInt aOff[], const PetscInt aOff_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar a_x[], PetscReal t, const PetscScalar x[], PetscInt numConstants, const PetscScalar constants[], PetscScalar v[]){ const PetscReal mu =76.923076923, lbda=115.384615385; PetscInt Ncomp = dim; PetscInt comp,d; PetscReal sigma[dim*dim]; for(comp=0;comp > Thanks, > > Matt > > Thanks, > Max >> >> Thanks, >> >> Matt >> >> I evaluate the current trial stress by adding a stress increment assuming elastic behaviour. If the trial stress lies beyond the yield stress I calculate the corrected stress to evaluate my residual for the displacements. But now I somehow need to update my plastic strain and the stress in the auxiliary fields. So in tspoststep I created another SNES to now calculate the stress and plastic strain while the displacement is the auxiliary field. >> >> I?m sure there?s an elegant solution on how to update internal variables but I have not found it. >> >> Thanks, >> Max >>> >>> Thanks, >>> >>> Matt >>> >>> Thanks, >>> Max >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> http://www.caam.rice.edu/~mk51/ >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 21 10:33:28 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 21 Sep 2017 11:33:28 -0400 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: <6BE7C5AF-4D2B-4192-81A7-D940CF8D0B0E@gmail.com> References: <6BE7C5AF-4D2B-4192-81A7-D940CF8D0B0E@gmail.com> Message-ID: On Thu, Sep 21, 2017 at 11:28 AM, Maximilian Hartig < imilian.hartig at gmail.com> wrote: > On 20. Sep 2017, at 22:51, Matthew Knepley wrote: > > On Wed, Sep 20, 2017 at 3:46 PM, Maximilian Hartig com> wrote: > >> >> On 20. Sep 2017, at 19:05, Matthew Knepley wrote: >> >> On Wed, Sep 20, 2017 at 12:57 PM, Maximilian Hartig < >> imilian.hartig at gmail.com> wrote: >> >>> On 20. Sep 2017, at 18:17, Matthew Knepley wrote: >>> >>> On Wed, Sep 20, 2017 at 11:46 AM, Maximilian Hartig < >>> imilian.hartig at gmail.com> wrote: >>> >>>> Hello, >>>> >>>> I?m trying to implement plasticity using petscFE but I am quite stuck >>>> since a while. Here?s what I?m trying to do: >>>> >>>> I have a TS which solves the following equation: >>>> gradient(stress) +Forces = density*acceleration >>>> where at the moment stress is a linear function of the strain and hence >>>> the gradient of the displacement. This works fine. Now I want to compare >>>> the stress to a reference value and if it lies above this yield stress, I >>>> have to reevaluate the stress at the respective location. Then I need to >>>> update the plastic strain / yield stress at this location. >>>> I tried doing that first by solving three fields at the same time: >>>> displacements, stresses and yield stress. This failed. >>>> Then, I tried solving only for displacement increments, storing the >>>> displacements, stresses and yield stress from the past time step in an >>>> auxiliary field. The auxiliary fields are updated after each time step with >>>> a second SNES, using the displacement increments from the current, >>>> converged time step. This also failed. >>>> In both cases the code had problems converging and when it did, I ended >>>> up with negative plastic strain. This is not possible and I don?t know how >>>> it happens because I explicitly only increment the plastic strain when the >>>> increment is positive. >>>> >>>> I am sure there is an easy solution to how I can update the internal >>>> variables and determine the correct stress for the residual but I just >>>> cannot figure it out. I?d be thankful for any hints. >>>> >>> >>> It looks like there are two problems above: >>> >>> 1) Convergence >>> >>> For any convergence question, we at minimum need to see the output of >>> >>> -snes_view -snes_converged_reason -snes_monitor >>> -ksp_monitor_true_residual -snes_linesearch_monitor >>> >>> However, this does not seem to be the main issue. >>> >>> 2) Negative plastic strain >>> >>> >>> This is what I?m mainly concerned with. >>> >>> >>> If the system really converged (I cannot tell without other >>> information), then the system formulation is wrong. Of course, its >>> really easy to check by just plugging your solution into the residual >>> function too. I do not understand your explanation above >>> completely however. Do you solve for the plastic strain or the increment? >>> >>> >>> I am trying to find a formulation that works and I think there is a core >>> concept I am just not ?getting?. >>> I want to solve for the displacements. >>> This works fine in an elastic case. When plasticity is involved, I need >>> to determine the actual stress for my residual evaluation and I have not >>> found a way to do that. >>> All formulations for stress I found in literature use strain increments >>> so I tried to just solve for increments each timestep and then add them >>> together in tspoststep. But I still need to somehow evaluate the stress for >>> my displacement increment residuals. So currently, I have auxiliary fields >>> with the stress and the plastic strain. >>> >> >> First question: Don't you get stress by just applying a local operator, >> rather than a solve? >> >> That depends on the type of plasticity. >> > > What type of plasticity is not local? > > I did not express myself correctly. The plasticity is local, but for > nonlinear hardening I would have to iteratively solve to get the correct > stress and plastic strain from the displacement increments. > Okay, I see. Lets leave that for the time being. > > >> For a linear hardening formulation it is correct that I could just apply >> a local operator. I?d be happy with that for now. But I?d still need to >> save stress state and plastic strain to determine whether or not I?m >> dealing with a plasticity. I don?t know how to do that inside the residual >> evaluation. >> > > I do not know what you mean by this, meaning why you can't just save these > as auxiliary fields. Also, it would seem to be enough to have the old > displacement and the plastic strain. > > Yes, I can update the auxiliary fields but only after I solved for the > displacements in my understanding. I still need the stress though as I have > to determine whether I am in the plastic or the elastic domain. > Right, but if the stress is local, then you can just compute it in the kernel (this is what I do). Even if the relationship is implicit, you can compute that solve local to the (Gauss) point. > Plus DMProjectField seems to have problems evaluating the gradient when >> boundary conditions are imposed. >> > > There are several examples where we do exactly this. Can you show me what > you mean by this? > > > Yes, of course. I sent an example a week ago but maybe there was a problem > with the attached files. I?ll copy and paste the code and the gmsh file as > text below. > Okay, I got this. Can you tell me how to run it and why you think stuff is wrong? Thanks, Matt > Thanks, > Max > > > > > #include > #include > #include > #include > #include > #include > #include > #include > #include > > /* define the pointwise functions we'd like to project */ > > void projectstress(PetscInt dim, PetscInt Nf, PetscInt NfAux, const > PetscInt uOff[], > const PetscInt uOff_x[], const PetscScalar u[], > const PetscScalar u_t[], > const PetscScalar u_x[], const PetscInt aOff[], > const PetscInt aOff_x[], > const PetscScalar a[], const PetscScalar a_t[], > const PetscScalar a_x[], > PetscReal t, const PetscScalar x[], PetscInt > numConstants, > const PetscScalar constants[], PetscScalar v[]){ > const PetscReal mu =76.923076923, lbda=115.384615385; > PetscInt Ncomp = dim; > PetscInt comp,d; > PetscReal sigma[dim*dim]; > > for(comp=0;comp for(d=0;d sigma[comp*dim+d]=mu*(u_x[comp*dim+d]+u_x[d*dim+comp]); > } > for(d=0;d sigma[comp*dim+comp]+=lbda*u_x[d*dim+d]; > } > } > > for(d=0;d v[d] = sigma[d*dim+d]; > } > v[3] = sigma[0*dim+1]; > v[4] = sigma[0*dim+2]; > v[5] = sigma[1*dim+2]; > > > } > > void projectdisplacement(PetscInt dim, PetscInt Nf, PetscInt NfAux, const > PetscInt uOff[], > const PetscInt uOff_x[], const PetscScalar u[], > const PetscScalar u_t[], > const PetscScalar u_x[], const PetscInt aOff[], > const PetscInt aOff_x[], > const PetscScalar a[], const PetscScalar a_t[], > const PetscScalar a_x[], > PetscReal t, const PetscScalar x[], PetscInt > numConstants, > const PetscScalar constants[], PetscScalar v[]){ > PetscInt d; > > for(d=0;d v[d] =u[d]; > } > } > > static PetscErrorCode initial_displacement_vector(PetscInt dim, PetscReal > time, const PetscReal x[], PetscInt Nf, PetscScalar *u, void *ctx) > { > u[0]=0.0; > u[1]=0.1*pow(x[2],2); > u[2]=0.1*x[2]; > return 0; > } > > static PetscErrorCode expected_stress_vector(PetscInt dim, PetscReal > time, const PetscReal x[], PetscInt Nf, PetscScalar *u, void *ctx) > { > const PetscReal mu =76.923076923, lbda=115.384615385; > PetscReal strain[dim*dim]; > PetscReal gradu[dim*dim]; > PetscReal stress[dim*dim]; > PetscInt i,j; > > /* gradient of the displacement field: */ > for(i=0;i for(j=0;j gradu[i*dim+j]=0.0; > } > } > > gradu[1*dim+2]=0.2*x[2]; > gradu[2*dim+2]=0.1; > > for(i=0;i for(j=0;j strain[i*dim+j]=0.5*(gradu[i*dim+j]+gradu[j*dim+i]); > } > } > > for(i=0;i for(j=0;j stress[i*dim+j]=2.0*mu*strain[i*dim+j]; > } > for(j=0;j stress[i*dim+i]+=lbda*strain[j*dim+j]; > } > } > > for(i=0;i u[i] = stress[i*dim+i]; > } > u[3] = stress[0*dim+1]; > u[4] = stress[0*dim+2]; > u[5] = stress[1*dim+2]; > > > return 0; > } > > static PetscErrorCode zero_stress(PetscInt dim, PetscReal time, const > PetscReal x[], PetscInt Nf, PetscScalar *u, void *ctx) > { > const PetscInt Ncomp = 2*dim; > PetscInt comp; > for (comp=0;comp return 0; > } > > static PetscErrorCode zero_vector(PetscInt dim, PetscReal time, const > PetscReal x[], PetscInt Nf, PetscScalar *u, void *ctx) > { > PetscInt comp; > for (comp=0;comp return 0; > } > > static PetscErrorCode SetupDiscretization(DM dm){ > > PetscInt dim = 3; > PetscFE fe_displacement, fe_stress; > PetscDS prob; > PetscErrorCode ierr; > PetscBool simplex = PETSC_TRUE; > PetscQuadrature q; > PetscInt order; > > /* get the dimension of the problem from the DM */ > > ierr = DMGetDimension(dm, &dim); CHKERRQ(ierr); > > /* Creating the FE for the displacement */ > ierr = PetscFECreateDefault(dm, dim, dim, simplex,"disp_",PETSC_DEFAULT, > &fe_displacement);CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject) fe_displacement, > "displacement");CHKERRQ(ierr); > ierr = PetscFEGetQuadrature(fe_displacement,&q);CHKERRQ(ierr); > ierr = PetscQuadratureGetOrder(q,&order);CHKERRQ(ierr); > /* Creating the FE for the stress */ > ierr = PetscFECreateDefault(dm,dim,2*dim,simplex,"stress_",PETSC_ > DEFAULT,&fe_stress);CHKERRQ(ierr); > ierr = PetscFESetQuadrature(fe_stress,q);CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject) fe_stress, > "cauchy_stress");CHKERRQ(ierr); > > > /* Discretization and boundary conditons: */ > ierr = DMGetDS(dm, &prob);CHKERRQ(ierr); > ierr = PetscDSSetDiscretization(prob, 0, (PetscObject) fe_displacement); > CHKERRQ(ierr); > ierr = PetscDSSetDiscretization(prob, 1, (PetscObject) > fe_stress);CHKERRQ(ierr); > ierr = DMSetDS(dm, prob); CHKERRQ(ierr); > > /* Define the boundaries */ > > const PetscInt Ncomp = dim; > const PetscInt Nfid = 1; > PetscInt fid[Nfid]; /* fixed faces [numer of fixed faces] */ > > PetscInt restrictall[3] = {0, 1, 2}; /* restricting all movement */ > > > > fid[0] = 3; /* fixed face */ > > ierr = DMAddBoundary(dm, PETSC_TRUE, "fixed", "Face > Sets",0,Ncomp,restrictall,(void (*)()) zero_vector, > Nfid,fid,NULL);CHKERRQ(ierr); > > > ierr = PetscFEDestroy(&fe_displacement); CHKERRQ(ierr); > > ierr = PetscFEDestroy(&fe_stress); CHKERRQ(ierr); > > return(0); > } > int main(int argc, char *argv[]) > { > DM dm, distributeddm; /* problem definition */ > Vec u,expected_solution,projected_solution; > PetscViewer viewer; > int dim; /* dimension of the anlysis */ > PetscErrorCode ierr; > PetscMPIInt rank, numProcs; > PetscPartitioner part; > > > > ierr = PetscInitialize(&argc,&argv,NULL,NULL); > ierr = DMPlexCreateFromFile(PETSC_COMM_WORLD,"testcube.msh", > PETSC_TRUE,&(dm)); > ierr = DMGetDimension(dm,&(dim)); > > > /* distribute the mesh */ > MPI_Comm_rank(PETSC_COMM_WORLD, &rank); > MPI_Comm_size(PETSC_COMM_WORLD, &numProcs); > DMPlexGetPartitioner(dm, &part); > PetscPartitionerSetType(part, PETSCPARTITIONERPARMETIS); > > ierr = DMPlexDistribute(dm,0,NULL,&distributeddm); > > if (distributeddm) { > ierr=DMDestroy(&(dm)); > dm = distributeddm; > } > > > ierr = DMView(dm,PETSC_VIEWER_STDOUT_WORLD); > ierr = DMSetMatType(dm, MATAIJ);CHKERRQ(ierr); > > > > ierr = SetupDiscretization(dm);CHKERRQ(ierr); > > ierr = DMPlexCreateClosureIndex(dm,NULL);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(dm, &(u)); CHKERRQ(ierr); > ierr = VecDuplicate(u,&expected_solution); CHKERRQ(ierr); > ierr = VecDuplicate(u,&projected_solution); CHKERRQ(ierr); > > /* intitialize the fields: */ > > PetscErrorCode (*initial[2])(PetscInt dim, PetscReal time, const > PetscReal x[], PetscInt Nf, PetscScalar u[], void* ctx) = > {initial_displacement_vector,zero_stress}; > ierr = DMProjectFunction(dm,0.0,initial,NULL, INSERT_ALL_VALUES, u); > CHKERRQ(ierr); > > PetscErrorCode (*expected_sol[2])(PetscInt dim, PetscReal time, const > PetscReal x[], PetscInt Nf, PetscScalar u[], void* ctx) = > {initial_displacement_vector,expected_stress_vector}; > ierr = DMProjectFunction(dm,0.0,expected_sol,NULL, INSERT_ALL_VALUES, > expected_solution); CHKERRQ(ierr); > > ierr = PetscViewerVTKOpen(PETSC_COMM_WORLD,"expected_solution.vtu", > FILE_MODE_WRITE,&viewer);CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject) expected_solution, > "expected_fields"); CHKERRQ(ierr); > ierr = VecView(expected_solution,viewer);CHKERRQ(ierr); > ierr= PetscViewerDestroy(&viewer); CHKERRQ(ierr); > > > > /* project the fields: */ > > void (*projection[2])(PetscInt dim, PetscInt Nf, PetscInt NfAux, const > PetscInt uOff[], > const PetscInt uOff_x[], const PetscScalar u[], > const PetscScalar u_t[], > const PetscScalar u_x[], const PetscInt aOff[], > const PetscInt aOff_x[], > const PetscScalar a[], const PetscScalar a_t[], > const PetscScalar a_x[], > PetscReal t, const PetscScalar x[], PetscInt > numConstants, > const PetscScalar constants[], PetscScalar v[]) = > {projectdisplacement, projectstress}; > > > ierr = DMProjectField(dm, 0.0, u, projection,INSERT_ALL_VALUES,projected_solution); > CHKERRQ(ierr); > ierr = PetscViewerVTKOpen(PETSC_COMM_WORLD,"projected_solution.vtu" > ,FILE_MODE_WRITE,&viewer);CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject) projected_solution, > "projected_fields"); CHKERRQ(ierr); > ierr = VecView(projected_solution,viewer);CHKERRQ(ierr); > ierr= PetscViewerDestroy(&viewer); CHKERRQ(ierr); > > > VecDestroy(&u); > VecDestroy(&expected_solution); > VecDestroy(&projected_solution); > DMDestroy(&dm); > > PetscFinalize(); > > return 0; > } > > > *testcube.msh:* > > $MeshFormat > 2.2 0 8 > $EndMeshFormat > $PhysicalNames > 6 > 2 1 "back" > 2 2 "front" > 2 3 "bottom" > 2 4 "right" > 2 5 "top" > 2 6 "left" > $EndPhysicalNames > $Nodes > 10 > 1 0 -0.05 0 > 2 0.1 -0.05 0 > 3 0.1 0.05 0 > 4 0 0.05 0 > 5 0 -0.05 0.1 > 6 0.1 -0.05 0.1 > 7 0.1 0.05 0.1 > 8 0 0.05 0.1 > 9 0.05 0 0 > 10 0.05 0 0.1 > $EndNodes > $Elements > 28 > 1 2 2 1 6 1 2 9 > 2 2 2 1 6 1 9 4 > 3 2 2 1 6 2 3 9 > 4 2 2 1 6 3 4 9 > 5 2 2 3 15 5 1 6 > 6 2 2 3 15 1 2 6 > 7 2 2 4 19 6 2 7 > 8 2 2 4 19 2 3 7 > 9 2 2 5 23 7 3 8 > 10 2 2 5 23 3 4 8 > 11 2 2 6 27 8 4 1 > 12 2 2 6 27 8 1 5 > 13 2 2 2 28 5 6 10 > 14 2 2 2 28 5 10 8 > 15 2 2 2 28 6 7 10 > 16 2 2 2 28 7 8 10 > 17 4 2 29 1 1 2 9 6 > 18 4 2 29 1 6 5 10 9 > 19 4 2 29 1 1 5 6 9 > 20 4 2 29 1 1 9 4 8 > 21 4 2 29 1 10 5 8 9 > 22 4 2 29 1 9 5 8 1 > 23 4 2 29 1 2 3 9 7 > 24 4 2 29 1 7 6 10 9 > 25 4 2 29 1 2 6 7 9 > 26 4 2 29 1 3 4 9 8 > 27 4 2 29 1 8 7 10 9 > 28 4 2 29 1 3 7 8 9 > $EndElements > > > > > > Thanks, > > Matt > > >> Thanks, >> Max >> >> >> Thanks, >> >> Matt >> >> >>> I evaluate the current trial stress by adding a stress increment >>> assuming elastic behaviour. If the trial stress lies beyond the yield >>> stress I calculate the corrected stress to evaluate my residual for the >>> displacements. But now I somehow need to update my plastic strain and the >>> stress in the auxiliary fields. So in tspoststep I created another SNES to >>> now calculate the stress and plastic strain while the displacement is the >>> auxiliary field. >>> >>> I?m sure there?s an elegant solution on how to update internal variables >>> but I have not found it. >>> >>> Thanks, >>> Max >>> >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Max >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> http://www.caam.rice.edu/~mk51/ >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From imilian.hartig at gmail.com Thu Sep 21 10:47:51 2017 From: imilian.hartig at gmail.com (Maximilian Hartig) Date: Thu, 21 Sep 2017 17:47:51 +0200 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: Message-ID: > On 21. Sep 2017, at 16:02, Mark Adams wrote: > >>> > > That depends on the type of plasticity. For a linear hardening formulation it is correct that I could just apply a local operator. I?d be happy with that for now. > > I would just do this for now and get it working. I agree, and the local version would be enough for me to continue for now. The issue was that I could not get DMProjectField to work to calculate my stress so I decided to just do a solve and set the residual of the fields to f0[]=u[]-calculatedstress[] and f0[] = u[uOff[1]]-calculatedplasticstrain[] respectively. My calculatedplasticstrain was equal to the one from the last step in case of elastic behaviour. In case of plastic behaviour it was the one from the last step plus a positive increment. Yet I ended up with negative plastic strains. I wouldn?t need to do a global solve if I managed to just project the correct stresses and strains in the auxiliary field. I don?t even want to do it. I just couldn?t get DMProjectFields to give me the correct gradients and tried this as a workaround. > > But I?d still need to save stress state and plastic strain to determine whether or not I?m dealing with a plasticity. I don?t know how to do that inside the residual evaluation. Plus DMProjectField seems to have problems evaluating the gradient when boundary conditions are imposed. > > After you get the local version working try your nonlocal thing without BCs. You may find bugs while you do this that are causing your problem, and if not we can think more about it. I don?t understand what you mean by ?without BC?. If I don?t impose boundary conditions there is nothing to solve. The DMProjectField thing does work without boundary conditions if you mean that. It is only when I impose Dirichlet BC on the dm that I get the wrong gradients. This is also true if I use a second, unconstrained dm_unconstrained to project the fields but the original vector comes from a constrained dm. I also tried creating a second vector on dm_unconstrained and using DMGlobalToLocal to transfer from the original vector to the second, unconstrained one and then this one as input for DMProjectField. It does not prevent the gradient issue from happening. I use petsc from bitbucket origin/master Thanks, Max > > > Thanks, > Max >> >> Thanks, >> >> Matt >> >> I evaluate the current trial stress by adding a stress increment assuming elastic behaviour. If the trial stress lies beyond the yield stress I calculate the corrected stress to evaluate my residual for the displacements. But now I somehow need to update my plastic strain and the stress in the auxiliary fields. So in tspoststep I created another SNES to now calculate the stress and plastic strain while the displacement is the auxiliary field. >> >> I?m sure there?s an elegant solution on how to update internal variables but I have not found it. >> >> Thanks, >> Max >>> >>> Thanks, >>> >>> Matt >>> >>> Thanks, >>> Max >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> http://www.caam.rice.edu/~mk51/ >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imilian.hartig at gmail.com Thu Sep 21 11:02:26 2017 From: imilian.hartig at gmail.com (Maximilian Hartig) Date: Thu, 21 Sep 2017 18:02:26 +0200 Subject: [petsc-users] Hints for using petscfe for plasticity -- how to update/access internal variables? In-Reply-To: References: <6BE7C5AF-4D2B-4192-81A7-D940CF8D0B0E@gmail.com> Message-ID: > On 21. Sep 2017, at 17:33, Matthew Knepley wrote: > > On Thu, Sep 21, 2017 at 11:28 AM, Maximilian Hartig > wrote: >> On 20. Sep 2017, at 22:51, Matthew Knepley > wrote: >> >> On Wed, Sep 20, 2017 at 3:46 PM, Maximilian Hartig > wrote: >> >>> On 20. Sep 2017, at 19:05, Matthew Knepley > wrote: >>> >>> On Wed, Sep 20, 2017 at 12:57 PM, Maximilian Hartig > wrote: >>>> On 20. Sep 2017, at 18:17, Matthew Knepley > wrote: >>>> >>>> On Wed, Sep 20, 2017 at 11:46 AM, Maximilian Hartig > wrote: >>>> Hello, >>>> >>>> I?m trying to implement plasticity using petscFE but I am quite stuck since a while. Here?s what I?m trying to do: >>>> >>>> I have a TS which solves the following equation: >>>> gradient(stress) +Forces = density*acceleration >>>> where at the moment stress is a linear function of the strain and hence the gradient of the displacement. This works fine. Now I want to compare the stress to a reference value and if it lies above this yield stress, I have to reevaluate the stress at the respective location. Then I need to update the plastic strain / yield stress at this location. >>>> I tried doing that first by solving three fields at the same time: displacements, stresses and yield stress. This failed. >>>> Then, I tried solving only for displacement increments, storing the displacements, stresses and yield stress from the past time step in an auxiliary field. The auxiliary fields are updated after each time step with a second SNES, using the displacement increments from the current, converged time step. This also failed. >>>> In both cases the code had problems converging and when it did, I ended up with negative plastic strain. This is not possible and I don?t know how it happens because I explicitly only increment the plastic strain when the increment is positive. >>>> >>>> I am sure there is an easy solution to how I can update the internal variables and determine the correct stress for the residual but I just cannot figure it out. I?d be thankful for any hints. >>>> >>>> It looks like there are two problems above: >>>> >>>> 1) Convergence >>>> >>>> For any convergence question, we at minimum need to see the output of >>>> >>>> -snes_view -snes_converged_reason -snes_monitor -ksp_monitor_true_residual -snes_linesearch_monitor >>>> >>>> However, this does not seem to be the main issue. >>>> >>>> 2) Negative plastic strain >>> >>> This is what I?m mainly concerned with. >>>> >>>> If the system really converged (I cannot tell without other information), then the system formulation is wrong. Of course, its >>>> really easy to check by just plugging your solution into the residual function too. I do not understand your explanation above >>>> completely however. Do you solve for the plastic strain or the increment? >>> >>> I am trying to find a formulation that works and I think there is a core concept I am just not ?getting?. >>> I want to solve for the displacements. >>> This works fine in an elastic case. When plasticity is involved, I need to determine the actual stress for my residual evaluation and I have not found a way to do that. >>> All formulations for stress I found in literature use strain increments so I tried to just solve for increments each timestep and then add them together in tspoststep. But I still need to somehow evaluate the stress for my displacement increment residuals. So currently, I have auxiliary fields with the stress and the plastic strain. >>> >>> First question: Don't you get stress by just applying a local operator, rather than a solve? >> That depends on the type of plasticity. >> >> What type of plasticity is not local? > I did not express myself correctly. The plasticity is local, but for nonlinear hardening I would have to iteratively solve to get the correct stress and plastic strain from the displacement increments. > > Okay, I see. Lets leave that for the time being. >> >> For a linear hardening formulation it is correct that I could just apply a local operator. I?d be happy with that for now. But I?d still need to save stress state and plastic strain to determine whether or not I?m dealing with a plasticity. I don?t know how to do that inside the residual evaluation. >> >> I do not know what you mean by this, meaning why you can't just save these as auxiliary fields. Also, it would seem to be enough to have the old displacement and the plastic strain. > Yes, I can update the auxiliary fields but only after I solved for the displacements in my understanding. I still need the stress though as I have to determine whether I am in the plastic or the elastic domain. > > Right, but if the stress is local, then you can just compute it in the kernel (this is what I do). Even if the relationship is implicit, you can > compute that solve local to the (Gauss) point. Just to clarify, with kernel you mean the residual function I give to PetscDSSetResidual(?), correct? I currently do this but did not find to save plastic strain and stress inside this function. So I decided to reevaluate them after I had the correct displacement field. Since I?d have to do that only once per time step, I thought the extra computational effort would be negligible if I could just project them. >> Plus DMProjectField seems to have problems evaluating the gradient when boundary conditions are imposed. >> >> There are several examples where we do exactly this. Can you show me what you mean by this? > > Yes, of course. I sent an example a week ago but maybe there was a problem with the attached files. I?ll copy and paste the code and the gmsh file as text below. > > Okay, I got this. Can you tell me how to run it and why you think stuff is wrong? Sure, I compile with the following makefile (evidently the c file is called projectfieldtest.c): PETSC_DIR = ... PETSC_ARCH = arch-darwin-c-debug CFLAGS = FFLAGS = CPPFLAGS = FPPFLAGS = MANSEC = LOCDIR = include ${PETSC_DIR}/${PETSC_ARCH}/lib/petsc/conf/petscvariables include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules projectfieldtest: projectfieldtest.o chkopts -${CLINKER} -o projectfieldtest projectfieldtest.o ${PETSC_LIB} ${RM} projectfieldtest.o Then run with the following options: -disp_petscspace_order 2 -stress_petscspace_order 2 I use petsc from bitbucket origin/master with configure options: --download-mumps --download-chacoa --download-scalapack --download-mpich --with-fc=gfortran-mp-6 --download-triangle --with-cc=gcc-mp-6 --with-cxx=g++-mp-6 --download-fblaspack --download-parmetis --download-metis I think the result is wrong because I compare it to a solution that I calculated by hand. I project a displacement function to the dm, then compare the stress I get from DMProjectField to the stress I calculated by hand and then projected using DMProjectFunction. Without this line: ierr = DMAddBoundary(dm, PETSC_TRUE, "fixed", "Face Sets",0,Ncomp,restrictall,(void (*)()) zero_vector, Nfid,fid,NULL);CHKERRQ(ierr); the expected solution is the same as the projected solution. With it, they differ. Thanks, Max > > Thanks, > > Matt > > Thanks, > Max > > > > > #include > #include > #include > #include > #include > #include > #include > #include > #include > > /* define the pointwise functions we'd like to project */ > > void projectstress(PetscInt dim, PetscInt Nf, PetscInt NfAux, const PetscInt uOff[], > const PetscInt uOff_x[], const PetscScalar u[], const PetscScalar u_t[], > const PetscScalar u_x[], const PetscInt aOff[], const PetscInt aOff_x[], > const PetscScalar a[], const PetscScalar a_t[], const PetscScalar a_x[], > PetscReal t, const PetscScalar x[], PetscInt numConstants, > const PetscScalar constants[], PetscScalar v[]){ > const PetscReal mu =76.923076923, lbda=115.384615385; > PetscInt Ncomp = dim; > PetscInt comp,d; > PetscReal sigma[dim*dim]; > > for(comp=0;comp for(d=0;d sigma[comp*dim+d]=mu*(u_x[comp*dim+d]+u_x[d*dim+comp]); > } > for(d=0;d sigma[comp*dim+comp]+=lbda*u_x[d*dim+d]; > } > } > > for(d=0;d v[d] = sigma[d*dim+d]; > } > v[3] = sigma[0*dim+1]; > v[4] = sigma[0*dim+2]; > v[5] = sigma[1*dim+2]; > > > } > > void projectdisplacement(PetscInt dim, PetscInt Nf, PetscInt NfAux, const PetscInt uOff[], > const PetscInt uOff_x[], const PetscScalar u[], const PetscScalar u_t[], > const PetscScalar u_x[], const PetscInt aOff[], const PetscInt aOff_x[], > const PetscScalar a[], const PetscScalar a_t[], const PetscScalar a_x[], > PetscReal t, const PetscScalar x[], PetscInt numConstants, > const PetscScalar constants[], PetscScalar v[]){ > PetscInt d; > > for(d=0;d v[d] =u[d]; > } > } > > static PetscErrorCode initial_displacement_vector(PetscInt dim, PetscReal time, const PetscReal x[], PetscInt Nf, PetscScalar *u, void *ctx) > { > u[0]=0.0; > u[1]=0.1*pow(x[2],2); > u[2]=0.1*x[2]; > return 0; > } > > static PetscErrorCode expected_stress_vector(PetscInt dim, PetscReal time, const PetscReal x[], PetscInt Nf, PetscScalar *u, void *ctx) > { > const PetscReal mu =76.923076923, lbda=115.384615385; > PetscReal strain[dim*dim]; > PetscReal gradu[dim*dim]; > PetscReal stress[dim*dim]; > PetscInt i,j; > > /* gradient of the displacement field: */ > for(i=0;i for(j=0;j gradu[i*dim+j]=0.0; > } > } > > gradu[1*dim+2]=0.2*x[2]; > gradu[2*dim+2]=0.1; > > for(i=0;i for(j=0;j strain[i*dim+j]=0.5*(gradu[i*dim+j]+gradu[j*dim+i]); > } > } > > for(i=0;i for(j=0;j stress[i*dim+j]=2.0*mu*strain[i*dim+j]; > } > for(j=0;j stress[i*dim+i]+=lbda*strain[j*dim+j]; > } > } > > for(i=0;i u[i] = stress[i*dim+i]; > } > u[3] = stress[0*dim+1]; > u[4] = stress[0*dim+2]; > u[5] = stress[1*dim+2]; > > > return 0; > } > > static PetscErrorCode zero_stress(PetscInt dim, PetscReal time, const PetscReal x[], PetscInt Nf, PetscScalar *u, void *ctx) > { > const PetscInt Ncomp = 2*dim; > PetscInt comp; > for (comp=0;comp return 0; > } > > static PetscErrorCode zero_vector(PetscInt dim, PetscReal time, const PetscReal x[], PetscInt Nf, PetscScalar *u, void *ctx) > { > PetscInt comp; > for (comp=0;comp return 0; > } > > static PetscErrorCode SetupDiscretization(DM dm){ > > PetscInt dim = 3; > PetscFE fe_displacement, fe_stress; > PetscDS prob; > PetscErrorCode ierr; > PetscBool simplex = PETSC_TRUE; > PetscQuadrature q; > PetscInt order; > > /* get the dimension of the problem from the DM */ > > ierr = DMGetDimension(dm, &dim); CHKERRQ(ierr); > > /* Creating the FE for the displacement */ > ierr = PetscFECreateDefault(dm, dim, dim, simplex,"disp_",PETSC_DEFAULT,&fe_displacement);CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject) fe_displacement, "displacement");CHKERRQ(ierr); > ierr = PetscFEGetQuadrature(fe_displacement,&q);CHKERRQ(ierr); > ierr = PetscQuadratureGetOrder(q,&order);CHKERRQ(ierr); > /* Creating the FE for the stress */ > ierr = PetscFECreateDefault(dm,dim,2*dim,simplex,"stress_",PETSC_DEFAULT,&fe_stress);CHKERRQ(ierr); > ierr = PetscFESetQuadrature(fe_stress,q);CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject) fe_stress, "cauchy_stress");CHKERRQ(ierr); > > > /* Discretization and boundary conditons: */ > ierr = DMGetDS(dm, &prob);CHKERRQ(ierr); > ierr = PetscDSSetDiscretization(prob, 0, (PetscObject) fe_displacement); CHKERRQ(ierr); > ierr = PetscDSSetDiscretization(prob, 1, (PetscObject) fe_stress);CHKERRQ(ierr); > ierr = DMSetDS(dm, prob); CHKERRQ(ierr); > > /* Define the boundaries */ > > const PetscInt Ncomp = dim; > const PetscInt Nfid = 1; > PetscInt fid[Nfid]; /* fixed faces [numer of fixed faces] */ > > PetscInt restrictall[3] = {0, 1, 2}; /* restricting all movement */ > > > > fid[0] = 3; /* fixed face */ > > ierr = DMAddBoundary(dm, PETSC_TRUE, "fixed", "Face Sets",0,Ncomp,restrictall,(void (*)()) zero_vector, Nfid,fid,NULL);CHKERRQ(ierr); > > > ierr = PetscFEDestroy(&fe_displacement); CHKERRQ(ierr); > > ierr = PetscFEDestroy(&fe_stress); CHKERRQ(ierr); > > return(0); > } > int main(int argc, char *argv[]) > { > DM dm, distributeddm; /* problem definition */ > Vec u,expected_solution,projected_solution; > PetscViewer viewer; > int dim; /* dimension of the anlysis */ > PetscErrorCode ierr; > PetscMPIInt rank, numProcs; > PetscPartitioner part; > > > > ierr = PetscInitialize(&argc,&argv,NULL,NULL); > ierr = DMPlexCreateFromFile(PETSC_COMM_WORLD,"testcube.msh", PETSC_TRUE,&(dm)); > ierr = DMGetDimension(dm,&(dim)); > > > /* distribute the mesh */ > MPI_Comm_rank(PETSC_COMM_WORLD, &rank); > MPI_Comm_size(PETSC_COMM_WORLD, &numProcs); > DMPlexGetPartitioner(dm, &part); > PetscPartitionerSetType(part, PETSCPARTITIONERPARMETIS); > > ierr = DMPlexDistribute(dm,0,NULL,&distributeddm); > > if (distributeddm) { > ierr=DMDestroy(&(dm)); > dm = distributeddm; > } > > > ierr = DMView(dm,PETSC_VIEWER_STDOUT_WORLD); > ierr = DMSetMatType(dm, MATAIJ);CHKERRQ(ierr); > > > > ierr = SetupDiscretization(dm);CHKERRQ(ierr); > > ierr = DMPlexCreateClosureIndex(dm,NULL);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(dm, &(u)); CHKERRQ(ierr); > ierr = VecDuplicate(u,&expected_solution); CHKERRQ(ierr); > ierr = VecDuplicate(u,&projected_solution); CHKERRQ(ierr); > > /* intitialize the fields: */ > > PetscErrorCode (*initial[2])(PetscInt dim, PetscReal time, const PetscReal x[], PetscInt Nf, PetscScalar u[], void* ctx) = {initial_displacement_vector,zero_stress}; > ierr = DMProjectFunction(dm,0.0,initial,NULL, INSERT_ALL_VALUES, u); CHKERRQ(ierr); > > PetscErrorCode (*expected_sol[2])(PetscInt dim, PetscReal time, const PetscReal x[], PetscInt Nf, PetscScalar u[], void* ctx) = {initial_displacement_vector,expected_stress_vector}; > ierr = DMProjectFunction(dm,0.0,expected_sol,NULL, INSERT_ALL_VALUES, expected_solution); CHKERRQ(ierr); > > ierr = PetscViewerVTKOpen(PETSC_COMM_WORLD,"expected_solution.vtu",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject) expected_solution, "expected_fields"); CHKERRQ(ierr); > ierr = VecView(expected_solution,viewer);CHKERRQ(ierr); > ierr= PetscViewerDestroy(&viewer); CHKERRQ(ierr); > > > > /* project the fields: */ > > void (*projection[2])(PetscInt dim, PetscInt Nf, PetscInt NfAux, const PetscInt uOff[], > const PetscInt uOff_x[], const PetscScalar u[], const PetscScalar u_t[], > const PetscScalar u_x[], const PetscInt aOff[], const PetscInt aOff_x[], > const PetscScalar a[], const PetscScalar a_t[], const PetscScalar a_x[], > PetscReal t, const PetscScalar x[], PetscInt numConstants, > const PetscScalar constants[], PetscScalar v[]) = {projectdisplacement, projectstress}; > > > ierr = DMProjectField(dm, 0.0, u, projection,INSERT_ALL_VALUES,projected_solution); CHKERRQ(ierr); > ierr = PetscViewerVTKOpen(PETSC_COMM_WORLD,"projected_solution.vtu",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject) projected_solution, "projected_fields"); CHKERRQ(ierr); > ierr = VecView(projected_solution,viewer);CHKERRQ(ierr); > ierr= PetscViewerDestroy(&viewer); CHKERRQ(ierr); > > > VecDestroy(&u); > VecDestroy(&expected_solution); > VecDestroy(&projected_solution); > DMDestroy(&dm); > > PetscFinalize(); > > return 0; > } > > > testcube.msh: > > $MeshFormat > 2.2 0 8 > $EndMeshFormat > $PhysicalNames > 6 > 2 1 "back" > 2 2 "front" > 2 3 "bottom" > 2 4 "right" > 2 5 "top" > 2 6 "left" > $EndPhysicalNames > $Nodes > 10 > 1 0 -0.05 0 > 2 0.1 -0.05 0 > 3 0.1 0.05 0 > 4 0 0.05 0 > 5 0 -0.05 0.1 > 6 0.1 -0.05 0.1 > 7 0.1 0.05 0.1 > 8 0 0.05 0.1 > 9 0.05 0 0 > 10 0.05 0 0.1 > $EndNodes > $Elements > 28 > 1 2 2 1 6 1 2 9 > 2 2 2 1 6 1 9 4 > 3 2 2 1 6 2 3 9 > 4 2 2 1 6 3 4 9 > 5 2 2 3 15 5 1 6 > 6 2 2 3 15 1 2 6 > 7 2 2 4 19 6 2 7 > 8 2 2 4 19 2 3 7 > 9 2 2 5 23 7 3 8 > 10 2 2 5 23 3 4 8 > 11 2 2 6 27 8 4 1 > 12 2 2 6 27 8 1 5 > 13 2 2 2 28 5 6 10 > 14 2 2 2 28 5 10 8 > 15 2 2 2 28 6 7 10 > 16 2 2 2 28 7 8 10 > 17 4 2 29 1 1 2 9 6 > 18 4 2 29 1 6 5 10 9 > 19 4 2 29 1 1 5 6 9 > 20 4 2 29 1 1 9 4 8 > 21 4 2 29 1 10 5 8 9 > 22 4 2 29 1 9 5 8 1 > 23 4 2 29 1 2 3 9 7 > 24 4 2 29 1 7 6 10 9 > 25 4 2 29 1 2 6 7 9 > 26 4 2 29 1 3 4 9 8 > 27 4 2 29 1 8 7 10 9 > 28 4 2 29 1 3 7 8 9 > $EndElements > > > > >> >> Thanks, >> >> Matt >> >> Thanks, >> Max >>> >>> Thanks, >>> >>> Matt >>> >>> I evaluate the current trial stress by adding a stress increment assuming elastic behaviour. If the trial stress lies beyond the yield stress I calculate the corrected stress to evaluate my residual for the displacements. But now I somehow need to update my plastic strain and the stress in the auxiliary fields. So in tspoststep I created another SNES to now calculate the stress and plastic strain while the displacement is the auxiliary field. >>> >>> I?m sure there?s an elegant solution on how to update internal variables but I have not found it. >>> >>> Thanks, >>> Max >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> Thanks, >>>> Max >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>>> >>>> http://www.caam.rice.edu/~mk51/ >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> http://www.caam.rice.edu/~mk51/ >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> http://www.caam.rice.edu/~mk51/ > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > http://www.caam.rice.edu/~mk51/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregory.meyer at berkeley.edu Thu Sep 21 15:47:39 2017 From: gregory.meyer at berkeley.edu (Gregory Meyer) Date: Thu, 21 Sep 2017 20:47:39 +0000 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types Message-ID: Hi all, I'm using shift-invert with EPS to solve for eigenvalues. I find that if I do only ... ierr = EPSGetST(eps,&st);CHKERRQ(ierr); ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); ... in my code I get correct eigenvalues. But if I do ... ierr = EPSGetST(eps,&st);CHKERRQ(ierr); ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); ... the eigenvalues found by EPS are completely wrong! Somehow I thought I was supposed to do the latter, from the examples etc, but I guess that was not correct? I attach the full piece of test code and a test matrix. Best, Greg -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_sinvert.c Type: text/x-csrc Size: 2346 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.mat Type: application/octet-stream Size: 2640 bytes Desc: not available URL: From hzhang at mcs.anl.gov Thu Sep 21 16:05:20 2017 From: hzhang at mcs.anl.gov (Hong) Date: Thu, 21 Sep 2017 16:05:20 -0500 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: Message-ID: Gregory : Use '-eps_view' for both runs to check the algorithms being used. Hong Hi all, > > I'm using shift-invert with EPS to solve for eigenvalues. I find that if I > do only > > ... > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > ... > > in my code I get correct eigenvalues. But if I do > > ... > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); > ... > > the eigenvalues found by EPS are completely wrong! Somehow I thought I was > supposed to do the latter, from the examples etc, but I guess that was not > correct? I attach the full piece of test code and a test matrix. > > Best, > Greg > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregory.meyer at gmail.com Thu Sep 21 16:11:28 2017 From: gregory.meyer at gmail.com (Greg Meyer) Date: Thu, 21 Sep 2017 21:11:28 +0000 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: Message-ID: OK, the difference is whether LU or Cholesky factorization is used. But I would hope that neither one should give incorrect eigenvalues, and when I run with the latter it does! On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: > Gregory : > Use '-eps_view' for both runs to check the algorithms being used. > Hong > > Hi all, >> >> I'm using shift-invert with EPS to solve for eigenvalues. I find that if >> I do only >> >> ... >> ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >> ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >> ... >> >> in my code I get correct eigenvalues. But if I do >> >> ... >> ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >> ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >> ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); >> ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >> ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); >> ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); >> ... >> >> the eigenvalues found by EPS are completely wrong! Somehow I thought I >> was supposed to do the latter, from the examples etc, but I guess that was >> not correct? I attach the full piece of test code and a test matrix. >> >> Best, >> Greg >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Sep 21 17:39:27 2017 From: hzhang at mcs.anl.gov (Hong) Date: Thu, 21 Sep 2017 17:39:27 -0500 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: Message-ID: Greg: > OK, the difference is whether LU or Cholesky factorization is used. But I > would hope that neither one should give incorrect eigenvalues, and when I > run with the latter it does! > Are your matrices symmetric/Hermitian? Hong > > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: > >> Gregory : >> Use '-eps_view' for both runs to check the algorithms being used. >> Hong >> >> Hi all, >>> >>> I'm using shift-invert with EPS to solve for eigenvalues. I find that if >>> I do only >>> >>> ... >>> ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>> ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>> ... >>> >>> in my code I get correct eigenvalues. But if I do >>> >>> ... >>> ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>> ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>> ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); >>> ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >>> ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); >>> ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); >>> ... >>> >>> the eigenvalues found by EPS are completely wrong! Somehow I thought I >>> was supposed to do the latter, from the examples etc, but I guess that was >>> not correct? I attach the full piece of test code and a test matrix. >>> >>> Best, >>> Greg >>> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregory.meyer at gmail.com Thu Sep 21 17:51:19 2017 From: gregory.meyer at gmail.com (Greg Meyer) Date: Thu, 21 Sep 2017 22:51:19 +0000 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: Message-ID: Yes, they are Hermitian. On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: > Greg: > > OK, the difference is whether LU or Cholesky factorization is used. But I >> would hope that neither one should give incorrect eigenvalues, and when I >> run with the latter it does! >> > Are your matrices symmetric/Hermitian? > Hong > >> >> On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: >> >>> Gregory : >>> Use '-eps_view' for both runs to check the algorithms being used. >>> Hong >>> >>> Hi all, >>>> >>>> I'm using shift-invert with EPS to solve for eigenvalues. I find that >>>> if I do only >>>> >>>> ... >>>> ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>>> ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>>> ... >>>> >>>> in my code I get correct eigenvalues. But if I do >>>> >>>> ... >>>> ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>>> ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>>> ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); >>>> ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >>>> ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); >>>> ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); >>>> ... >>>> >>>> the eigenvalues found by EPS are completely wrong! Somehow I thought I >>>> was supposed to do the latter, from the examples etc, but I guess that was >>>> not correct? I attach the full piece of test code and a test matrix. >>>> >>>> Best, >>>> Greg >>>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Sep 21 19:24:29 2017 From: hzhang at mcs.anl.gov (Hong) Date: Thu, 21 Sep 2017 19:24:29 -0500 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: Message-ID: Greg : > Yes, they are Hermitian. > PETSc does not support Cholesky factorization for Hermitian. It seems mumps does not support Hermitian either https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-November/027541.html Hong > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: > >> Greg: >> >> OK, the difference is whether LU or Cholesky factorization is used. But I >>> would hope that neither one should give incorrect eigenvalues, and when I >>> run with the latter it does! >>> >> Are your matrices symmetric/Hermitian? >> Hong >> >>> >>> On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: >>> >>>> Gregory : >>>> Use '-eps_view' for both runs to check the algorithms being used. >>>> Hong >>>> >>>> Hi all, >>>>> >>>>> I'm using shift-invert with EPS to solve for eigenvalues. I find that >>>>> if I do only >>>>> >>>>> ... >>>>> ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>>>> ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>>>> ... >>>>> >>>>> in my code I get correct eigenvalues. But if I do >>>>> >>>>> ... >>>>> ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>>>> ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>>>> ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); >>>>> ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >>>>> ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); >>>>> ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); >>>>> ... >>>>> >>>>> the eigenvalues found by EPS are completely wrong! Somehow I thought I >>>>> was supposed to do the latter, from the examples etc, but I guess that was >>>>> not correct? I attach the full piece of test code and a test matrix. >>>>> >>>>> Best, >>>>> Greg >>>>> >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregory.meyer at gmail.com Thu Sep 21 19:51:04 2017 From: gregory.meyer at gmail.com (Greg Meyer) Date: Fri, 22 Sep 2017 00:51:04 +0000 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: Message-ID: Ok, thanks. It seems that PETSc clearly should throw an error in this case instead of just giving incorrect answers? I am surprised that it does not throw an error... On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: > Greg : > >> Yes, they are Hermitian. >> > > PETSc does not support Cholesky factorization for Hermitian. > It seems mumps does not support Hermitian either > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-November/027541.html > > Hong > > >> On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: >> >>> Greg: >>> >>> OK, the difference is whether LU or Cholesky factorization is used. But >>>> I would hope that neither one should give incorrect eigenvalues, and when I >>>> run with the latter it does! >>>> >>> Are your matrices symmetric/Hermitian? >>> Hong >>> >>>> >>>> On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: >>>> >>>>> Gregory : >>>>> Use '-eps_view' for both runs to check the algorithms being used. >>>>> Hong >>>>> >>>>> Hi all, >>>>>> >>>>>> I'm using shift-invert with EPS to solve for eigenvalues. I find that >>>>>> if I do only >>>>>> >>>>>> ... >>>>>> ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>>>>> ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>>>>> ... >>>>>> >>>>>> in my code I get correct eigenvalues. But if I do >>>>>> >>>>>> ... >>>>>> ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>>>>> ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>>>>> ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); >>>>>> ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >>>>>> ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); >>>>>> ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); >>>>>> ... >>>>>> >>>>>> the eigenvalues found by EPS are completely wrong! Somehow I thought >>>>>> I was supposed to do the latter, from the examples etc, but I guess that >>>>>> was not correct? I attach the full piece of test code and a test matrix. >>>>>> >>>>>> Best, >>>>>> Greg >>>>>> >>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mlohry at princeton.edu Sat Sep 23 12:48:48 2017 From: mlohry at princeton.edu (Mark W. Lohry) Date: Sat, 23 Sep 2017 17:48:48 +0000 Subject: [petsc-users] preconditioning matrix-free newton-krylov Message-ID: <1C4B04A3719F56479255009C095BE3B593D45932@CSGMBX214W.pu.win.princeton.edu> I'm currently using JFNK in an application where I don't have a hand-coded jacobian, and it's working well enough but as expected the scaling isn't great. What is the general process for using PC with MatMFFDComputeJacobian? Does it make sense to occasionally have petsc re-compute the jacobian via finite differences, and then recompute the preconditioner? Any that just need the sparsity structure? Are there any PCs that don't work in the matrix-free context? Are there any example codes I overlooked? Last but not least... can the Boomer-AMG preconditioner work with JFNK? To really show my ignorance of AMG, can it actually be written as a matrix P^-1(Ax-b)=0, , or is it just a linear operator? Thanks, Mark From bsmith at mcs.anl.gov Sat Sep 23 14:12:49 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 23 Sep 2017 14:12:49 -0500 Subject: [petsc-users] preconditioning matrix-free newton-krylov In-Reply-To: <1C4B04A3719F56479255009C095BE3B593D45932@CSGMBX214W.pu.win.princeton.edu> References: <1C4B04A3719F56479255009C095BE3B593D45932@CSGMBX214W.pu.win.princeton.edu> Message-ID: > On Sep 23, 2017, at 12:48 PM, Mark W. Lohry wrote: > > I'm currently using JFNK in an application where I don't have a hand-coded jacobian, and it's working well enough but as expected the scaling isn't great. > > What is the general process for using PC with MatMFFDComputeJacobian? Does it make sense to occasionally have petsc re-compute the jacobian via finite differences, and then recompute the preconditioner? Any that just need the sparsity structure? Mark Yes, this is a common approach. SNESSetLagJacobian -snes_lag_jacobian The normal approach in SNES to use matrix-free for the operator and use finite differences to compute an approximate Jacobian used to construct preconditioners is to to create a sparse matrix with the sparsity of the approximate Jacobian (yes you need a way to figure out the sparsity, if you use DMDA it will figure out the sparsity for you). Then you use SNESSetJacobian(snes,J,J, SNESComputeJacobianDefaultColor, NULL); and use the options database option -snes_mf_operator > Are there any PCs that don't work in the matrix-free context? If you do the above you can use almost all the PC since you are providing an explicit matrix from which to build the preconditioner > Are there any example codes I overlooked? > > Last but not least... can the Boomer-AMG preconditioner work with JFNK? To really show my ignorance of AMG, can it actually be written as a matrix P^-1(Ax-b)=0, , or is it just a linear operator? Again, if you provide an approximate Jacobian like above you can use it with BoomerAMG, if you provide NO explicit matrix you cannot use BoomerAMG or almost any other preconditioner. Barry > > Thanks, > Mark From mlohry at gmail.com Sat Sep 23 14:28:54 2017 From: mlohry at gmail.com (Mark Lohry) Date: Sat, 23 Sep 2017 15:28:54 -0400 Subject: [petsc-users] preconditioning matrix-free newton-krylov In-Reply-To: References: <1C4B04A3719F56479255009C095BE3B593D45932@CSGMBX214W.pu.win.princeton.edu> Message-ID: Great, thanks Barry. On Sat, Sep 23, 2017 at 3:12 PM, Barry Smith wrote: > > > On Sep 23, 2017, at 12:48 PM, Mark W. Lohry > wrote: > > > > I'm currently using JFNK in an application where I don't have a > hand-coded jacobian, and it's working well enough but as expected the > scaling isn't great. > > > > What is the general process for using PC with MatMFFDComputeJacobian? > Does it make sense to occasionally have petsc re-compute the jacobian via > finite differences, and then recompute the preconditioner? Any that just > need the sparsity structure? > > Mark > > Yes, this is a common approach. SNESSetLagJacobian -snes_lag_jacobian > > The normal approach in SNES to use matrix-free for the operator and > use finite differences to compute an approximate Jacobian used to construct > preconditioners is to to create a sparse matrix with the sparsity of the > approximate Jacobian (yes you need a way to figure out the sparsity, if you > use DMDA it will figure out the sparsity for you). Then you use > > SNESSetJacobian(snes,J,J, SNESComputeJacobianDefaultColor, NULL); > > and use the options database option -snes_mf_operator > > > > Are there any PCs that don't work in the matrix-free context? > > If you do the above you can use almost all the PC since you are > providing an explicit matrix from which to build the preconditioner > > > Are there any example codes I overlooked? > > > > Last but not least... can the Boomer-AMG preconditioner work with JFNK? > To really show my ignorance of AMG, can it actually be written as a matrix > P^-1(Ax-b)=0, , or is it just a linear operator? > > Again, if you provide an approximate Jacobian like above you can use it > with BoomerAMG, if you provide NO explicit matrix you cannot use BoomerAMG > or almost any other preconditioner. > > Barry > > > > > Thanks, > > Mark > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From evanum at gmail.com Sat Sep 23 20:38:38 2017 From: evanum at gmail.com (Evan Um) Date: Sat, 23 Sep 2017 18:38:38 -0700 Subject: [petsc-users] Using factored complex matrices from MUMPS as a preconditioner in PETSC Message-ID: Dear PETSC Users, My system matrix comes from finite element modeling and is complex and unstructured. Its typical size is a few millions-by a few millions. I wondering if I can use MUMPS parallel direct solver as a preconditioner in PETSC. For example, I want to pass factored matrices to Krylov iterative solvers such as QMR. Is there any PETSC+MUMPS example code for the purpose? Can PETSC call the latest MUMPS that supports block low rank approximation? In advance, thank you very much for your comments. Best, Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Sep 23 20:45:04 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 23 Sep 2017 20:45:04 -0500 Subject: [petsc-users] Using factored complex matrices from MUMPS as a preconditioner in PETSC In-Reply-To: References: Message-ID: > On Sep 23, 2017, at 8:38 PM, Evan Um wrote: > > Dear PETSC Users, > > My system matrix comes from finite element modeling and is complex and unstructured. Its typical size is a few millions-by a few millions. I wondering if I can use MUMPS parallel direct solver as a preconditioner in PETSC. For example, I want to pass factored matrices to Krylov iterative solvers such as QMR. Is there any PETSC+MUMPS example code for the purpose? You don't pass factored matrices you just pass the original matrix and use -pc_type lu -pc_factor_mat_solver_package mumps > Can PETSC call the latest MUMPS that supports block low rank approximation? No, send us info on it and we'll see if we can add an interface > > In advance, thank you very much for your comments. > > Best, > Evan > > > > > From evanum at gmail.com Sun Sep 24 15:42:28 2017 From: evanum at gmail.com (Evan Um) Date: Sun, 24 Sep 2017 13:42:28 -0700 Subject: [petsc-users] Using factored complex matrices from MUMPS as a preconditioner in PETSC In-Reply-To: References: Message-ID: Hi Barry, Thanks for your comments. To activate block low rank (BLR) approximation in MUMPS version 5.1.1, a user needs to turn on the functionality (i.e. ICNTL(35)=1) and specify the tolerance value (e.g. CNTL(7)=1e-4). In PETSC, I think that we can set up ICNTL and CNTL parameters for MUMPS. I was wondering if we can still use BLR approximation for a preconditioner for Krylov solvers. Best, Evan On Sat, Sep 23, 2017 at 6:45 PM, Barry Smith wrote: > > > On Sep 23, 2017, at 8:38 PM, Evan Um wrote: > > > > Dear PETSC Users, > > > > My system matrix comes from finite element modeling and is complex and > unstructured. Its typical size is a few millions-by a few millions. I > wondering if I can use MUMPS parallel direct solver as a preconditioner in > PETSC. For example, I want to pass factored matrices to Krylov iterative > solvers such as QMR. Is there any PETSC+MUMPS example code for the purpose? > > You don't pass factored matrices you just pass the original matrix and > use -pc_type lu -pc_factor_mat_solver_package mumps > > > Can PETSC call the latest MUMPS that supports block low rank > approximation? > > No, send us info on it and we'll see if we can add an interface > > > > > > In advance, thank you very much for your comments. > > > > Best, > > Evan > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Sun Sep 24 20:08:05 2017 From: hzhang at mcs.anl.gov (Hong) Date: Sun, 24 Sep 2017 20:08:05 -0500 Subject: [petsc-users] Using factored complex matrices from MUMPS as a preconditioner in PETSC In-Reply-To: References: Message-ID: I'll check it. Hong On Sun, Sep 24, 2017 at 3:42 PM, Evan Um wrote: > Hi Barry, > > Thanks for your comments. To activate block low rank (BLR) approximation > in MUMPS version 5.1.1, a user needs to turn on the functionality (i.e. > ICNTL(35)=1) and specify the tolerance value (e.g. CNTL(7)=1e-4). In PETSC, > I think that we can set up ICNTL and CNTL parameters for MUMPS. I was > wondering if we can still use BLR approximation for a preconditioner for > Krylov solvers. > > Best, > Evan > > > On Sat, Sep 23, 2017 at 6:45 PM, Barry Smith wrote: > >> >> > On Sep 23, 2017, at 8:38 PM, Evan Um wrote: >> > >> > Dear PETSC Users, >> > >> > My system matrix comes from finite element modeling and is complex and >> unstructured. Its typical size is a few millions-by a few millions. I >> wondering if I can use MUMPS parallel direct solver as a preconditioner in >> PETSC. For example, I want to pass factored matrices to Krylov iterative >> solvers such as QMR. Is there any PETSC+MUMPS example code for the purpose? >> >> You don't pass factored matrices you just pass the original matrix and >> use -pc_type lu -pc_factor_mat_solver_package mumps >> >> > Can PETSC call the latest MUMPS that supports block low rank >> approximation? >> >> No, send us info on it and we'll see if we can add an interface >> >> >> > >> > In advance, thank you very much for your comments. >> > >> > Best, >> > Evan >> > >> > >> > >> > >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregory.meyer at gmail.com Mon Sep 25 00:17:53 2017 From: gregory.meyer at gmail.com (Greg Meyer) Date: Mon, 25 Sep 2017 05:17:53 +0000 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: Message-ID: Hi all, Hong--looking at your link, there may be no special algorithm for Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like it would any matrix. Furthermore it appears that Cholesky of complex matrices is supported from this link: https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html So do you or anyone have any idea why I get incorrect eigenvalues? Thanks, Greg On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer wrote: > Ok, thanks. It seems that PETSc clearly should throw an error in this case > instead of just giving incorrect answers? I am surprised that it does not > throw an error... > > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: > >> Greg : >> >>> Yes, they are Hermitian. >>> >> >> PETSc does not support Cholesky factorization for Hermitian. >> It seems mumps does not support Hermitian either >> >> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-November/027541.html >> >> Hong >> >> >>> On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: >>> >>>> Greg: >>>> >>>> OK, the difference is whether LU or Cholesky factorization is used. But >>>>> I would hope that neither one should give incorrect eigenvalues, and when I >>>>> run with the latter it does! >>>>> >>>> Are your matrices symmetric/Hermitian? >>>> Hong >>>> >>>>> >>>>> On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: >>>>> >>>>>> Gregory : >>>>>> Use '-eps_view' for both runs to check the algorithms being used. >>>>>> Hong >>>>>> >>>>>> Hi all, >>>>>>> >>>>>>> I'm using shift-invert with EPS to solve for eigenvalues. I find >>>>>>> that if I do only >>>>>>> >>>>>>> ... >>>>>>> ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>>>>>> ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>>>>>> ... >>>>>>> >>>>>>> in my code I get correct eigenvalues. But if I do >>>>>>> >>>>>>> ... >>>>>>> ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>>>>>> ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>>>>>> ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); >>>>>>> ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >>>>>>> ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); >>>>>>> ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); >>>>>>> ... >>>>>>> >>>>>>> the eigenvalues found by EPS are completely wrong! Somehow I thought >>>>>>> I was supposed to do the latter, from the examples etc, but I guess that >>>>>>> was not correct? I attach the full piece of test code and a test matrix. >>>>>>> >>>>>>> Best, >>>>>>> Greg >>>>>>> >>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From pietro.benedusi at usi.ch Mon Sep 25 03:26:37 2017 From: pietro.benedusi at usi.ch (Benedusi Pietro) Date: Mon, 25 Sep 2017 08:26:37 +0000 Subject: [petsc-users] Access to submatrix owned by an other processor Message-ID: Dear all, there is a way to access an arbitrary block of a parallel matrix? It should work also in the case where the block is owned by an other processor. I have seen that you can use MatCreateRedundantMatrix() to copy the entire matrix in a processor, but this solution is not very efficient if I just need a (small) block of it. I have also tried to use MatGetSubMatrix() but does not seem to work. Thank you very much for any help, Pietro -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Sep 25 03:46:37 2017 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 25 Sep 2017 10:46:37 +0200 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: Message-ID: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> Greg, The linear solver table probably needs to be updated. I have tried several Cholesky solvers. With mkl_pardiso I get an explicit error message that it does not support Cholesky with complex scalars. The rest (PETSc, MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is not related to your matrix, nor to shift-and-invert in SLEPc. I tried with ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works in complex scalars, but the matrix is real. As soon as you add complex entries Cholesky fails, for instance adding this: ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); I don't know if it is a bug or that the algorithm cannot support complex Hermitian matrices. Maybe Hong can confirm any of these. In the latter case, I agree that all packages should give an error message, as mkl_pardiso does. As a side comment, I would suggest using LU instead of Cholesky. Cholesky performs less flops but it does not mean it will be faster - I have seen many cases where it is slower than LU, maybe because in shift-and-invert computations the matrix is often indefinite, so an indefinite factorization is computed rather than Cholesky. Some SLEPc eigensolvers (e.g. LOBPCG) require that the preconditioner is symmetric (Hermitian), but the default solver (Krylov-Schur) is quite robust when you use LU instead of Cholesky in Hermitian problems. And you can always solve the problem as non-Hermitian (the difference in accuracy should not be too noticeable). Jose > El 25 sept 2017, a las 7:17, Greg Meyer escribi?: > > Hi all, > > Hong--looking at your link, there may be no special algorithm for Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like it would any matrix. Furthermore it appears that Cholesky of complex matrices is supported from this link: https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html > > So do you or anyone have any idea why I get incorrect eigenvalues? > > Thanks, > Greg > > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer wrote: > Ok, thanks. It seems that PETSc clearly should throw an error in this case instead of just giving incorrect answers? I am surprised that it does not throw an error... > > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: > Greg : > Yes, they are Hermitian. > > PETSc does not support Cholesky factorization for Hermitian. > It seems mumps does not support Hermitian either > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-November/027541.html > > Hong > > > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: > Greg: > > OK, the difference is whether LU or Cholesky factorization is used. But I would hope that neither one should give incorrect eigenvalues, and when I run with the latter it does! > Are your matrices symmetric/Hermitian? > Hong > > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: > Gregory : > Use '-eps_view' for both runs to check the algorithms being used. > Hong > > Hi all, > > I'm using shift-invert with EPS to solve for eigenvalues. I find that if I do only > > ... > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > ... > > in my code I get correct eigenvalues. But if I do > > ... > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); > ... > > the eigenvalues found by EPS are completely wrong! Somehow I thought I was supposed to do the latter, from the examples etc, but I guess that was not correct? I attach the full piece of test code and a test matrix. > > Best, > Greg > From knepley at gmail.com Mon Sep 25 05:15:31 2017 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 25 Sep 2017 06:15:31 -0400 Subject: [petsc-users] Access to submatrix owned by an other processor In-Reply-To: References: Message-ID: On Mon, Sep 25, 2017 at 4:26 AM, Benedusi Pietro wrote: > Dear all, > > there is a way to access an arbitrary block of a parallel matrix? It > should work also in the case where the block is owned by an other processor. > > I have seen that you can use MatCreateRedundantMatrix() to copy the entire > matrix in a processor, but this solution is not very efficient if I just > need a (small) block of it. > I have also tried to use MatGetSubMatrix() but does not seem to work. > Why does MatGetSubMatrix() not work? Thanks, Matt > Thank you very much for any help, > Pietro > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Mon Sep 25 07:32:30 2017 From: cpraveen at gmail.com (Praveen C) Date: Mon, 25 Sep 2017 18:02:30 +0530 Subject: [petsc-users] SNES first residual Message-ID: Dear all Is there a way to get the residual in first step of SNES ? Thanks praveen -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 25 08:01:54 2017 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 25 Sep 2017 09:01:54 -0400 Subject: [petsc-users] SNES first residual In-Reply-To: References: Message-ID: Does -snes_monitor not give that? Thanks, Matt On Mon, Sep 25, 2017 at 8:32 AM, Praveen C wrote: > Dear all > > Is there a way to get the residual in first step of SNES ? > > Thanks > praveen > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Mon Sep 25 08:17:21 2017 From: cpraveen at gmail.com (Praveen C) Date: Mon, 25 Sep 2017 18:47:21 +0530 Subject: [petsc-users] SNES first residual In-Reply-To: References: Message-ID: Hello Matt I want to get the first residual norm within my code to decide on convergence. I am solving a steady state problem with pseudo-timestepping and SNES. Currently I am calling my FormFunction before call to SNES and using VecNorm. while(res_norm > res_tol) { uold = u; FormFunction(snes,u,r,&ctx); VecNorm(u,NORM_2,&res_norm); SNESSolve(snes,NULL,u); } But this leads to a duplication of function evaluation. What I want is something like this while(res_norm > res_tol) { uold = u; SNESSolve(snes,NULL,u); // get res_norm = first residual of SNES } Best praveen On Mon, Sep 25, 2017 at 6:31 PM, Matthew Knepley wrote: > Does -snes_monitor not give that? > > Thanks, > > Matt > > On Mon, Sep 25, 2017 at 8:32 AM, Praveen C wrote: > >> Dear all >> >> Is there a way to get the residual in first step of SNES ? >> >> Thanks >> praveen >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 25 08:19:04 2017 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 25 Sep 2017 09:19:04 -0400 Subject: [petsc-users] SNES first residual In-Reply-To: References: Message-ID: If you want to control SNES convergence, you should use your own convergence check http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetConvergenceTest.html Matt On Mon, Sep 25, 2017 at 9:17 AM, Praveen C wrote: > Hello Matt > > I want to get the first residual norm within my code to decide on > convergence. > > I am solving a steady state problem with pseudo-timestepping and SNES. > > Currently I am calling my FormFunction before call to SNES and using > VecNorm. > > while(res_norm > res_tol) > { > uold = u; > FormFunction(snes,u,r,&ctx); > VecNorm(u,NORM_2,&res_norm); > SNESSolve(snes,NULL,u); > } > > But this leads to a duplication of function evaluation. > > What I want is something like this > > while(res_norm > res_tol) > { > uold = u; > SNESSolve(snes,NULL,u); > // get res_norm = first residual of SNES > } > > Best > praveen > > On Mon, Sep 25, 2017 at 6:31 PM, Matthew Knepley > wrote: > >> Does -snes_monitor not give that? >> >> Thanks, >> >> Matt >> >> On Mon, Sep 25, 2017 at 8:32 AM, Praveen C wrote: >> >>> Dear all >>> >>> Is there a way to get the residual in first step of SNES ? >>> >>> Thanks >>> praveen >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Sep 25 09:41:35 2017 From: hzhang at mcs.anl.gov (Hong) Date: Mon, 25 Sep 2017 09:41:35 -0500 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> Message-ID: Greg and Jose, I'll check this case, at least have petsc provide error message. Please give me some time because I'm in the middle of several tasks. I'll let you know once I add this support. Hong On Mon, Sep 25, 2017 at 3:46 AM, Jose E. Roman wrote: > Greg, > > The linear solver table probably needs to be updated. I have tried several > Cholesky solvers. With mkl_pardiso I get an explicit error message that it > does not support Cholesky with complex scalars. The rest (PETSc, MUMPS, > CHOLMOD) give a wrong answer (without error message). The problem is not > related to your matrix, nor to shift-and-invert in SLEPc. I tried with > ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works > in complex scalars, but the matrix is real. As soon as you add complex > entries Cholesky fails, for instance adding this: > ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > I don't know if it is a bug or that the algorithm cannot support complex > Hermitian matrices. Maybe Hong can confirm any of these. In the latter > case, I agree that all packages should give an error message, as > mkl_pardiso does. > > As a side comment, I would suggest using LU instead of Cholesky. Cholesky > performs less flops but it does not mean it will be faster - I have seen > many cases where it is slower than LU, maybe because in shift-and-invert > computations the matrix is often indefinite, so an indefinite factorization > is computed rather than Cholesky. Some SLEPc eigensolvers (e.g. LOBPCG) > require that the preconditioner is symmetric (Hermitian), but the default > solver (Krylov-Schur) is quite robust when you use LU instead of Cholesky > in Hermitian problems. And you can always solve the problem as > non-Hermitian (the difference in accuracy should not be too noticeable). > > Jose > > > > El 25 sept 2017, a las 7:17, Greg Meyer > escribi?: > > > > Hi all, > > > > Hong--looking at your link, there may be no special algorithm for > Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like > it would any matrix. Furthermore it appears that Cholesky of complex > matrices is supported from this link: https://www.mcs.anl.gov/petsc/ > documentation/linearsolvertable.html > > > > So do you or anyone have any idea why I get incorrect eigenvalues? > > > > Thanks, > > Greg > > > > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer > wrote: > > Ok, thanks. It seems that PETSc clearly should throw an error in this > case instead of just giving incorrect answers? I am surprised that it does > not throw an error... > > > > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: > > Greg : > > Yes, they are Hermitian. > > > > PETSc does not support Cholesky factorization for Hermitian. > > It seems mumps does not support Hermitian either > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/ > 2015-November/027541.html > > > > Hong > > > > > > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: > > Greg: > > > > OK, the difference is whether LU or Cholesky factorization is used. But > I would hope that neither one should give incorrect eigenvalues, and when I > run with the latter it does! > > Are your matrices symmetric/Hermitian? > > Hong > > > > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: > > Gregory : > > Use '-eps_view' for both runs to check the algorithms being used. > > Hong > > > > Hi all, > > > > I'm using shift-invert with EPS to solve for eigenvalues. I find that if > I do only > > > > ... > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > ... > > > > in my code I get correct eigenvalues. But if I do > > > > ... > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); > > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); > > ... > > > > the eigenvalues found by EPS are completely wrong! Somehow I thought I > was supposed to do the latter, from the examples etc, but I guess that was > not correct? I attach the full piece of test code and a test matrix. > > > > Best, > > Greg > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregory.meyer at gmail.com Mon Sep 25 10:20:38 2017 From: gregory.meyer at gmail.com (Greg Meyer) Date: Mon, 25 Sep 2017 15:20:38 +0000 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> Message-ID: Hi Jose and Hong, Thanks so much for the detailed response. I am glad to know the reason behind it--hopefully we eventually figure out why the solvers have this behavior! Hong, I really appreciate you working on a patch to throw an error in this case. It definitely bit me and some people using my code... Hopefully it doesn't happen to anyone else! :) Best, Greg On Mon, Sep 25, 2017, 7:44 AM Hong wrote: > Greg and Jose, > I'll check this case, at least have petsc provide error message. > Please give me some time because I'm in the middle of several tasks. > I'll let you know once I add this support. > > Hong > > > On Mon, Sep 25, 2017 at 3:46 AM, Jose E. Roman wrote: > >> Greg, >> >> The linear solver table probably needs to be updated. I have tried >> several Cholesky solvers. With mkl_pardiso I get an explicit error message >> that it does not support Cholesky with complex scalars. The rest (PETSc, >> MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is >> not related to your matrix, nor to shift-and-invert in SLEPc. I tried with >> ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works in >> complex scalars, but the matrix is real. As soon as you add complex entries >> Cholesky fails, for instance adding this: >> ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); >> ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); >> >> I don't know if it is a bug or that the algorithm cannot support complex >> Hermitian matrices. Maybe Hong can confirm any of these. In the latter >> case, I agree that all packages should give an error message, as >> mkl_pardiso does. >> >> As a side comment, I would suggest using LU instead of Cholesky. Cholesky >> performs less flops but it does not mean it will be faster - I have seen >> many cases where it is slower than LU, maybe because in shift-and-invert >> computations the matrix is often indefinite, so an indefinite factorization >> is computed rather than Cholesky. Some SLEPc eigensolvers (e.g. LOBPCG) >> require that the preconditioner is symmetric (Hermitian), but the default >> solver (Krylov-Schur) is quite robust when you use LU instead of Cholesky >> in Hermitian problems. And you can always solve the problem as >> non-Hermitian (the difference in accuracy should not be too noticeable). >> >> Jose >> >> >> > El 25 sept 2017, a las 7:17, Greg Meyer >> escribi?: >> > >> > Hi all, >> > >> > Hong--looking at your link, there may be no special algorithm for >> Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like >> it would any matrix. Furthermore it appears that Cholesky of complex >> matrices is supported from this link: >> https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html >> > >> > So do you or anyone have any idea why I get incorrect eigenvalues? >> > >> > Thanks, >> > Greg >> > >> > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer >> wrote: >> > Ok, thanks. It seems that PETSc clearly should throw an error in this >> case instead of just giving incorrect answers? I am surprised that it does >> not throw an error... >> > >> > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: >> > Greg : >> > Yes, they are Hermitian. >> > >> > PETSc does not support Cholesky factorization for Hermitian. >> > It seems mumps does not support Hermitian either >> > >> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-November/027541.html >> > >> > Hong >> > >> > >> > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: >> > Greg: >> > >> > OK, the difference is whether LU or Cholesky factorization is used. But >> I would hope that neither one should give incorrect eigenvalues, and when I >> run with the latter it does! >> > Are your matrices symmetric/Hermitian? >> > Hong >> > >> > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: >> > Gregory : >> > Use '-eps_view' for both runs to check the algorithms being used. >> > Hong >> > >> > Hi all, >> > >> > I'm using shift-invert with EPS to solve for eigenvalues. I find that >> if I do only >> > >> > ... >> > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >> > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >> > ... >> > >> > in my code I get correct eigenvalues. But if I do >> > >> > ... >> > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >> > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >> > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); >> > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >> > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); >> > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); >> > ... >> > >> > the eigenvalues found by EPS are completely wrong! Somehow I thought I >> was supposed to do the latter, from the examples etc, but I guess that was >> not correct? I attach the full piece of test code and a test matrix. >> > >> > Best, >> > Greg >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Tue Sep 26 07:28:46 2017 From: cpraveen at gmail.com (Praveen C) Date: Tue, 26 Sep 2017 17:58:46 +0530 Subject: [petsc-users] matrix-free snes with frozen preconditioner Message-ID: Dear all I am solving a nonlinear problem using matrix-free approach in snes and I give a preconditioner matrix via SNESSetJacobian. My preconditioner is based on a first order scheme which is linear and hence preconditioner matrix will not change. How do I tell snes not to recompute it ? Thanks praveen -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 26 07:30:49 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 26 Sep 2017 08:30:49 -0400 Subject: [petsc-users] matrix-free snes with frozen preconditioner In-Reply-To: References: Message-ID: On Tue, Sep 26, 2017 at 8:28 AM, Praveen C wrote: > Dear all > > I am solving a nonlinear problem using matrix-free approach in snes and I > give a preconditioner matrix via SNESSetJacobian. My preconditioner is > based on a first order scheme which is linear and hence preconditioner > matrix will not change. How do I tell snes not to recompute it ? > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetLagPreconditioner.html Matt > Thanks > praveen > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Tue Sep 26 07:59:20 2017 From: cpraveen at gmail.com (Praveen C) Date: Tue, 26 Sep 2017 18:29:20 +0530 Subject: [petsc-users] matrix-free snes with frozen preconditioner In-Reply-To: References: Message-ID: Thank you. I had to do this ierr = SNESSetLagJacobian(snes,-2); CHKERRQ(ierr); ierr = SNESSetLagPreconditioner(snes,-2); CHKERRQ(ierr); Now I see that my FormJacobian function is called only once. Is this correct way to use it ? Best praveen On Tue, Sep 26, 2017 at 6:00 PM, Matthew Knepley wrote: > On Tue, Sep 26, 2017 at 8:28 AM, Praveen C wrote: > >> Dear all >> >> I am solving a nonlinear problem using matrix-free approach in snes and I >> give a preconditioner matrix via SNESSetJacobian. My preconditioner is >> based on a first order scheme which is linear and hence preconditioner >> matrix will not change. How do I tell snes not to recompute it ? >> > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/ > SNESSetLagPreconditioner.html > > Matt > > >> Thanks >> praveen >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 26 08:11:01 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 26 Sep 2017 09:11:01 -0400 Subject: [petsc-users] matrix-free snes with frozen preconditioner In-Reply-To: References: Message-ID: On Tue, Sep 26, 2017 at 8:59 AM, Praveen C wrote: > Thank you. I had to do this > > ierr = SNESSetLagJacobian(snes,-2); CHKERRQ(ierr); > ierr = SNESSetLagPreconditioner(snes,-2); CHKERRQ(ierr); > > Now I see that my FormJacobian function is called only once. Is this > correct way to use it ? > Yes Matt > Best > praveen > > On Tue, Sep 26, 2017 at 6:00 PM, Matthew Knepley > wrote: > >> On Tue, Sep 26, 2017 at 8:28 AM, Praveen C wrote: >> >>> Dear all >>> >>> I am solving a nonlinear problem using matrix-free approach in snes and >>> I give a preconditioner matrix via SNESSetJacobian. My preconditioner is >>> based on a first order scheme which is linear and hence preconditioner >>> matrix will not change. How do I tell snes not to recompute it ? >>> >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/ >> SNES/SNESSetLagPreconditioner.html >> >> Matt >> >> >>> Thanks >>> praveen >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From aliberkkahraman at yahoo.com Tue Sep 26 17:33:28 2017 From: aliberkkahraman at yahoo.com (Ali Berk Kahraman) Date: Wed, 27 Sep 2017 01:33:28 +0300 Subject: [petsc-users] Creating a Matrix Using Inner Products of Columns and Rows of Two matrices Message-ID: Hello all, I have to generate a matrix C such that ,Ck,m=?pAk,pDp,mC_{k,m}=\sum_{p}A_{k,p}D_{p,m} where A and D are two square matrices of the same size. This is simply the inner products of rows of A and columns of D. I do not want to use MatGetValues to get values form the matrices, then create Vectors with those values and use dot product function of petsc, because that would be super non-efficient. What do you think? How can it be done more efficiently? Best Regards, Ali Berk Kahraman CFD-FMS Research Group, Mech. Eng. Department Bo?azi?i University, Istanbul, Turkey -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Sep 26 17:58:55 2017 From: jed at jedbrown.org (Jed Brown) Date: Tue, 26 Sep 2017 16:58:55 -0600 Subject: [petsc-users] Creating a Matrix Using Inner Products of Columns and Rows of Two matrices In-Reply-To: References: Message-ID: <87zi9h83tc.fsf@jedbrown.org> Why is this not MatMatMult()? Ali Berk Kahraman writes: > Hello all, > > I have to generate a matrix C such that > ,Ck,m=?pAk,pDp,mC_{k,m}=\sum_{p}A_{k,p}D_{p,m} where A and D are two > square matrices of the same size. This is simply the inner products of > rows of A and columns of D. I do not want to use MatGetValues to get > values form the matrices, then create Vectors with those values and use > dot product function of petsc, because that would be super non-efficient. > > What do you think? How can it be done more efficiently? > > > Best Regards, > > > Ali Berk Kahraman > > CFD-FMS Research Group, Mech. Eng. Department > > Bo?azi?i University, Istanbul, Turkey From aliberkkahraman at yahoo.com Wed Sep 27 01:39:17 2017 From: aliberkkahraman at yahoo.com (Ali Berk Kahraman) Date: Wed, 27 Sep 2017 09:39:17 +0300 Subject: [petsc-users] Creating a Matrix Using Inner Products of Columns and Rows of Two matrices In-Reply-To: <87zi9h83tc.fsf@jedbrown.org> References: <87zi9h83tc.fsf@jedbrown.org> Message-ID: Well, this is embarrasing. Yes, it is. I do not know how I did not see that. I guess the sum sign distracted me. Problem is solved. Thank you for the answer. Ali On 27-09-2017 01:58, Jed Brown wrote: > Why is this not MatMatMult()? > > Ali Berk Kahraman writes: > >> Hello all, >> >> I have to generate a matrix C such that >> ,Ck,m=?pAk,pDp,mC_{k,m}=\sum_{p}A_{k,p}D_{p,m} where A and D are two >> square matrices of the same size. This is simply the inner products of >> rows of A and columns of D. I do not want to use MatGetValues to get >> values form the matrices, then create Vectors with those values and use >> dot product function of petsc, because that would be super non-efficient. >> >> What do you think? How can it be done more efficiently? >> >> >> Best Regards, >> >> >> Ali Berk Kahraman >> >> CFD-FMS Research Group, Mech. Eng. Department >> >> Bo?azi?i University, Istanbul, Turkey From k_burkart at yahoo.com Thu Sep 28 14:43:38 2017 From: k_burkart at yahoo.com (Klaus Burkart) Date: Thu, 28 Sep 2017 19:43:38 +0000 (UTC) Subject: [petsc-users] How to copy vectors to/from petsc? References: <369401411.97459.1506627818824.ref@mail.yahoo.com> Message-ID: <369401411.97459.1506627818824@mail.yahoo.com> Hello, I want to solve a physics problem with support of petsc i.e. solve the linear system using petsc. So far I have been using another linear algebra library. I am using c++. Now I am struggling to load the rhs vector/array from the physics application to petsc and copying the result x back to the physics application. I found information about how to load data from binaries or from files but both is not the case here. The computation will take place on a multi core CPU using MPI. This is how far I am and includes my questions: ... #include // vector x, rhs size n; "matrix" is the matrix from the physics application PetscMPIInt n = matrix.diag().size(); Vec??????????? rhs; // Question 1: How to set the value type to double? VecSetSizes(rhs, n, PETSC_DECIDE); // Question 2: What is "global size"? Vec??????????? x; VecSetSizes(x, n, PETSC_DECIDE); // Question 3: // How to load petsc rhs vector with data from vector (= pointer array) named rhs_source? // Current load/copy syntax is:??? fast_copy(rhs_source.begin(), rhs_source.end(), rhs.begin()); // Question 4: // How to export petsc result vector x i.e. load the result x into array named x_result? // Current load/copy syntax is:??? fast_copy(x.begin(), x.end(), x_result.begin()); // destroy petsc vectors VecDestroy(&rhs); VecDestroy(&x); PetscFinalize(); // What's the best way to copy rhs to petsc and to copy the result x back to the physics application? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rupp at iue.tuwien.ac.at Thu Sep 28 15:46:45 2017 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Thu, 28 Sep 2017 15:46:45 -0500 Subject: [petsc-users] How to copy vectors to/from petsc? In-Reply-To: <369401411.97459.1506627818824@mail.yahoo.com> References: <369401411.97459.1506627818824.ref@mail.yahoo.com> <369401411.97459.1506627818824@mail.yahoo.com> Message-ID: <5ebe2b61-6655-079b-bea3-32480eaa9c84@iue.tuwien.ac.at> Hi Klaus, there are several options, e.g.: * http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecCreateSeqWithArray.html (sequential) * http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecSetValues.html (use with multiple MPI ranks if you cannot easily determine the local parts) * http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray.html (get a pointer to the data and assign values yourself) Best regards, Karli On 09/28/2017 02:43 PM, Klaus Burkart wrote: > Hello, > > I want to solve a physics problem with support of petsc i.e. solve the > linear system using petsc. So far I have been using another linear > algebra library. I am using c++. Now I am struggling to load the rhs > vector/array from the physics application to petsc and copying the > result x back to the physics application. I found information about how > to load data from binaries or from files but both is not the case here. > The computation will take place on a multi core CPU using MPI. > > This is how far I am and includes my questions: > > ... > #include > > > // vector x, rhs size n; "matrix" is the matrix from the physics application > PetscMPIInt n = matrix.diag().size(); > > Vec??????????? rhs; // Question 1: How to set the value type to double? > VecSetSizes(rhs, n, PETSC_DECIDE); // Question 2: What is "global size"? > > Vec??????????? x; > VecSetSizes(x, n, PETSC_DECIDE); > > > > > // Question 3: > // How to load petsc rhs vector with data from vector (= pointer array) > named rhs_source? > // Current load/copy syntax is:??? fast_copy(rhs_source.begin(), > rhs_source.end(), rhs.begin()); > > > > // Question 4: > // How to export petsc result vector x i.e. load the result x into array > named x_result? > // Current load/copy syntax is:??? fast_copy(x.begin(), x.end(), > x_result.begin()); > > > > // destroy petsc vectors > VecDestroy(&rhs); > VecDestroy(&x); > PetscFinalize(); > > // What's the best way to copy rhs to petsc and to copy the result x > back to the physics application? > From rachitp at vt.edu Thu Sep 28 15:53:41 2017 From: rachitp at vt.edu (Rachit Prasad) Date: Thu, 28 Sep 2017 16:53:41 -0400 Subject: [petsc-users] Using higher order ILU preconditioner Message-ID: Hi, I am trying to solve a highly ill-conditioned matrix and am unable to get convergence when using ILU(0) or Jacobi and SOR preconditioners. I tried to implement a higher order ILU by increasing the level of fill-in to 1 by calling call PCFactorSetLevels(pc,1,ierr) However, when I do that I get the following memory error: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Out of memory. This could be due to allocating [0]PETSC ERROR: too large an object or bleeding by not properly [0]PETSC ERROR: destroying unneeded objects. [0]PETSC ERROR: Memory allocated -2147483648 Memory used by process -2147483648 [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [0]PETSC ERROR: Memory requested 18446744071863943168! While I do understand that increasing the level of fill-in will require higher memory, the memory requested seems to be way too high. Am I implementing the higher order ILU correctly? Is there any other subroutine which I need to call? Regards, Rachit -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Sep 28 15:56:53 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 28 Sep 2017 15:56:53 -0500 Subject: [petsc-users] How to copy vectors to/from petsc? In-Reply-To: <5ebe2b61-6655-079b-bea3-32480eaa9c84@iue.tuwien.ac.at> References: <369401411.97459.1506627818824.ref@mail.yahoo.com> <369401411.97459.1506627818824@mail.yahoo.com> <5ebe2b61-6655-079b-bea3-32480eaa9c84@iue.tuwien.ac.at> Message-ID: On Thu, 28 Sep 2017, Karl Rupp wrote: > Hi Klaus, > > there are several options, e.g.: > * > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecCreateSeqWithArray.html > (sequential) For parallel http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecCreateMPIWithArray.html > * > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecSetValues.html > (use with multiple MPI ranks if you cannot easily determine the local parts) > * > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray.html > (get a pointer to the data and assign values yourself) > > Best regards, > Karli > > > On 09/28/2017 02:43 PM, Klaus Burkart wrote: > > Hello, > > > > I want to solve a physics problem with support of petsc i.e. solve the > > linear system using petsc. So far I have been using another linear algebra > > library. I am using c++. Now I am struggling to load the rhs vector/array > > from the physics application to petsc and copying the result x back to the > > physics application. I found information about how to load data from > > binaries or from files but both is not the case here. The computation will > > take place on a multi core CPU using MPI. > > > > This is how far I am and includes my questions: > > > > ... > > #include > > > > > > // vector x, rhs size n; "matrix" is the matrix from the physics application > > PetscMPIInt n = matrix.diag().size(); > > > > Vec??????????? rhs; // Question 1: How to set the value type to double? PETSc provides configure options --with-precision=single,double,.. --with-scalar-type=real,complex We default to double precision real values. The corresponding datatype is PetscScalar > > VecSetSizes(rhs, n, PETSC_DECIDE); // Question 2: What is "global size"? Sum of local sizes - when running parallely. Satish > > > > Vec??????????? x; > > VecSetSizes(x, n, PETSC_DECIDE); > > > > > > > > > > // Question 3: > > // How to load petsc rhs vector with data from vector (= pointer array) > > named rhs_source? > > // Current load/copy syntax is:??? fast_copy(rhs_source.begin(), > > rhs_source.end(), rhs.begin()); > > > > > > > > // Question 4: > > // How to export petsc result vector x i.e. load the result x into array > > named x_result? > > // Current load/copy syntax is:??? fast_copy(x.begin(), x.end(), > > x_result.begin()); > > > > > > > > // destroy petsc vectors > > VecDestroy(&rhs); > > VecDestroy(&x); > > PetscFinalize(); > > > > // What's the best way to copy rhs to petsc and to copy the result x back to > > the physics application? > > > From fande.kong at inl.gov Thu Sep 28 16:13:32 2017 From: fande.kong at inl.gov (Kong, Fande) Date: Thu, 28 Sep 2017 15:13:32 -0600 Subject: [petsc-users] Using higher order ILU preconditioner In-Reply-To: References: Message-ID: Calling PCFactorSetMatOrderingType() (or command line option: -pc_factor_mat_ordering_type) to use RCM or 1WD usually helps me a lot. My application is based on highly-unstructured meshes. Fande, On Thu, Sep 28, 2017 at 2:53 PM, Rachit Prasad wrote: > Hi, > > I am trying to solve a highly ill-conditioned matrix and am unable to get > convergence when using ILU(0) or Jacobi and SOR preconditioners. I tried to > implement a higher order ILU by increasing the level of fill-in to 1 by > calling > > call PCFactorSetLevels(pc,1,ierr) > > However, when I do that I get the following memory error: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Out of memory. This could be due to allocating > [0]PETSC ERROR: too large an object or bleeding by not properly > [0]PETSC ERROR: destroying unneeded objects. > [0]PETSC ERROR: Memory allocated -2147483648 Memory used by process > -2147483648 > [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. > [0]PETSC ERROR: Memory requested 18446744071863943168! > > While I do understand that increasing the level of fill-in will require > higher memory, the memory requested seems to be way too high. Am I > implementing the higher order ILU correctly? Is there any other subroutine > which I need to call? > > Regards, > Rachit > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rachitp at vt.edu Thu Sep 28 16:27:09 2017 From: rachitp at vt.edu (Rachit Prasad) Date: Thu, 28 Sep 2017 17:27:09 -0400 Subject: [petsc-users] Using higher order ILU preconditioner In-Reply-To: References: Message-ID: I guess I missed out an important piece of information, due to legacy code issues, I am working with Petsc version 2.3.2. Unfortunately, it doesn't have PCFactorSetMatOrderingType(). Thanks for the advice though. I still don't understand, why would it ask for so much memory, it is almost 16 Petabytes! Any other advice would be greatly appreciated. Regards, Rachit Prasad On Thu, Sep 28, 2017 at 5:13 PM, Kong, Fande wrote: > Calling PCFactorSetMatOrderingType() (or command line option: > -pc_factor_mat_ordering_type) to use RCM or 1WD usually helps me a lot. My > application is based on highly-unstructured meshes. > > Fande, > > On Thu, Sep 28, 2017 at 2:53 PM, Rachit Prasad wrote: > >> Hi, >> >> I am trying to solve a highly ill-conditioned matrix and am unable to get >> convergence when using ILU(0) or Jacobi and SOR preconditioners. I tried to >> implement a higher order ILU by increasing the level of fill-in to 1 by >> calling >> >> call PCFactorSetLevels(pc,1,ierr) >> >> However, when I do that I get the following memory error: >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Out of memory. This could be due to allocating >> [0]PETSC ERROR: too large an object or bleeding by not properly >> [0]PETSC ERROR: destroying unneeded objects. >> [0]PETSC ERROR: Memory allocated -2147483648 <(214)%20748-3648> Memory >> used by process -2147483648 <(214)%20748-3648> >> [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. >> [0]PETSC ERROR: Memory requested 18446744071863943168! >> >> While I do understand that increasing the level of fill-in will require >> higher memory, the memory requested seems to be way too high. Am I >> implementing the higher order ILU correctly? Is there any other subroutine >> which I need to call? >> >> Regards, >> Rachit >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Sep 28 19:26:23 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 28 Sep 2017 17:26:23 -0700 Subject: [petsc-users] Using higher order ILU preconditioner In-Reply-To: References: Message-ID: <98A3B775-CF9F-4DE7-A244-F7524396AE94@mcs.anl.gov> > On Sep 28, 2017, at 2:27 PM, Rachit Prasad wrote: > > I guess I missed out an important piece of information, due to legacy code issues, I am working with Petsc version 2.3.2. Unfortunately, it doesn't have PCFactorSetMatOrderingType(). Thanks for the advice though. I still don't understand, why would it ask for so much memory, it is almost 16 Petabytes! Any other advice would be greatly appreciated. You really, really, really need to change ASAP to the latest PETSc release. That is the only advice I can provide. > > Regards, > Rachit Prasad > > On Thu, Sep 28, 2017 at 5:13 PM, Kong, Fande wrote: > Calling PCFactorSetMatOrderingType() (or command line option: -pc_factor_mat_ordering_type) to use RCM or 1WD usually helps me a lot. My application is based on highly-unstructured meshes. > > Fande, > > On Thu, Sep 28, 2017 at 2:53 PM, Rachit Prasad wrote: > Hi, > > I am trying to solve a highly ill-conditioned matrix and am unable to get convergence when using ILU(0) or Jacobi and SOR preconditioners. I tried to implement a higher order ILU by increasing the level of fill-in to 1 by calling > > call PCFactorSetLevels(pc,1,ierr) > > However, when I do that I get the following memory error: > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Out of memory. This could be due to allocating > [0]PETSC ERROR: too large an object or bleeding by not properly > [0]PETSC ERROR: destroying unneeded objects. > [0]PETSC ERROR: Memory allocated -2147483648 Memory used by process -2147483648 > [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. > [0]PETSC ERROR: Memory requested 18446744071863943168! > > While I do understand that increasing the level of fill-in will require higher memory, the memory requested seems to be way too high. Am I implementing the higher order ILU correctly? Is there any other subroutine which I need to call? > > Regards, > Rachit > > > From hzhang at mcs.anl.gov Thu Sep 28 21:28:04 2017 From: hzhang at mcs.anl.gov (Hong) Date: Thu, 28 Sep 2017 21:28:04 -0500 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> Message-ID: Greg: > Thanks so much for the detailed response. I am glad to know the reason > behind it--hopefully we eventually figure out why the solvers have this > behavior! Hong, I really appreciate you working on a patch to throw an > error in this case. It definitely bit me and some people using my code... > Hopefully it doesn't happen to anyone else! :) > I added an error flag for using MUMPS Cholesky factorization on Hermitian matrix. See branch hzhang/mumps-HermitianCholesky-errflag https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7 Jose, PETSc does not support Cholesky for Hermitian matrix. > >>> The linear solver table probably needs to be updated. I have tried >>> several Cholesky solvers. With mkl_pardiso I get an explicit error message >>> that it does not support Cholesky with complex scalars. The rest (PETSc, >>> MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is >>> not related to your matrix, nor to shift-and-invert in SLEPc. I tried with >>> ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works >>> in complex scalars, but the matrix is real. As soon as you add complex >>> entries Cholesky fails, for instance adding this: >>> ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); >>> ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); >>> >> In this case, you must call MatSetOption(A,MAT_HERMITIAN,PETSC_TRUE); Then, petsc will throw an error for '-pc_type cholesky'. > >>> I don't know if it is a bug or that the algorithm cannot support complex >>> Hermitian matrices. Maybe Hong can confirm any of these. In the latter >>> case, I agree that all packages should give an error message, as >>> mkl_pardiso does. >>> >> I also add an error flag for Cholesky/ICC if user does not set MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE) for aij matrix. See https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 Let me know if you have any comments about this fix. Hong > >>> > El 25 sept 2017, a las 7:17, Greg Meyer >>> escribi?: >>> > >>> > Hi all, >>> > >>> > Hong--looking at your link, there may be no special algorithm for >>> Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like >>> it would any matrix. Furthermore it appears that Cholesky of complex >>> matrices is supported from this link: https://www.mcs.anl.gov/petsc/ >>> documentation/linearsolvertable.html >>> > >>> > So do you or anyone have any idea why I get incorrect eigenvalues? >>> > >>> > Thanks, >>> > Greg >>> > >>> > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer >>> wrote: >>> > Ok, thanks. It seems that PETSc clearly should throw an error in this >>> case instead of just giving incorrect answers? I am surprised that it does >>> not throw an error... >>> > >>> > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: >>> > Greg : >>> > Yes, they are Hermitian. >>> > >>> > PETSc does not support Cholesky factorization for Hermitian. >>> > It seems mumps does not support Hermitian either >>> > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/ >>> 2015-November/027541.html >>> > >>> > Hong >>> > >>> > >>> > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: >>> > Greg: >>> > >>> > OK, the difference is whether LU or Cholesky factorization is used. >>> But I would hope that neither one should give incorrect eigenvalues, and >>> when I run with the latter it does! >>> > Are your matrices symmetric/Hermitian? >>> > Hong >>> > >>> > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: >>> > Gregory : >>> > Use '-eps_view' for both runs to check the algorithms being used. >>> > Hong >>> > >>> > Hi all, >>> > >>> > I'm using shift-invert with EPS to solve for eigenvalues. I find that >>> if I do only >>> > >>> > ... >>> > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>> > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>> > ... >>> > >>> > in my code I get correct eigenvalues. But if I do >>> > >>> > ... >>> > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>> > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>> > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); >>> > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >>> > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); >>> > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); >>> > ... >>> > >>> > the eigenvalues found by EPS are completely wrong! Somehow I thought I >>> was supposed to do the latter, from the examples etc, but I guess that was >>> not correct? I attach the full piece of test code and a test matrix. >>> > >>> > Best, >>> > Greg >>> > >>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbcbh1999 at gmail.com Thu Sep 28 22:57:07 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Thu, 28 Sep 2017 23:57:07 -0400 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE Message-ID: I'm experiencing error when doing mesh refinement(mesh x2 for X, Y and Z direction) for 3D poisson problem. PETSc produces no errors when no mesh refinement. I've pasted some debugging info here. Please advise. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid on a arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 23:47:39 2017 [0]PETSC ERROR: Configure options --with-mpi-dir=/cm/shared/apps/openmpi/gcc/64/1.8.5/ --download-fblaslapack=/home/haozhang/tmp/fblaslapack-3.4.2.tar.gz --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz --with-valgrind-include=/home/haozhang/include/valgrind/ [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/vec/vec/interface/rvector.c [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/impls/bcgs/bcgs.c [0]PETSC ERROR: #3 KSPSolve() line 460 in /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/interface/itfunc.c [vn18:99838] *** Process received signal *** [vn18:99838] Signal: Aborted (6) [vn18:99838] Signal code: (-6) [vn18:99848] *** Process received signal *** [vn18:99849] *** Process received signal *** [vn18:99849] Signal: Aborted (6) [vn18:99849] Signal code: (-6) [vn18:99848] Signal: Aborted (6) [vn18:99848] Signal code: (-6) [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790)[0x2aaaadbb4790] [vn18:99838] [ 1] [vn18:99848] [ 0] /lib64/libpthread.so.0(+0xf790)[0x2aaaadbb4790] [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] [vn18:99838] [ 3] [vn18:99849] [ 0] /lib64/libpthread.so.0(+0xf790)[0x2aaaadbb4790] [vn18:99849] [ 1] [vn15:122106] *** Process received signal *** [vn15:122106] Signal: Aborted (6) [vn15:122106] Signal code: (-6) /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscTraceBackErrorHandler+0x503)[0x2aaaaae1d7e1] [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] [vn18:99849] [ 3] [vn15:122107] *** Process received signal *** [vn15:122107] Signal: Aborted (6) [vn15:122107] Signal code: (-6) [vn16:86295] *** Process received signal *** [vn16:86295] Signal: Aborted (6) [vn16:86295] Signal code: (-6) [vn18:99824] *** Process received signal *** [vn18:99824] Signal: Aborted (6) [vn18:99824] Signal code: (-6) [vn18:99844] *** Process received signal *** [vn18:99844] Signal: Aborted (6) [vn18:99844] Signal code: (-6) /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Sep 28 23:19:09 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 28 Sep 2017 21:19:09 -0700 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: Please upgrade to the latest 3.8 version of PETSc. and then run with -fp_trap This message is usually an indication that a NaN or Inf got into the numerical computation. Using the latest PETSc will make it much easier to track down. Barry > On Sep 28, 2017, at 8:57 PM, Hao Zhang wrote: > > I'm experiencing error when doing mesh refinement(mesh x2 for X, Y and Z direction) for 3D poisson problem. > PETSc produces no errors when no mesh refinement. I've pasted some debugging info here. Please advise. > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid on a arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 23:47:39 2017 > [0]PETSC ERROR: Configure options --with-mpi-dir=/cm/shared/apps/openmpi/gcc/64/1.8.5/ --download-fblaslapack=/home/haozhang/tmp/fblaslapack-3.4.2.tar.gz --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz --with-valgrind-include=/home/haozhang/include/valgrind/ > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/vec/vec/interface/rvector.c > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: #3 KSPSolve() line 460 in /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/interface/itfunc.c > [vn18:99838] *** Process received signal *** > [vn18:99838] Signal: Aborted (6) > [vn18:99838] Signal code: (-6) > [vn18:99848] *** Process received signal *** > [vn18:99849] *** Process received signal *** > [vn18:99849] Signal: Aborted (6) > [vn18:99849] Signal code: (-6) > [vn18:99848] Signal: Aborted (6) > [vn18:99848] Signal code: (-6) > [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790)[0x2aaaadbb4790] > [vn18:99838] [ 1] [vn18:99848] [ 0] /lib64/libpthread.so.0(+0xf790)[0x2aaaadbb4790] > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > [vn18:99838] [ 3] [vn18:99849] [ 0] /lib64/libpthread.so.0(+0xf790)[0x2aaaadbb4790] > [vn18:99849] [ 1] [vn15:122106] *** Process received signal *** > [vn15:122106] Signal: Aborted (6) > [vn15:122106] Signal code: (-6) > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscTraceBackErrorHandler+0x503)[0x2aaaaae1d7e1] > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > [vn18:99849] [ 3] [vn15:122107] *** Process received signal *** > [vn15:122107] Signal: Aborted (6) > [vn15:122107] Signal code: (-6) > [vn16:86295] *** Process received signal *** > [vn16:86295] Signal: Aborted (6) > [vn16:86295] Signal code: (-6) > [vn18:99824] *** Process received signal *** > [vn18:99824] Signal: Aborted (6) > [vn18:99824] Signal code: (-6) > [vn18:99844] *** Process received signal *** > [vn18:99844] Signal: Aborted (6) > [vn18:99844] Signal code: (-6) > /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] > > > > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 From stefano.zampini at gmail.com Fri Sep 29 01:35:41 2017 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Fri, 29 Sep 2017 09:35:41 +0300 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> Message-ID: Hong, I personally believe that commit https://bitbucket.org/petsc/petsc/commits/ 966c94c9cf4fa13d455726ec36800a577e00b171 should be reverted. I agree on the fact that when the user sets an option (the hermitian one in this case) and that feature is not supported we should throw an error ( https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7) , but I don't like the fact that the user is forced to set on option to use a feature that he already requested (as in https://bitbucket.org/petsc/ petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171). Barry, what do you think? 2017-09-29 5:28 GMT+03:00 Hong : > Greg: > >> Thanks so much for the detailed response. I am glad to know the reason >> behind it--hopefully we eventually figure out why the solvers have this >> behavior! Hong, I really appreciate you working on a patch to throw an >> error in this case. It definitely bit me and some people using my code... >> Hopefully it doesn't happen to anyone else! :) >> > I added an error flag for using MUMPS Cholesky factorization on Hermitian > matrix. See branch hzhang/mumps-HermitianCholesky-errflag > https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d > 0c742078c7 > > Jose, > PETSc does not support Cholesky for Hermitian matrix. > >> >>>> The linear solver table probably needs to be updated. I have tried >>>> several Cholesky solvers. With mkl_pardiso I get an explicit error message >>>> that it does not support Cholesky with complex scalars. The rest (PETSc, >>>> MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is >>>> not related to your matrix, nor to shift-and-invert in SLEPc. I tried with >>>> ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example >>>> works in complex scalars, but the matrix is real. As soon as you add >>>> complex entries Cholesky fails, for instance adding this: >>>> ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); >>>> ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); >>>> >>> > In this case, you must call > MatSetOption(A,MAT_HERMITIAN,PETSC_TRUE); > > Then, petsc will throw an error for '-pc_type cholesky'. > >> >>>> I don't know if it is a bug or that the algorithm cannot support >>>> complex Hermitian matrices. Maybe Hong can confirm any of these. In the >>>> latter case, I agree that all packages should give an error message, as >>>> mkl_pardiso does. >>>> >>> > I also add an error flag for Cholesky/ICC if user does not set > MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE) for aij matrix. > See https://bitbucket.org/petsc/petsc/commits/ > 966c94c9cf4fa13d455726ec36800a577e00b171 > > Let me know if you have any comments about this fix. > > Hong > >> >>>> > El 25 sept 2017, a las 7:17, Greg Meyer >>>> escribi?: >>>> > >>>> > Hi all, >>>> > >>>> > Hong--looking at your link, there may be no special algorithm for >>>> Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like >>>> it would any matrix. Furthermore it appears that Cholesky of complex >>>> matrices is supported from this link: https://www.mcs.anl.gov/petsc/ >>>> documentation/linearsolvertable.html >>>> > >>>> > So do you or anyone have any idea why I get incorrect eigenvalues? >>>> > >>>> > Thanks, >>>> > Greg >>>> > >>>> > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer >>>> wrote: >>>> > Ok, thanks. It seems that PETSc clearly should throw an error in this >>>> case instead of just giving incorrect answers? I am surprised that it does >>>> not throw an error... >>>> > >>>> > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: >>>> > Greg : >>>> > Yes, they are Hermitian. >>>> > >>>> > PETSc does not support Cholesky factorization for Hermitian. >>>> > It seems mumps does not support Hermitian either >>>> > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015- >>>> November/027541.html >>>> > >>>> > Hong >>>> > >>>> > >>>> > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: >>>> > Greg: >>>> > >>>> > OK, the difference is whether LU or Cholesky factorization is used. >>>> But I would hope that neither one should give incorrect eigenvalues, and >>>> when I run with the latter it does! >>>> > Are your matrices symmetric/Hermitian? >>>> > Hong >>>> > >>>> > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: >>>> > Gregory : >>>> > Use '-eps_view' for both runs to check the algorithms being used. >>>> > Hong >>>> > >>>> > Hi all, >>>> > >>>> > I'm using shift-invert with EPS to solve for eigenvalues. I find that >>>> if I do only >>>> > >>>> > ... >>>> > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>>> > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>>> > ... >>>> > >>>> > in my code I get correct eigenvalues. But if I do >>>> > >>>> > ... >>>> > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>>> > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>>> > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); >>>> > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >>>> > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); >>>> > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); >>>> > ... >>>> > >>>> > the eigenvalues found by EPS are completely wrong! Somehow I thought >>>> I was supposed to do the latter, from the examples etc, but I guess that >>>> was not correct? I attach the full piece of test code and a test matrix. >>>> > >>>> > Best, >>>> > Greg >>>> > >>>> >>>> >>> > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Fri Sep 29 02:50:50 2017 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Fri, 29 Sep 2017 10:50:50 +0300 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> Message-ID: Perhaps you can check for the hermitian option and complex values in MatGetFactor_XXX_mumps? https://bitbucket.org/petsc/petsc/src/8f21f52c465b775a76cda90fe9c51d0c742078c7/src/mat/impls/aij/mpi/mumps/mumps.c?at=hzhang%2Fmumps-HermitianCholesky-errflag&fileviewer=file-view-default#mumps.c-2159 2017-09-29 9:35 GMT+03:00 Stefano Zampini : > Hong, > > I personally believe that commit https://bitbucket.org/ > petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 should be > reverted. > I agree on the fact that when the user sets an option (the hermitian one > in this case) and that feature is not supported we should throw an error ( > https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d > 0c742078c7) , but I don't like the fact that the user is forced to set on > option to use a feature that he already requested (as in > https://bitbucket.org/petsc/petsc/commits/966c94c9cf > 4fa13d455726ec36800a577e00b171). > > Barry, what do you think? > > 2017-09-29 5:28 GMT+03:00 Hong : > >> Greg: >> >>> Thanks so much for the detailed response. I am glad to know the reason >>> behind it--hopefully we eventually figure out why the solvers have this >>> behavior! Hong, I really appreciate you working on a patch to throw an >>> error in this case. It definitely bit me and some people using my code... >>> Hopefully it doesn't happen to anyone else! :) >>> >> I added an error flag for using MUMPS Cholesky factorization on Hermitian >> matrix. See branch hzhang/mumps-HermitianCholesky-errflag >> https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76 >> cda90fe9c51d0c742078c7 >> >> Jose, >> PETSc does not support Cholesky for Hermitian matrix. >> >>> >>>>> The linear solver table probably needs to be updated. I have tried >>>>> several Cholesky solvers. With mkl_pardiso I get an explicit error message >>>>> that it does not support Cholesky with complex scalars. The rest (PETSc, >>>>> MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is >>>>> not related to your matrix, nor to shift-and-invert in SLEPc. I tried with >>>>> ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example >>>>> works in complex scalars, but the matrix is real. As soon as you add >>>>> complex entries Cholesky fails, for instance adding this: >>>>> ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); >>>>> ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); >>>>> >>>> >> In this case, you must call >> MatSetOption(A,MAT_HERMITIAN,PETSC_TRUE); >> >> Then, petsc will throw an error for '-pc_type cholesky'. >> >>> >>>>> I don't know if it is a bug or that the algorithm cannot support >>>>> complex Hermitian matrices. Maybe Hong can confirm any of these. In the >>>>> latter case, I agree that all packages should give an error message, as >>>>> mkl_pardiso does. >>>>> >>>> >> I also add an error flag for Cholesky/ICC if user does not set >> MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE) for aij matrix. >> See https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d45 >> 5726ec36800a577e00b171 >> >> Let me know if you have any comments about this fix. >> >> Hong >> >>> >>>>> > El 25 sept 2017, a las 7:17, Greg Meyer >>>>> escribi?: >>>>> > >>>>> > Hi all, >>>>> > >>>>> > Hong--looking at your link, there may be no special algorithm for >>>>> Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like >>>>> it would any matrix. Furthermore it appears that Cholesky of complex >>>>> matrices is supported from this link: https://www.mcs.anl.gov/petsc/ >>>>> documentation/linearsolvertable.html >>>>> > >>>>> > So do you or anyone have any idea why I get incorrect eigenvalues? >>>>> > >>>>> > Thanks, >>>>> > Greg >>>>> > >>>>> > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer >>>>> wrote: >>>>> > Ok, thanks. It seems that PETSc clearly should throw an error in >>>>> this case instead of just giving incorrect answers? I am surprised that it >>>>> does not throw an error... >>>>> > >>>>> > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: >>>>> > Greg : >>>>> > Yes, they are Hermitian. >>>>> > >>>>> > PETSc does not support Cholesky factorization for Hermitian. >>>>> > It seems mumps does not support Hermitian either >>>>> > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-Nov >>>>> ember/027541.html >>>>> > >>>>> > Hong >>>>> > >>>>> > >>>>> > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: >>>>> > Greg: >>>>> > >>>>> > OK, the difference is whether LU or Cholesky factorization is used. >>>>> But I would hope that neither one should give incorrect eigenvalues, and >>>>> when I run with the latter it does! >>>>> > Are your matrices symmetric/Hermitian? >>>>> > Hong >>>>> > >>>>> > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: >>>>> > Gregory : >>>>> > Use '-eps_view' for both runs to check the algorithms being used. >>>>> > Hong >>>>> > >>>>> > Hi all, >>>>> > >>>>> > I'm using shift-invert with EPS to solve for eigenvalues. I find >>>>> that if I do only >>>>> > >>>>> > ... >>>>> > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>>>> > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>>>> > ... >>>>> > >>>>> > in my code I get correct eigenvalues. But if I do >>>>> > >>>>> > ... >>>>> > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >>>>> > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >>>>> > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); >>>>> > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >>>>> > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); >>>>> > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); >>>>> > ... >>>>> > >>>>> > the eigenvalues found by EPS are completely wrong! Somehow I thought >>>>> I was supposed to do the latter, from the examples etc, but I guess that >>>>> was not correct? I attach the full piece of test code and a test matrix. >>>>> > >>>>> > Best, >>>>> > Greg >>>>> > >>>>> >>>>> >>>> >> > > > -- > Stefano > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Fri Sep 29 06:19:54 2017 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Fri, 29 Sep 2017 12:19:54 +0100 Subject: [petsc-users] Understanding matmult memory performance Message-ID: <8D683AE4-75B9-40AF-97E6-6801657DE095@imperial.ac.uk> Dear all, I'm attempting to understand some results I'm getting for matmult performance. In particular, it looks like I'm obtaining timings that suggest that I'm getting more main memory bandwidth than I think is possible. The run setup is using 2 24 core (dual socket) ivybridge nodes (Xeon E5-2697 v2). The specced main memory bandwidth is 85.3 GB/s per node, and I measure a STREAM triad bandwidth using 48 MPI processes (two nodes) of 148.2 GB/s. The last level cache is 30MB (shared between 12 cores) The matrix I'm using is respectively a P1, P2, and P3 discretisation of the Laplacian on a regular tetrahedral grid. The matrix sizes are respectively: P1: Mat Object: 48 MPI processes type: mpiaij rows=8120601, cols=8120601 total: nonzeros=120841801, allocated nonzeros=120841801 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines P2: Mat Object: 48 MPI processes type: mpiaij rows=8120601, cols=8120601 total: nonzeros=231382401, allocated nonzeros=231382401 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines P3: Mat Object: 48 MPI processes type: mpiaij rows=13997521, cols=13997521 total: nonzeros=674173201, allocated nonzeros=674173201 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Both sizeof(PetscScalar) and sizeof(PetscInt) are 8 bytes. Ignoring data for vector and row indices, then, for a matmult I need to move 16*nonzeros bytes. MatMults take, respectively: P1: 0.0114362s P2: 0.0196032s P3: 0.0524525s So the estimated achieved memory bandwidth is: P1: 120841801 * 16 / 0.0114362 = 157.45GB/s P2: 231382401 * 16 / 0.0196032 = 175.88GB/s P3: 674173201 * 16 / 0.0524525 = 191.52GB/s So all of those numbers are higher than the stream bandwidth, and the P2 and P3 numbers are higher than the spec sheet bandwidth. I don't think PETSc is doing anything magic, but hints appreciated, it would be nice to explain this. Cheers, Lawrence Full -log_view output: -------------------------------------------------------------------------------- *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** -------------------------------------------------------------------------------- Int Type has 8 bytes, Scalar Type has 8 bytes P1: Mat Object: 48 MPI processes type: mpiaij rows=8120601, cols=8120601 total: nonzeros=120841801, allocated nonzeros=120841801 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines P2: Mat Object: 48 MPI processes type: mpiaij rows=8120601, cols=8120601 total: nonzeros=231382401, allocated nonzeros=231382401 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines P3: Mat Object: 48 MPI processes type: mpiaij rows=13997521, cols=13997521 total: nonzeros=674173201, allocated nonzeros=674173201 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- profile-matvec.py on a petsc-gnu51-ivybridge-int64 named nid00013 with 48 processors, by lmn01 Fri Sep 29 11:58:21 2017 Using Petsc Development GIT revision: v3.7.5-3014-g413f72f GIT Date: 2017-02-05 17:50:57 -0600 Max Max/Min Avg Total Time (sec): 1.150e+02 1.00000 1.150e+02 Objects: 1.832e+03 1.50534 1.269e+03 Flops: 2.652e+10 1.16244 2.486e+10 1.193e+12 Flops/sec: 2.306e+08 1.16244 2.162e+08 1.038e+10 MPI Messages: 1.021e+04 3.00279 5.091e+03 2.444e+05 MPI Message Lengths: 3.314e+09 1.97310 3.697e+05 9.035e+10 MPI Reductions: 2.630e+02 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.0701e+02 93.1% 5.5715e+11 46.7% 1.942e+05 79.4% 3.644e+05 98.6% 2.560e+02 97.3% 1: P(1) aij matrix: 1.5561e+00 1.4% 5.5574e+10 4.7% 1.688e+04 6.9% 9.789e+02 0.3% 2.000e+00 0.8% 2: P(2) aij matrix: 1.9378e+00 1.7% 8.8214e+10 7.4% 1.688e+04 6.9% 1.483e+03 0.4% 2.000e+00 0.8% 3: P(3) aij matrix: 4.4890e+00 3.9% 4.9225e+11 41.3% 1.648e+04 6.7% 2.829e+03 0.8% 2.000e+00 0.8% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage PetscBarrier 4 1.0 2.7271e+00 1.0 0.00e+00 0.0 3.8e+03 2.4e+01 2.0e+01 2 0 2 0 8 3 0 2 0 8 0 BuildTwoSided 124 1.0 9.0858e+00 7.2 0.00e+00 0.0 2.7e+04 8.0e+00 0.0e+00 6 0 11 0 0 7 0 14 0 0 0 VecSet 16 1.0 5.8370e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 3 1.0 2.1945e-0269.7 0.00e+00 0.0 1.3e+03 2.6e+04 0.0e+00 0 0 1 0 0 0 0 1 0 0 0 VecScatterEnd 3 1.0 2.2460e-0218.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSetRandom 3 1.0 4.0847e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMult 3 1.0 9.4907e-02 1.2 4.50e+07 1.1 1.3e+03 2.6e+04 0.0e+00 0 0 1 0 0 0 0 1 0 0 21311 MatAssemblyBegin 12 1.0 2.6438e-03235.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 12 1.0 6.6632e-01 2.5 0.00e+00 0.0 2.5e+03 1.3e+04 2.4e+01 0 0 1 0 9 0 0 1 0 9 0 MatView 9 1.0 5.3831e-0112.9 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 3 0 0 0 0 4 0 Mesh Partition 6 1.0 1.3552e+01 1.0 0.00e+00 0.0 1.0e+05 5.9e+04 3.3e+01 12 0 41 7 13 13 0 52 7 13 0 Mesh Migration 6 1.0 1.8341e+01 1.0 0.00e+00 0.0 7.5e+04 1.0e+06 7.2e+01 16 0 31 85 27 17 0 39 86 28 0 DMPlexInterp 3 1.0 1.3771e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 12 0 0 0 2 13 0 0 0 2 0 DMPlexDistribute 3 1.0 1.0266e+01 1.0 0.00e+00 0.0 4.9e+04 5.9e+04 2.7e+01 9 0 20 3 10 10 0 25 3 11 0 DMPlexDistCones 6 1.0 6.9775e+00 1.5 0.00e+00 0.0 1.2e+04 2.3e+06 0.0e+00 5 0 5 32 0 6 0 6 32 0 0 DMPlexDistLabels 6 1.0 7.9111e+00 1.0 0.00e+00 0.0 4.0e+04 9.8e+05 6.0e+00 7 0 16 43 2 7 0 21 44 2 0 DMPlexDistribOL 3 1.0 2.2335e+01 1.0 0.00e+00 0.0 1.3e+05 6.6e+05 7.8e+01 19 0 53 94 30 21 0 66 95 30 0 DMPlexDistField 9 1.0 7.2773e-01 1.0 0.00e+00 0.0 1.7e+04 2.0e+05 6.0e+00 1 0 7 4 2 1 0 9 4 2 0 DMPlexDistData 6 1.0 8.0047e+00 9.4 0.00e+00 0.0 8.6e+04 1.2e+04 0.0e+00 6 0 35 1 0 6 0 45 1 0 0 DMPlexStratify 19 1.0 1.8531e+01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.9e+01 15 0 0 0 7 16 0 0 0 7 0 SFSetGraph 141 1.0 2.2412e+00 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 SFBcastBegin 271 1.0 1.1975e+01 2.0 0.00e+00 0.0 1.8e+05 4.8e+05 0.0e+00 9 0 75 98 0 10 0 95100 0 0 SFBcastEnd 271 1.0 6.4306e+00 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 SFReduceBegin 12 1.0 1.7538e-0112.8 0.00e+00 0.0 4.8e+03 5.9e+04 0.0e+00 0 0 2 0 0 0 0 2 0 0 0 SFReduceEnd 12 1.0 2.2638e-01 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFFetchOpBegin 3 1.0 9.9087e-0415.6 0.00e+00 0.0 6.3e+02 3.9e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFFetchOpEnd 3 1.0 3.6049e-02 6.4 0.00e+00 0.0 6.3e+02 3.9e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 CreateMesh 15 1.0 4.9047e+01 1.0 0.00e+00 0.0 1.8e+05 4.9e+05 1.2e+02 42 0 73 97 44 45 0 92 98 46 0 CreateFunctionSpace 3 1.0 4.2819e+01 1.0 0.00e+00 0.0 1.4e+05 6.3e+05 1.2e+02 37 0 56 95 44 40 0 71 97 45 0 Mesh: reorder 3 1.0 1.5455e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 1 0 0 0 2 1 0 0 0 2 0 Mesh: numbering 3 1.0 1.0627e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 9 0 0 0 2 10 0 0 0 2 0 CreateSparsity 3 1.0 2.0243e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 MatZeroInitial 3 1.0 2.7938e+00 1.0 0.00e+00 0.0 2.5e+03 1.3e+04 2.7e+01 2 0 1 0 10 3 0 1 0 11 0 ParLoopExecute 6 1.0 3.1709e+00 1.2 1.24e+10 1.2 0.0e+00 0.0e+00 0.0e+00 3 47 0 0 0 3100 0 0 0 175069 ParLoopset_4 2 1.0 1.1100e-0222.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopHaloEnd 6 1.0 2.9564e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopRednBegin 6 1.0 7.0810e-05 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopRednEnd 6 1.0 6.5088e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopCells 9 1.0 2.9736e+00 1.2 1.24e+10 1.2 0.0e+00 0.0e+00 0.0e+00 2 47 0 0 0 3100 0 0 0 186686 ParLoopset_10 2 1.0 1.1411e-03 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopset_16 2 1.0 1.1880e-03 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 1: P(1) aij matrix VecScatterBegin 40 1.0 1.1312e-02 8.5 0.00e+00 0.0 1.7e+04 1.4e+04 0.0e+00 0 0 7 0 0 0 0100100 0 0 VecScatterEnd 40 1.0 2.6442e-0161.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 10 0 0 0 0 0 VecSetRandom 40 1.0 4.4251e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 27 0 0 0 0 0 MatMult 40 1.0 4.5745e-01 1.1 2.06e+08 1.1 1.7e+04 1.4e+04 0.0e+00 0 1 7 0 0 28 17100100 0 20423 MatAssemblyBegin 3 1.0 2.3842e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 3 1.0 1.8371e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatZeroEntries 1 1.0 5.2531e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 2 1.0 1.8248e-012468.9 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 1 5 0 0 0100 0 AssembleMat 1 1.0 7.0037e-01 1.0 1.01e+09 1.1 0.0e+00 0.0e+00 2.0e+00 1 4 0 0 1 45 83 0 0100 66009 ParLoopExecute 1 1.0 6.7369e-01 1.4 1.01e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 38 83 0 0 0 68623 ParLoopHaloEnd 1 1.0 1.3113e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopRednBegin 1 1.0 1.3113e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopRednEnd 1 1.0 1.0967e-05 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopCells 3 1.0 6.7352e-01 1.4 1.01e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 38 83 0 0 0 68641 --- Event Stage 2: P(2) aij matrix VecScatterBegin 40 1.0 1.2448e-02 6.3 0.00e+00 0.0 1.7e+04 2.1e+04 0.0e+00 0 0 7 0 0 0 0100100 0 0 VecScatterEnd 40 1.0 4.3488e-0156.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 14 0 0 0 0 0 VecSetRandom 40 1.0 4.4287e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 22 0 0 0 0 0 MatMult 40 1.0 7.8413e-01 1.1 4.04e+08 1.1 1.7e+04 2.1e+04 0.0e+00 1 2 7 0 0 39 21100100 0 23192 MatAssemblyBegin 3 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 3 1.0 2.4675e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatZeroEntries 1 1.0 9.4781e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 2 1.0 1.4482e-01344.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 1 3 0 0 0100 0 AssembleMat 1 1.0 7.5959e-01 1.0 1.57e+09 1.2 0.0e+00 0.0e+00 2.0e+00 1 6 0 0 1 39 79 0 0100 92192 ParLoopExecute 1 1.0 7.1835e-01 1.2 1.57e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 6 0 0 0 34 79 0 0 0 97484 ParLoopHaloEnd 1 1.0 1.1921e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopRednBegin 1 1.0 1.7881e-05 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopRednEnd 1 1.0 1.4067e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopCells 3 1.0 7.1820e-01 1.2 1.57e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 6 0 0 0 34 79 0 0 0 97505 --- Event Stage 3: P(3) aij matrix VecScatterBegin 40 1.0 2.3520e-0210.9 0.00e+00 0.0 1.6e+04 4.2e+04 0.0e+00 0 0 7 1 0 0 0100100 0 0 VecScatterEnd 40 1.0 6.6521e-0138.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 0 0 0 0 0 VecSetRandom 40 1.0 7.5565e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 16 0 0 0 0 0 MatMult 40 1.0 2.0981e+00 1.0 1.19e+09 1.1 1.6e+04 4.2e+04 0.0e+00 2 4 7 1 0 46 11100100 0 25439 MatAssemblyBegin 3 1.0 2.8610e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 3 1.0 5.6094e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatZeroEntries 1 1.0 2.9610e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatView 2 1.0 2.8071e-01958.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 1 3 0 0 0100 0 AssembleMat 1 1.0 1.7038e+00 1.0 9.94e+09 1.2 0.0e+00 0.0e+00 2.0e+00 1 37 0 0 1 38 89 0 0100 257591 ParLoopExecute 1 1.0 1.6101e+00 1.2 9.94e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 37 0 0 0 32 89 0 0 0 272582 ParLoopHaloEnd 1 1.0 1.4067e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopRednBegin 1 1.0 1.7166e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopRednEnd 1 1.0 1.4067e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ParLoopCells 3 1.0 1.6099e+00 1.2 9.94e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 37 0 0 0 32 89 0 0 0 272617 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Container 12 11 6776 0. Viewer 4 0 0 0. PetscRandom 3 3 2058 0. Index Set 1095 1085 1383616 0. IS L to G Mapping 15 14 204830392 0. Section 222 209 158840 0. Vector 31 28 41441632 0. Vector Scatter 3 2 2416 0. Matrix 22 18 131705576 0. Distributed Mesh 40 37 182200 0. GraphPartitioner 19 18 11808 0. Star Forest Bipartite Graph 206 200 178256 0. Discrete System 40 37 34336 0. --- Event Stage 1: P(1) aij matrix PetscRandom 40 40 27440 0. --- Event Stage 2: P(2) aij matrix PetscRandom 40 40 27440 0. --- Event Stage 3: P(3) aij matrix PetscRandom 40 40 27440 0. ======================================================================================================================== Average time to get PetscTime(): 0. Average time for MPI_Barrier(): 1.06335e-05 Average time for zero size MPI_Send(): 1.41561e-06 #PETSc Option Table entries: --dimension 3 --output-file poisson-matvecs.csv --problem poisson -log_view -mat_view ::ascii_info #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8 Configure options: --COPTFLAGS="-march=ivybridge -O3" --CXXOPTFLAGS="-march=ivybridge -O3" --FOPTFLAGS="-march=ivybridge -O3" --PETSC_ARCH=petsc-gnu51-ivybridge-int64 --download-exodusii --download-hypre --download-metis --download-netcdf --download-parmetis --download-sowing=1 --known-bits-per-byte=8 --known-has-attribute-aligned=1 --known-level1-dcache-assoc=8 --known-level1-dcache-linesize=64 --known-level1-dcache-size=32768 --known-memcmp-ok=1 --known-mpi-c-double-complex=1 --known-mpi-int64_t=1 --known-mpi-long-double=1 --known-mpi-shared-libraries=1 --known-sdot-returns-double=0 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8 --known-sizeof-void-p=8 --known-snrm2-returns-double=0 --prefix=/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64 --with-64-bit-indices=1 --with-batch=1 --with-blas-lapack-lib="-L/opt/cray/libsci/16.03.1/GNU/5.1/x86_64/lib -lsci_gnu_mp" --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn --with-fortranlib-autodetect=0 --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.14/GNU/5.1 --with-hdf5=1 --with-make-np=4 --with-pic=1 --with-shared-libraries=1 --with-x=0 --download-eigen ----------------------------------------- Libraries compiled on Tue Feb 14 12:07:09 2017 on eslogin003 Machine characteristics: Linux-3.0.101-0.47.86.1.11753.0.PTF-default-x86_64-with-SuSE-11-x86_64 Using PETSc directory: /home2/n01/n01/lmn01/src/petsc Using PETSc arch: petsc-gnu51-ivybridge-int64 ----------------------------------------- Using C compiler: cc -fPIC -march=ivybridge -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: ftn -fPIC -march=ivybridge -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/include -I/home2/n01/n01/lmn01/src/petsc/include -I/home2/n01/n01/lmn01/src/petsc/include -I/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/include -I/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/include -I/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/include/eigen3 -I/opt/cray/hdf5-parallel/1.8.14/GNU/5.1/include ----------------------------------------- Using C linker: cc Using Fortran linker: ftn Using libraries: -Wl,-rpath,/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/lib -L/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/lib -lpetsc -Wl,-rpath,/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/lib -L/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/lib -lHYPRE -lparmetis -lmetis -lexoIIv2for -lexodus -lnetcdf -Wl,-rpath,/opt/cray/hdf5-parallel/1.8.14/GNU/5.1/lib -L/opt/cray/hdf5-parallel/1.8.14/GNU/5.1/lib -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lssl -lcrypto -ldl ----------------------------------------- Application 28632506 resources: utime ~4100s, stime ~428s, Rss ~2685552, inblocks ~2935062, outblocks ~42464 -------------------------------------------------------------------------------- Resources requested: ncpus=48,place=free,walltime=00:20:00 Resources allocated: cpupercent=0,cput=00:00:02,mem=8980kb,ncpus=48,vmem=172968kb,walltime=00:02:20 *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** -------------------------------------------------------------------------------- From tisaac at cc.gatech.edu Fri Sep 29 08:04:47 2017 From: tisaac at cc.gatech.edu (Tobin Isaac) Date: Fri, 29 Sep 2017 09:04:47 -0400 Subject: [petsc-users] Understanding matmult memory performance In-Reply-To: <8D683AE4-75B9-40AF-97E6-6801657DE095@imperial.ac.uk> References: <8D683AE4-75B9-40AF-97E6-6801657DE095@imperial.ac.uk> Message-ID: <20170929130447.kmvqn3yintmnqrcw@gatech.edu> On Fri, Sep 29, 2017 at 12:19:54PM +0100, Lawrence Mitchell wrote: > Dear all, > > I'm attempting to understand some results I'm getting for matmult performance. In particular, it looks like I'm obtaining timings that suggest that I'm getting more main memory bandwidth than I think is possible. > > The run setup is using 2 24 core (dual socket) ivybridge nodes (Xeon E5-2697 v2). The specced main memory bandwidth is 85.3 GB/s per node, and I measure a STREAM triad bandwidth using 48 MPI processes (two nodes) of 148.2 GB/s. The last level cache is 30MB (shared between 12 cores) One thought: triad has a 1:2 write:read ratio, but with your MatMult() for P3 you would have about 1:50. Unless triad used nontemporal stores, the reported bandwidth from triad will be about 3./4. of the bandwidth available to pure streaming reads, so maybe you actually have ~197 GB/s of read bandwidth available. MatMult() would still be doing suspiciously well, but it would be within the measurements. How confident are you in the specced bandwidth? Cheers, Toby > > The matrix I'm using is respectively a P1, P2, and P3 discretisation of the Laplacian on a regular tetrahedral grid. > > The matrix sizes are respectively: > > P1: > Mat Object: 48 MPI processes > type: mpiaij > rows=8120601, cols=8120601 > total: nonzeros=120841801, allocated nonzeros=120841801 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > P2: > Mat Object: 48 MPI processes > type: mpiaij > rows=8120601, cols=8120601 > total: nonzeros=231382401, allocated nonzeros=231382401 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > P3: > Mat Object: 48 MPI processes > type: mpiaij > rows=13997521, cols=13997521 > total: nonzeros=674173201, allocated nonzeros=674173201 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > Both sizeof(PetscScalar) and sizeof(PetscInt) are 8 bytes. > > Ignoring data for vector and row indices, then, for a matmult I need to move 16*nonzeros bytes. > > MatMults take, respectively: > > P1: 0.0114362s > P2: 0.0196032s > P3: 0.0524525s > > So the estimated achieved memory bandwidth is: > > P1: 120841801 * 16 / 0.0114362 = 157.45GB/s > P2: 231382401 * 16 / 0.0196032 = 175.88GB/s > P3: 674173201 * 16 / 0.0524525 = 191.52GB/s > > So all of those numbers are higher than the stream bandwidth, and the P2 and P3 numbers are higher than the spec sheet bandwidth. > > I don't think PETSc is doing anything magic, but hints appreciated, it would be nice to explain this. > > Cheers, > > Lawrence > > Full -log_view output: > > -------------------------------------------------------------------------------- > *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** > *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** > *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** > *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** > > -------------------------------------------------------------------------------- > Int Type has 8 bytes, Scalar Type has 8 bytes > > P1: > Mat Object: 48 MPI processes > type: mpiaij > rows=8120601, cols=8120601 > total: nonzeros=120841801, allocated nonzeros=120841801 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > P2: > Mat Object: 48 MPI processes > type: mpiaij > rows=8120601, cols=8120601 > total: nonzeros=231382401, allocated nonzeros=231382401 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > P3: > Mat Object: 48 MPI processes > type: mpiaij > rows=13997521, cols=13997521 > total: nonzeros=674173201, allocated nonzeros=674173201 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > ************************************************************************************************************************ > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** > ************************************************************************************************************************ > > ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- > > profile-matvec.py on a petsc-gnu51-ivybridge-int64 named nid00013 with 48 processors, by lmn01 Fri Sep 29 11:58:21 2017 > Using Petsc Development GIT revision: v3.7.5-3014-g413f72f GIT Date: 2017-02-05 17:50:57 -0600 > > Max Max/Min Avg Total > Time (sec): 1.150e+02 1.00000 1.150e+02 > Objects: 1.832e+03 1.50534 1.269e+03 > Flops: 2.652e+10 1.16244 2.486e+10 1.193e+12 > Flops/sec: 2.306e+08 1.16244 2.162e+08 1.038e+10 > MPI Messages: 1.021e+04 3.00279 5.091e+03 2.444e+05 > MPI Message Lengths: 3.314e+09 1.97310 3.697e+05 9.035e+10 > MPI Reductions: 2.630e+02 1.00000 > > Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length N --> 2N flops > and VecAXPY() for complex vectors of length N --> 8N flops > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts %Total Avg %Total counts %Total > 0: Main Stage: 1.0701e+02 93.1% 5.5715e+11 46.7% 1.942e+05 79.4% 3.644e+05 98.6% 2.560e+02 97.3% > 1: P(1) aij matrix: 1.5561e+00 1.4% 5.5574e+10 4.7% 1.688e+04 6.9% 9.789e+02 0.3% 2.000e+00 0.8% > 2: P(2) aij matrix: 1.9378e+00 1.7% 8.8214e+10 7.4% 1.688e+04 6.9% 1.483e+03 0.4% 2.000e+00 0.8% > 3: P(3) aij matrix: 4.4890e+00 3.9% 4.9225e+11 41.3% 1.648e+04 6.7% 2.829e+03 0.8% 2.000e+00 0.8% > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flops: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > Avg. len: average message length (bytes) > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). > %T - percent time in this phase %F - percent flops in this phase > %M - percent messages in this phase %L - percent message lengths in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flops --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > PetscBarrier 4 1.0 2.7271e+00 1.0 0.00e+00 0.0 3.8e+03 2.4e+01 2.0e+01 2 0 2 0 8 3 0 2 0 8 0 > BuildTwoSided 124 1.0 9.0858e+00 7.2 0.00e+00 0.0 2.7e+04 8.0e+00 0.0e+00 6 0 11 0 0 7 0 14 0 0 0 > VecSet 16 1.0 5.8370e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 3 1.0 2.1945e-0269.7 0.00e+00 0.0 1.3e+03 2.6e+04 0.0e+00 0 0 1 0 0 0 0 1 0 0 0 > VecScatterEnd 3 1.0 2.2460e-0218.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSetRandom 3 1.0 4.0847e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatMult 3 1.0 9.4907e-02 1.2 4.50e+07 1.1 1.3e+03 2.6e+04 0.0e+00 0 0 1 0 0 0 0 1 0 0 21311 > MatAssemblyBegin 12 1.0 2.6438e-03235.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 12 1.0 6.6632e-01 2.5 0.00e+00 0.0 2.5e+03 1.3e+04 2.4e+01 0 0 1 0 9 0 0 1 0 9 0 > MatView 9 1.0 5.3831e-0112.9 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 3 0 0 0 0 4 0 > Mesh Partition 6 1.0 1.3552e+01 1.0 0.00e+00 0.0 1.0e+05 5.9e+04 3.3e+01 12 0 41 7 13 13 0 52 7 13 0 > Mesh Migration 6 1.0 1.8341e+01 1.0 0.00e+00 0.0 7.5e+04 1.0e+06 7.2e+01 16 0 31 85 27 17 0 39 86 28 0 > DMPlexInterp 3 1.0 1.3771e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 12 0 0 0 2 13 0 0 0 2 0 > DMPlexDistribute 3 1.0 1.0266e+01 1.0 0.00e+00 0.0 4.9e+04 5.9e+04 2.7e+01 9 0 20 3 10 10 0 25 3 11 0 > DMPlexDistCones 6 1.0 6.9775e+00 1.5 0.00e+00 0.0 1.2e+04 2.3e+06 0.0e+00 5 0 5 32 0 6 0 6 32 0 0 > DMPlexDistLabels 6 1.0 7.9111e+00 1.0 0.00e+00 0.0 4.0e+04 9.8e+05 6.0e+00 7 0 16 43 2 7 0 21 44 2 0 > DMPlexDistribOL 3 1.0 2.2335e+01 1.0 0.00e+00 0.0 1.3e+05 6.6e+05 7.8e+01 19 0 53 94 30 21 0 66 95 30 0 > DMPlexDistField 9 1.0 7.2773e-01 1.0 0.00e+00 0.0 1.7e+04 2.0e+05 6.0e+00 1 0 7 4 2 1 0 9 4 2 0 > DMPlexDistData 6 1.0 8.0047e+00 9.4 0.00e+00 0.0 8.6e+04 1.2e+04 0.0e+00 6 0 35 1 0 6 0 45 1 0 0 > DMPlexStratify 19 1.0 1.8531e+01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.9e+01 15 0 0 0 7 16 0 0 0 7 0 > SFSetGraph 141 1.0 2.2412e+00 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 > SFBcastBegin 271 1.0 1.1975e+01 2.0 0.00e+00 0.0 1.8e+05 4.8e+05 0.0e+00 9 0 75 98 0 10 0 95100 0 0 > SFBcastEnd 271 1.0 6.4306e+00 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 > SFReduceBegin 12 1.0 1.7538e-0112.8 0.00e+00 0.0 4.8e+03 5.9e+04 0.0e+00 0 0 2 0 0 0 0 2 0 0 0 > SFReduceEnd 12 1.0 2.2638e-01 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > SFFetchOpBegin 3 1.0 9.9087e-0415.6 0.00e+00 0.0 6.3e+02 3.9e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > SFFetchOpEnd 3 1.0 3.6049e-02 6.4 0.00e+00 0.0 6.3e+02 3.9e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > CreateMesh 15 1.0 4.9047e+01 1.0 0.00e+00 0.0 1.8e+05 4.9e+05 1.2e+02 42 0 73 97 44 45 0 92 98 46 0 > CreateFunctionSpace 3 1.0 4.2819e+01 1.0 0.00e+00 0.0 1.4e+05 6.3e+05 1.2e+02 37 0 56 95 44 40 0 71 97 45 0 > Mesh: reorder 3 1.0 1.5455e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 1 0 0 0 2 1 0 0 0 2 0 > Mesh: numbering 3 1.0 1.0627e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 9 0 0 0 2 10 0 0 0 2 0 > CreateSparsity 3 1.0 2.0243e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 > MatZeroInitial 3 1.0 2.7938e+00 1.0 0.00e+00 0.0 2.5e+03 1.3e+04 2.7e+01 2 0 1 0 10 3 0 1 0 11 0 > ParLoopExecute 6 1.0 3.1709e+00 1.2 1.24e+10 1.2 0.0e+00 0.0e+00 0.0e+00 3 47 0 0 0 3100 0 0 0 175069 > ParLoopset_4 2 1.0 1.1100e-0222.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopHaloEnd 6 1.0 2.9564e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednBegin 6 1.0 7.0810e-05 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednEnd 6 1.0 6.5088e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopCells 9 1.0 2.9736e+00 1.2 1.24e+10 1.2 0.0e+00 0.0e+00 0.0e+00 2 47 0 0 0 3100 0 0 0 186686 > ParLoopset_10 2 1.0 1.1411e-03 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopset_16 2 1.0 1.1880e-03 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > --- Event Stage 1: P(1) aij matrix > > VecScatterBegin 40 1.0 1.1312e-02 8.5 0.00e+00 0.0 1.7e+04 1.4e+04 0.0e+00 0 0 7 0 0 0 0100100 0 0 > VecScatterEnd 40 1.0 2.6442e-0161.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 10 0 0 0 0 0 > VecSetRandom 40 1.0 4.4251e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 27 0 0 0 0 0 > MatMult 40 1.0 4.5745e-01 1.1 2.06e+08 1.1 1.7e+04 1.4e+04 0.0e+00 0 1 7 0 0 28 17100100 0 20423 > MatAssemblyBegin 3 1.0 2.3842e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 3 1.0 1.8371e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > MatZeroEntries 1 1.0 5.2531e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatView 2 1.0 1.8248e-012468.9 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 1 5 0 0 0100 0 > AssembleMat 1 1.0 7.0037e-01 1.0 1.01e+09 1.1 0.0e+00 0.0e+00 2.0e+00 1 4 0 0 1 45 83 0 0100 66009 > ParLoopExecute 1 1.0 6.7369e-01 1.4 1.01e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 38 83 0 0 0 68623 > ParLoopHaloEnd 1 1.0 1.3113e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednBegin 1 1.0 1.3113e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednEnd 1 1.0 1.0967e-05 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopCells 3 1.0 6.7352e-01 1.4 1.01e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 38 83 0 0 0 68641 > > --- Event Stage 2: P(2) aij matrix > > VecScatterBegin 40 1.0 1.2448e-02 6.3 0.00e+00 0.0 1.7e+04 2.1e+04 0.0e+00 0 0 7 0 0 0 0100100 0 0 > VecScatterEnd 40 1.0 4.3488e-0156.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 14 0 0 0 0 0 > VecSetRandom 40 1.0 4.4287e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 22 0 0 0 0 0 > MatMult 40 1.0 7.8413e-01 1.1 4.04e+08 1.1 1.7e+04 2.1e+04 0.0e+00 1 2 7 0 0 39 21100100 0 23192 > MatAssemblyBegin 3 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 3 1.0 2.4675e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > MatZeroEntries 1 1.0 9.4781e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatView 2 1.0 1.4482e-01344.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 1 3 0 0 0100 0 > AssembleMat 1 1.0 7.5959e-01 1.0 1.57e+09 1.2 0.0e+00 0.0e+00 2.0e+00 1 6 0 0 1 39 79 0 0100 92192 > ParLoopExecute 1 1.0 7.1835e-01 1.2 1.57e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 6 0 0 0 34 79 0 0 0 97484 > ParLoopHaloEnd 1 1.0 1.1921e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednBegin 1 1.0 1.7881e-05 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednEnd 1 1.0 1.4067e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopCells 3 1.0 7.1820e-01 1.2 1.57e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 6 0 0 0 34 79 0 0 0 97505 > > --- Event Stage 3: P(3) aij matrix > > VecScatterBegin 40 1.0 2.3520e-0210.9 0.00e+00 0.0 1.6e+04 4.2e+04 0.0e+00 0 0 7 1 0 0 0100100 0 0 > VecScatterEnd 40 1.0 6.6521e-0138.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 0 0 0 0 0 > VecSetRandom 40 1.0 7.5565e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 16 0 0 0 0 0 > MatMult 40 1.0 2.0981e+00 1.0 1.19e+09 1.1 1.6e+04 4.2e+04 0.0e+00 2 4 7 1 0 46 11100100 0 25439 > MatAssemblyBegin 3 1.0 2.8610e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 3 1.0 5.6094e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > MatZeroEntries 1 1.0 2.9610e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > MatView 2 1.0 2.8071e-01958.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 1 3 0 0 0100 0 > AssembleMat 1 1.0 1.7038e+00 1.0 9.94e+09 1.2 0.0e+00 0.0e+00 2.0e+00 1 37 0 0 1 38 89 0 0100 257591 > ParLoopExecute 1 1.0 1.6101e+00 1.2 9.94e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 37 0 0 0 32 89 0 0 0 272582 > ParLoopHaloEnd 1 1.0 1.4067e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednBegin 1 1.0 1.7166e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednEnd 1 1.0 1.4067e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopCells 3 1.0 1.6099e+00 1.2 9.94e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 37 0 0 0 32 89 0 0 0 272617 > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Container 12 11 6776 0. > Viewer 4 0 0 0. > PetscRandom 3 3 2058 0. > Index Set 1095 1085 1383616 0. > IS L to G Mapping 15 14 204830392 0. > Section 222 209 158840 0. > Vector 31 28 41441632 0. > Vector Scatter 3 2 2416 0. > Matrix 22 18 131705576 0. > Distributed Mesh 40 37 182200 0. > GraphPartitioner 19 18 11808 0. > Star Forest Bipartite Graph 206 200 178256 0. > Discrete System 40 37 34336 0. > > --- Event Stage 1: P(1) aij matrix > > PetscRandom 40 40 27440 0. > > --- Event Stage 2: P(2) aij matrix > > PetscRandom 40 40 27440 0. > > --- Event Stage 3: P(3) aij matrix > > PetscRandom 40 40 27440 0. > ======================================================================================================================== > Average time to get PetscTime(): 0. > Average time for MPI_Barrier(): 1.06335e-05 > Average time for zero size MPI_Send(): 1.41561e-06 > #PETSc Option Table entries: > --dimension 3 > --output-file poisson-matvecs.csv > --problem poisson > -log_view > -mat_view ::ascii_info > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8 > Configure options: --COPTFLAGS="-march=ivybridge -O3" --CXXOPTFLAGS="-march=ivybridge -O3" --FOPTFLAGS="-march=ivybridge -O3" --PETSC_ARCH=petsc-gnu51-ivybridge-int64 --download-exodusii --download-hypre --download-metis --download-netcdf --download-parmetis --download-sowing=1 --known-bits-per-byte=8 --known-has-attribute-aligned=1 --known-level1-dcache-assoc=8 --known-level1-dcache-linesize=64 --known-level1-dcache-size=32768 --known-memcmp-ok=1 --known-mpi-c-double-complex=1 --known-mpi-int64_t=1 --known-mpi-long-double=1 --known-mpi-shared-libraries=1 --known-sdot-returns-double=0 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8 --known-sizeof-void-p=8 --known-snrm2-returns-double=0 --prefix=/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64 --with-64-bit-indices=1 --with-batch=1 --with-blas-lapack-lib="-L/opt/cray/libsci/16.03.1/GNU/5.1/x86_64/lib -lsci_gnu_mp" --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn --with-fortranlib-autodetect=0 --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.14/GNU/5.1 --with-hdf5=1 --with-make-np=4 --with-pic=1 --with-shared-libraries=1 --with-x=0 --download-eigen > ----------------------------------------- > Libraries compiled on Tue Feb 14 12:07:09 2017 on eslogin003 > Machine characteristics: Linux-3.0.101-0.47.86.1.11753.0.PTF-default-x86_64-with-SuSE-11-x86_64 > Using PETSc directory: /home2/n01/n01/lmn01/src/petsc > Using PETSc arch: petsc-gnu51-ivybridge-int64 > ----------------------------------------- > > Using C compiler: cc -fPIC -march=ivybridge -O3 ${COPTFLAGS} ${CFLAGS} > Using Fortran compiler: ftn -fPIC -march=ivybridge -O3 ${FOPTFLAGS} ${FFLAGS} > ----------------------------------------- > > Using include paths: -I/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/include -I/home2/n01/n01/lmn01/src/petsc/include -I/home2/n01/n01/lmn01/src/petsc/include -I/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/include -I/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/include -I/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/include/eigen3 -I/opt/cray/hdf5-parallel/1.8.14/GNU/5.1/include > ----------------------------------------- > > Using C linker: cc > Using Fortran linker: ftn > Using libraries: -Wl,-rpath,/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/lib -L/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/lib -lpetsc -Wl,-rpath,/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/lib -L/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/lib -lHYPRE -lparmetis -lmetis -lexoIIv2for -lexodus -lnetcdf -Wl,-rpath,/opt/cray/hdf5-parallel/1.8.14/GNU/5.1/lib -L/opt/cray/hdf5-parallel/1.8.14/GNU/5.1/lib -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lssl -lcrypto -ldl > ----------------------------------------- > > Application 28632506 resources: utime ~4100s, stime ~428s, Rss ~2685552, inblocks ~2935062, outblocks ~42464 > -------------------------------------------------------------------------------- > > Resources requested: ncpus=48,place=free,walltime=00:20:00 > Resources allocated: cpupercent=0,cput=00:00:02,mem=8980kb,ncpus=48,vmem=172968kb,walltime=00:02:20 > > *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** > *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** > *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** > *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** > -------------------------------------------------------------------------------- > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: not available URL: From tisaac at cc.gatech.edu Fri Sep 29 09:05:01 2017 From: tisaac at cc.gatech.edu (Tobin Isaac) Date: Fri, 29 Sep 2017 10:05:01 -0400 Subject: [petsc-users] Understanding matmult memory performance In-Reply-To: <20170929130447.kmvqn3yintmnqrcw@gatech.edu> References: <8D683AE4-75B9-40AF-97E6-6801657DE095@imperial.ac.uk> <20170929130447.kmvqn3yintmnqrcw@gatech.edu> Message-ID: <20170929140501.ukincl2vdohl7mok@gatech.edu> On Fri, Sep 29, 2017 at 09:04:47AM -0400, Tobin Isaac wrote: > On Fri, Sep 29, 2017 at 12:19:54PM +0100, Lawrence Mitchell wrote: > > Dear all, > > > > I'm attempting to understand some results I'm getting for matmult performance. In particular, it looks like I'm obtaining timings that suggest that I'm getting more main memory bandwidth than I think is possible. > > > > The run setup is using 2 24 core (dual socket) ivybridge nodes (Xeon E5-2697 v2). The specced main memory bandwidth is 85.3 GB/s per node, and I measure a STREAM triad bandwidth using 48 MPI processes (two nodes) of 148.2 GB/s. The last level cache is 30MB (shared between 12 cores) > > One thought: triad has a 1:2 write:read ratio, but with your MatMult() > for P3 you would have about 1:50. Unless triad used nontemporal > stores, the reported bandwidth from triad will be about 3./4. of the > bandwidth available to pure streaming reads, so maybe you actually > have ~197 GB/s of read bandwidth available. MatMult() would still be > doing suspiciously well, but it would be within the measurements. How > confident are you in the specced bandwidth? Are you running on archer? I found one site [1] that lists the bandwidth you gave, which corresponds to DDR3-1333, but other sites [2] all say the nodes have DDR3-1833, in which case you would be getting about 80% of spec bandwidth. [1]: https://www.archer.ac.uk/documentation/best-practice-guide/arch.php [2]: https://www.epcc.ed.ac.uk/blog/2013/11/20/archer-next-national-hpc-service-academic-research > > Cheers, > Toby > > > > > The matrix I'm using is respectively a P1, P2, and P3 discretisation of the Laplacian on a regular tetrahedral grid. > > > > The matrix sizes are respectively: > > > > P1: > > Mat Object: 48 MPI processes > > type: mpiaij > > rows=8120601, cols=8120601 > > total: nonzeros=120841801, allocated nonzeros=120841801 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > > > > > P2: > > Mat Object: 48 MPI processes > > type: mpiaij > > rows=8120601, cols=8120601 > > total: nonzeros=231382401, allocated nonzeros=231382401 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > > > > > P3: > > Mat Object: 48 MPI processes > > type: mpiaij > > rows=13997521, cols=13997521 > > total: nonzeros=674173201, allocated nonzeros=674173201 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > > > > > Both sizeof(PetscScalar) and sizeof(PetscInt) are 8 bytes. > > > > Ignoring data for vector and row indices, then, for a matmult I need to move 16*nonzeros bytes. > > > > MatMults take, respectively: > > > > P1: 0.0114362s > > P2: 0.0196032s > > P3: 0.0524525s > > > > So the estimated achieved memory bandwidth is: > > > > P1: 120841801 * 16 / 0.0114362 = 157.45GB/s > > P2: 231382401 * 16 / 0.0196032 = 175.88GB/s > > P3: 674173201 * 16 / 0.0524525 = 191.52GB/s > > > > So all of those numbers are higher than the stream bandwidth, and the P2 and P3 numbers are higher than the spec sheet bandwidth. > > > > I don't think PETSc is doing anything magic, but hints appreciated, it would be nice to explain this. > > > > Cheers, > > > > Lawrence > > > > Full -log_view output: > > > > -------------------------------------------------------------------------------- > > *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** > > *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** > > *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** > > *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** > > > > -------------------------------------------------------------------------------- > > Int Type has 8 bytes, Scalar Type has 8 bytes > > > > P1: > > Mat Object: 48 MPI processes > > type: mpiaij > > rows=8120601, cols=8120601 > > total: nonzeros=120841801, allocated nonzeros=120841801 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > > > P2: > > Mat Object: 48 MPI processes > > type: mpiaij > > rows=8120601, cols=8120601 > > total: nonzeros=231382401, allocated nonzeros=231382401 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > > > P3: > > Mat Object: 48 MPI processes > > type: mpiaij > > rows=13997521, cols=13997521 > > total: nonzeros=674173201, allocated nonzeros=674173201 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > > > ************************************************************************************************************************ > > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** > > ************************************************************************************************************************ > > > > ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- > > > > profile-matvec.py on a petsc-gnu51-ivybridge-int64 named nid00013 with 48 processors, by lmn01 Fri Sep 29 11:58:21 2017 > > Using Petsc Development GIT revision: v3.7.5-3014-g413f72f GIT Date: 2017-02-05 17:50:57 -0600 > > > > Max Max/Min Avg Total > > Time (sec): 1.150e+02 1.00000 1.150e+02 > > Objects: 1.832e+03 1.50534 1.269e+03 > > Flops: 2.652e+10 1.16244 2.486e+10 1.193e+12 > > Flops/sec: 2.306e+08 1.16244 2.162e+08 1.038e+10 > > MPI Messages: 1.021e+04 3.00279 5.091e+03 2.444e+05 > > MPI Message Lengths: 3.314e+09 1.97310 3.697e+05 9.035e+10 > > MPI Reductions: 2.630e+02 1.00000 > > > > Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) > > e.g., VecAXPY() for real vectors of length N --> 2N flops > > and VecAXPY() for complex vectors of length N --> 8N flops > > > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- > > Avg %Total Avg %Total counts %Total Avg %Total counts %Total > > 0: Main Stage: 1.0701e+02 93.1% 5.5715e+11 46.7% 1.942e+05 79.4% 3.644e+05 98.6% 2.560e+02 97.3% > > 1: P(1) aij matrix: 1.5561e+00 1.4% 5.5574e+10 4.7% 1.688e+04 6.9% 9.789e+02 0.3% 2.000e+00 0.8% > > 2: P(2) aij matrix: 1.9378e+00 1.7% 8.8214e+10 7.4% 1.688e+04 6.9% 1.483e+03 0.4% 2.000e+00 0.8% > > 3: P(3) aij matrix: 4.4890e+00 3.9% 4.9225e+11 41.3% 1.648e+04 6.7% 2.829e+03 0.8% 2.000e+00 0.8% > > > > ------------------------------------------------------------------------------------------------------------------------ > > See the 'Profiling' chapter of the users' manual for details on interpreting output. > > Phase summary info: > > Count: number of times phase was executed > > Time and Flops: Max - maximum over all processors > > Ratio - ratio of maximum to minimum over all processors > > Mess: number of messages sent > > Avg. len: average message length (bytes) > > Reduct: number of global reductions > > Global: entire computation > > Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). > > %T - percent time in this phase %F - percent flops in this phase > > %M - percent messages in this phase %L - percent message lengths in this phase > > %R - percent reductions in this phase > > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) > > ------------------------------------------------------------------------------------------------------------------------ > > Event Count Time (sec) Flops --- Global --- --- Stage --- Total > > Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > > ------------------------------------------------------------------------------------------------------------------------ > > > > --- Event Stage 0: Main Stage > > > > PetscBarrier 4 1.0 2.7271e+00 1.0 0.00e+00 0.0 3.8e+03 2.4e+01 2.0e+01 2 0 2 0 8 3 0 2 0 8 0 > > BuildTwoSided 124 1.0 9.0858e+00 7.2 0.00e+00 0.0 2.7e+04 8.0e+00 0.0e+00 6 0 11 0 0 7 0 14 0 0 0 > > VecSet 16 1.0 5.8370e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecScatterBegin 3 1.0 2.1945e-0269.7 0.00e+00 0.0 1.3e+03 2.6e+04 0.0e+00 0 0 1 0 0 0 0 1 0 0 0 > > VecScatterEnd 3 1.0 2.2460e-0218.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecSetRandom 3 1.0 4.0847e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatMult 3 1.0 9.4907e-02 1.2 4.50e+07 1.1 1.3e+03 2.6e+04 0.0e+00 0 0 1 0 0 0 0 1 0 0 21311 > > MatAssemblyBegin 12 1.0 2.6438e-03235.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatAssemblyEnd 12 1.0 6.6632e-01 2.5 0.00e+00 0.0 2.5e+03 1.3e+04 2.4e+01 0 0 1 0 9 0 0 1 0 9 0 > > MatView 9 1.0 5.3831e-0112.9 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 3 0 0 0 0 4 0 > > Mesh Partition 6 1.0 1.3552e+01 1.0 0.00e+00 0.0 1.0e+05 5.9e+04 3.3e+01 12 0 41 7 13 13 0 52 7 13 0 > > Mesh Migration 6 1.0 1.8341e+01 1.0 0.00e+00 0.0 7.5e+04 1.0e+06 7.2e+01 16 0 31 85 27 17 0 39 86 28 0 > > DMPlexInterp 3 1.0 1.3771e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 12 0 0 0 2 13 0 0 0 2 0 > > DMPlexDistribute 3 1.0 1.0266e+01 1.0 0.00e+00 0.0 4.9e+04 5.9e+04 2.7e+01 9 0 20 3 10 10 0 25 3 11 0 > > DMPlexDistCones 6 1.0 6.9775e+00 1.5 0.00e+00 0.0 1.2e+04 2.3e+06 0.0e+00 5 0 5 32 0 6 0 6 32 0 0 > > DMPlexDistLabels 6 1.0 7.9111e+00 1.0 0.00e+00 0.0 4.0e+04 9.8e+05 6.0e+00 7 0 16 43 2 7 0 21 44 2 0 > > DMPlexDistribOL 3 1.0 2.2335e+01 1.0 0.00e+00 0.0 1.3e+05 6.6e+05 7.8e+01 19 0 53 94 30 21 0 66 95 30 0 > > DMPlexDistField 9 1.0 7.2773e-01 1.0 0.00e+00 0.0 1.7e+04 2.0e+05 6.0e+00 1 0 7 4 2 1 0 9 4 2 0 > > DMPlexDistData 6 1.0 8.0047e+00 9.4 0.00e+00 0.0 8.6e+04 1.2e+04 0.0e+00 6 0 35 1 0 6 0 45 1 0 0 > > DMPlexStratify 19 1.0 1.8531e+01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.9e+01 15 0 0 0 7 16 0 0 0 7 0 > > SFSetGraph 141 1.0 2.2412e+00 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 > > SFBcastBegin 271 1.0 1.1975e+01 2.0 0.00e+00 0.0 1.8e+05 4.8e+05 0.0e+00 9 0 75 98 0 10 0 95100 0 0 > > SFBcastEnd 271 1.0 6.4306e+00 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 > > SFReduceBegin 12 1.0 1.7538e-0112.8 0.00e+00 0.0 4.8e+03 5.9e+04 0.0e+00 0 0 2 0 0 0 0 2 0 0 0 > > SFReduceEnd 12 1.0 2.2638e-01 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > SFFetchOpBegin 3 1.0 9.9087e-0415.6 0.00e+00 0.0 6.3e+02 3.9e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > SFFetchOpEnd 3 1.0 3.6049e-02 6.4 0.00e+00 0.0 6.3e+02 3.9e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > CreateMesh 15 1.0 4.9047e+01 1.0 0.00e+00 0.0 1.8e+05 4.9e+05 1.2e+02 42 0 73 97 44 45 0 92 98 46 0 > > CreateFunctionSpace 3 1.0 4.2819e+01 1.0 0.00e+00 0.0 1.4e+05 6.3e+05 1.2e+02 37 0 56 95 44 40 0 71 97 45 0 > > Mesh: reorder 3 1.0 1.5455e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 1 0 0 0 2 1 0 0 0 2 0 > > Mesh: numbering 3 1.0 1.0627e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 9 0 0 0 2 10 0 0 0 2 0 > > CreateSparsity 3 1.0 2.0243e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 > > MatZeroInitial 3 1.0 2.7938e+00 1.0 0.00e+00 0.0 2.5e+03 1.3e+04 2.7e+01 2 0 1 0 10 3 0 1 0 11 0 > > ParLoopExecute 6 1.0 3.1709e+00 1.2 1.24e+10 1.2 0.0e+00 0.0e+00 0.0e+00 3 47 0 0 0 3100 0 0 0 175069 > > ParLoopset_4 2 1.0 1.1100e-0222.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopHaloEnd 6 1.0 2.9564e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopRednBegin 6 1.0 7.0810e-05 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopRednEnd 6 1.0 6.5088e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopCells 9 1.0 2.9736e+00 1.2 1.24e+10 1.2 0.0e+00 0.0e+00 0.0e+00 2 47 0 0 0 3100 0 0 0 186686 > > ParLoopset_10 2 1.0 1.1411e-03 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopset_16 2 1.0 1.1880e-03 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > > > --- Event Stage 1: P(1) aij matrix > > > > VecScatterBegin 40 1.0 1.1312e-02 8.5 0.00e+00 0.0 1.7e+04 1.4e+04 0.0e+00 0 0 7 0 0 0 0100100 0 0 > > VecScatterEnd 40 1.0 2.6442e-0161.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 10 0 0 0 0 0 > > VecSetRandom 40 1.0 4.4251e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 27 0 0 0 0 0 > > MatMult 40 1.0 4.5745e-01 1.1 2.06e+08 1.1 1.7e+04 1.4e+04 0.0e+00 0 1 7 0 0 28 17100100 0 20423 > > MatAssemblyBegin 3 1.0 2.3842e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatAssemblyEnd 3 1.0 1.8371e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > > MatZeroEntries 1 1.0 5.2531e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatView 2 1.0 1.8248e-012468.9 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 1 5 0 0 0100 0 > > AssembleMat 1 1.0 7.0037e-01 1.0 1.01e+09 1.1 0.0e+00 0.0e+00 2.0e+00 1 4 0 0 1 45 83 0 0100 66009 > > ParLoopExecute 1 1.0 6.7369e-01 1.4 1.01e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 38 83 0 0 0 68623 > > ParLoopHaloEnd 1 1.0 1.3113e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopRednBegin 1 1.0 1.3113e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopRednEnd 1 1.0 1.0967e-05 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopCells 3 1.0 6.7352e-01 1.4 1.01e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 38 83 0 0 0 68641 > > > > --- Event Stage 2: P(2) aij matrix > > > > VecScatterBegin 40 1.0 1.2448e-02 6.3 0.00e+00 0.0 1.7e+04 2.1e+04 0.0e+00 0 0 7 0 0 0 0100100 0 0 > > VecScatterEnd 40 1.0 4.3488e-0156.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 14 0 0 0 0 0 > > VecSetRandom 40 1.0 4.4287e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 22 0 0 0 0 0 > > MatMult 40 1.0 7.8413e-01 1.1 4.04e+08 1.1 1.7e+04 2.1e+04 0.0e+00 1 2 7 0 0 39 21100100 0 23192 > > MatAssemblyBegin 3 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatAssemblyEnd 3 1.0 2.4675e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > > MatZeroEntries 1 1.0 9.4781e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatView 2 1.0 1.4482e-01344.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 1 3 0 0 0100 0 > > AssembleMat 1 1.0 7.5959e-01 1.0 1.57e+09 1.2 0.0e+00 0.0e+00 2.0e+00 1 6 0 0 1 39 79 0 0100 92192 > > ParLoopExecute 1 1.0 7.1835e-01 1.2 1.57e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 6 0 0 0 34 79 0 0 0 97484 > > ParLoopHaloEnd 1 1.0 1.1921e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopRednBegin 1 1.0 1.7881e-05 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopRednEnd 1 1.0 1.4067e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopCells 3 1.0 7.1820e-01 1.2 1.57e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 6 0 0 0 34 79 0 0 0 97505 > > > > --- Event Stage 3: P(3) aij matrix > > > > VecScatterBegin 40 1.0 2.3520e-0210.9 0.00e+00 0.0 1.6e+04 4.2e+04 0.0e+00 0 0 7 1 0 0 0100100 0 0 > > VecScatterEnd 40 1.0 6.6521e-0138.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 0 0 0 0 0 > > VecSetRandom 40 1.0 7.5565e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 16 0 0 0 0 0 > > MatMult 40 1.0 2.0981e+00 1.0 1.19e+09 1.1 1.6e+04 4.2e+04 0.0e+00 2 4 7 1 0 46 11100100 0 25439 > > MatAssemblyBegin 3 1.0 2.8610e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatAssemblyEnd 3 1.0 5.6094e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > > MatZeroEntries 1 1.0 2.9610e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > > MatView 2 1.0 2.8071e-01958.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 1 3 0 0 0100 0 > > AssembleMat 1 1.0 1.7038e+00 1.0 9.94e+09 1.2 0.0e+00 0.0e+00 2.0e+00 1 37 0 0 1 38 89 0 0100 257591 > > ParLoopExecute 1 1.0 1.6101e+00 1.2 9.94e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 37 0 0 0 32 89 0 0 0 272582 > > ParLoopHaloEnd 1 1.0 1.4067e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopRednBegin 1 1.0 1.7166e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopRednEnd 1 1.0 1.4067e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ParLoopCells 3 1.0 1.6099e+00 1.2 9.94e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 37 0 0 0 32 89 0 0 0 272617 > > ------------------------------------------------------------------------------------------------------------------------ > > > > Memory usage is given in bytes: > > > > Object Type Creations Destructions Memory Descendants' Mem. > > Reports information only for process 0. > > > > --- Event Stage 0: Main Stage > > > > Container 12 11 6776 0. > > Viewer 4 0 0 0. > > PetscRandom 3 3 2058 0. > > Index Set 1095 1085 1383616 0. > > IS L to G Mapping 15 14 204830392 0. > > Section 222 209 158840 0. > > Vector 31 28 41441632 0. > > Vector Scatter 3 2 2416 0. > > Matrix 22 18 131705576 0. > > Distributed Mesh 40 37 182200 0. > > GraphPartitioner 19 18 11808 0. > > Star Forest Bipartite Graph 206 200 178256 0. > > Discrete System 40 37 34336 0. > > > > --- Event Stage 1: P(1) aij matrix > > > > PetscRandom 40 40 27440 0. > > > > --- Event Stage 2: P(2) aij matrix > > > > PetscRandom 40 40 27440 0. > > > > --- Event Stage 3: P(3) aij matrix > > > > PetscRandom 40 40 27440 0. > > ======================================================================================================================== > > Average time to get PetscTime(): 0. > > Average time for MPI_Barrier(): 1.06335e-05 > > Average time for zero size MPI_Send(): 1.41561e-06 > > #PETSc Option Table entries: > > --dimension 3 > > --output-file poisson-matvecs.csv > > --problem poisson > > -log_view > > -mat_view ::ascii_info > > #End of PETSc Option Table entries > > Compiled without FORTRAN kernels > > Compiled with full precision matrices (default) > > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8 > > Configure options: --COPTFLAGS="-march=ivybridge -O3" --CXXOPTFLAGS="-march=ivybridge -O3" --FOPTFLAGS="-march=ivybridge -O3" --PETSC_ARCH=petsc-gnu51-ivybridge-int64 --download-exodusii --download-hypre --download-metis --download-netcdf --download-parmetis --download-sowing=1 --known-bits-per-byte=8 --known-has-attribute-aligned=1 --known-level1-dcache-assoc=8 --known-level1-dcache-linesize=64 --known-level1-dcache-size=32768 --known-memcmp-ok=1 --known-mpi-c-double-complex=1 --known-mpi-int64_t=1 --known-mpi-long-double=1 --known-mpi-shared-libraries=1 --known-sdot-returns-double=0 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8 --known-sizeof-void-p=8 --known-snrm2-returns-double=0 --prefix=/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64 --with-64-bit-indices=1 --with-batch=1 --with-blas-lapack-lib="-L/opt/cray/libsci/16.03.1/GNU/5.1/x86_64/lib -lsci_gnu_mp" --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn --with-fortranlib-autodetect=0 --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.14/GNU/5.1 --with-hdf5=1 --with-make-np=4 --with-pic=1 --with-shared-libraries=1 --with-x=0 --download-eigen > > ----------------------------------------- > > Libraries compiled on Tue Feb 14 12:07:09 2017 on eslogin003 > > Machine characteristics: Linux-3.0.101-0.47.86.1.11753.0.PTF-default-x86_64-with-SuSE-11-x86_64 > > Using PETSc directory: /home2/n01/n01/lmn01/src/petsc > > Using PETSc arch: petsc-gnu51-ivybridge-int64 > > ----------------------------------------- > > > > Using C compiler: cc -fPIC -march=ivybridge -O3 ${COPTFLAGS} ${CFLAGS} > > Using Fortran compiler: ftn -fPIC -march=ivybridge -O3 ${FOPTFLAGS} ${FFLAGS} > > ----------------------------------------- > > > > Using include paths: -I/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/include -I/home2/n01/n01/lmn01/src/petsc/include -I/home2/n01/n01/lmn01/src/petsc/include -I/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/include -I/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/include -I/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/include/eigen3 -I/opt/cray/hdf5-parallel/1.8.14/GNU/5.1/include > > ----------------------------------------- > > > > Using C linker: cc > > Using Fortran linker: ftn > > Using libraries: -Wl,-rpath,/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/lib -L/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/lib -lpetsc -Wl,-rpath,/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/lib -L/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/lib -lHYPRE -lparmetis -lmetis -lexoIIv2for -lexodus -lnetcdf -Wl,-rpath,/opt/cray/hdf5-parallel/1.8.14/GNU/5.1/lib -L/opt/cray/hdf5-parallel/1.8.14/GNU/5.1/lib -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lssl -lcrypto -ldl > > ----------------------------------------- > > > > Application 28632506 resources: utime ~4100s, stime ~428s, Rss ~2685552, inblocks ~2935062, outblocks ~42464 > > -------------------------------------------------------------------------------- > > > > Resources requested: ncpus=48,place=free,walltime=00:20:00 > > Resources allocated: cpupercent=0,cput=00:00:02,mem=8980kb,ncpus=48,vmem=172968kb,walltime=00:02:20 > > > > *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** > > *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** > > *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** > > *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** > > -------------------------------------------------------------------------------- > > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: not available URL: From rupp at iue.tuwien.ac.at Fri Sep 29 09:08:18 2017 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Fri, 29 Sep 2017 09:08:18 -0500 Subject: [petsc-users] Understanding matmult memory performance In-Reply-To: <8D683AE4-75B9-40AF-97E6-6801657DE095@imperial.ac.uk> References: <8D683AE4-75B9-40AF-97E6-6801657DE095@imperial.ac.uk> Message-ID: <2d3b433e-f812-f223-aa92-64be25e829fa@iue.tuwien.ac.at> Hi Lawrence, according to https://ark.intel.com/products/75283/Intel-Xeon-Processor-E5-2697-v2-30M-Cache-2_70-GHz you get 59.7 GB/sec of peak memory bandwidth per CPU, so you should get about 240 GB/sec for your two-node system. If you use PETSc's `make streams`, then processor placement may - unfortunately - not be ideal and hence underestimating the achievable performance. Have a look at the new PETSc 3.8 manual [1], Chapter 14, where Richard and I nailed down some of these performance aspects. Best regards, Karli [1] http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf On 09/29/2017 06:19 AM, Lawrence Mitchell wrote: > Dear all, > > I'm attempting to understand some results I'm getting for matmult performance. In particular, it looks like I'm obtaining timings that suggest that I'm getting more main memory bandwidth than I think is possible. > > The run setup is using 2 24 core (dual socket) ivybridge nodes (Xeon E5-2697 v2). The specced main memory bandwidth is 85.3 GB/s per node, and I measure a STREAM triad bandwidth using 48 MPI processes (two nodes) of 148.2 GB/s. The last level cache is 30MB (shared between 12 cores) > > The matrix I'm using is respectively a P1, P2, and P3 discretisation of the Laplacian on a regular tetrahedral grid. > > The matrix sizes are respectively: > > P1: > Mat Object: 48 MPI processes > type: mpiaij > rows=8120601, cols=8120601 > total: nonzeros=120841801, allocated nonzeros=120841801 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > P2: > Mat Object: 48 MPI processes > type: mpiaij > rows=8120601, cols=8120601 > total: nonzeros=231382401, allocated nonzeros=231382401 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > P3: > Mat Object: 48 MPI processes > type: mpiaij > rows=13997521, cols=13997521 > total: nonzeros=674173201, allocated nonzeros=674173201 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > Both sizeof(PetscScalar) and sizeof(PetscInt) are 8 bytes. > > Ignoring data for vector and row indices, then, for a matmult I need to move 16*nonzeros bytes. > > MatMults take, respectively: > > P1: 0.0114362s > P2: 0.0196032s > P3: 0.0524525s > > So the estimated achieved memory bandwidth is: > > P1: 120841801 * 16 / 0.0114362 = 157.45GB/s > P2: 231382401 * 16 / 0.0196032 = 175.88GB/s > P3: 674173201 * 16 / 0.0524525 = 191.52GB/s > > So all of those numbers are higher than the stream bandwidth, and the P2 and P3 numbers are higher than the spec sheet bandwidth. > > I don't think PETSc is doing anything magic, but hints appreciated, it would be nice to explain this. > > Cheers, > > Lawrence > > Full -log_view output: > > -------------------------------------------------------------------------------- > *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** > *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** > *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** > *** lmn01 Job: 4820277.sdb started: 29/09/17 11:56:03 host: mom1 *** > > -------------------------------------------------------------------------------- > Int Type has 8 bytes, Scalar Type has 8 bytes > > P1: > Mat Object: 48 MPI processes > type: mpiaij > rows=8120601, cols=8120601 > total: nonzeros=120841801, allocated nonzeros=120841801 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > P2: > Mat Object: 48 MPI processes > type: mpiaij > rows=8120601, cols=8120601 > total: nonzeros=231382401, allocated nonzeros=231382401 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > P3: > Mat Object: 48 MPI processes > type: mpiaij > rows=13997521, cols=13997521 > total: nonzeros=674173201, allocated nonzeros=674173201 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > ************************************************************************************************************************ > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** > ************************************************************************************************************************ > > ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- > > profile-matvec.py on a petsc-gnu51-ivybridge-int64 named nid00013 with 48 processors, by lmn01 Fri Sep 29 11:58:21 2017 > Using Petsc Development GIT revision: v3.7.5-3014-g413f72f GIT Date: 2017-02-05 17:50:57 -0600 > > Max Max/Min Avg Total > Time (sec): 1.150e+02 1.00000 1.150e+02 > Objects: 1.832e+03 1.50534 1.269e+03 > Flops: 2.652e+10 1.16244 2.486e+10 1.193e+12 > Flops/sec: 2.306e+08 1.16244 2.162e+08 1.038e+10 > MPI Messages: 1.021e+04 3.00279 5.091e+03 2.444e+05 > MPI Message Lengths: 3.314e+09 1.97310 3.697e+05 9.035e+10 > MPI Reductions: 2.630e+02 1.00000 > > Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length N --> 2N flops > and VecAXPY() for complex vectors of length N --> 8N flops > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts %Total Avg %Total counts %Total > 0: Main Stage: 1.0701e+02 93.1% 5.5715e+11 46.7% 1.942e+05 79.4% 3.644e+05 98.6% 2.560e+02 97.3% > 1: P(1) aij matrix: 1.5561e+00 1.4% 5.5574e+10 4.7% 1.688e+04 6.9% 9.789e+02 0.3% 2.000e+00 0.8% > 2: P(2) aij matrix: 1.9378e+00 1.7% 8.8214e+10 7.4% 1.688e+04 6.9% 1.483e+03 0.4% 2.000e+00 0.8% > 3: P(3) aij matrix: 4.4890e+00 3.9% 4.9225e+11 41.3% 1.648e+04 6.7% 2.829e+03 0.8% 2.000e+00 0.8% > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flops: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > Avg. len: average message length (bytes) > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). > %T - percent time in this phase %F - percent flops in this phase > %M - percent messages in this phase %L - percent message lengths in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flops --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > PetscBarrier 4 1.0 2.7271e+00 1.0 0.00e+00 0.0 3.8e+03 2.4e+01 2.0e+01 2 0 2 0 8 3 0 2 0 8 0 > BuildTwoSided 124 1.0 9.0858e+00 7.2 0.00e+00 0.0 2.7e+04 8.0e+00 0.0e+00 6 0 11 0 0 7 0 14 0 0 0 > VecSet 16 1.0 5.8370e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 3 1.0 2.1945e-0269.7 0.00e+00 0.0 1.3e+03 2.6e+04 0.0e+00 0 0 1 0 0 0 0 1 0 0 0 > VecScatterEnd 3 1.0 2.2460e-0218.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSetRandom 3 1.0 4.0847e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatMult 3 1.0 9.4907e-02 1.2 4.50e+07 1.1 1.3e+03 2.6e+04 0.0e+00 0 0 1 0 0 0 0 1 0 0 21311 > MatAssemblyBegin 12 1.0 2.6438e-03235.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 12 1.0 6.6632e-01 2.5 0.00e+00 0.0 2.5e+03 1.3e+04 2.4e+01 0 0 1 0 9 0 0 1 0 9 0 > MatView 9 1.0 5.3831e-0112.9 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 3 0 0 0 0 4 0 > Mesh Partition 6 1.0 1.3552e+01 1.0 0.00e+00 0.0 1.0e+05 5.9e+04 3.3e+01 12 0 41 7 13 13 0 52 7 13 0 > Mesh Migration 6 1.0 1.8341e+01 1.0 0.00e+00 0.0 7.5e+04 1.0e+06 7.2e+01 16 0 31 85 27 17 0 39 86 28 0 > DMPlexInterp 3 1.0 1.3771e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 12 0 0 0 2 13 0 0 0 2 0 > DMPlexDistribute 3 1.0 1.0266e+01 1.0 0.00e+00 0.0 4.9e+04 5.9e+04 2.7e+01 9 0 20 3 10 10 0 25 3 11 0 > DMPlexDistCones 6 1.0 6.9775e+00 1.5 0.00e+00 0.0 1.2e+04 2.3e+06 0.0e+00 5 0 5 32 0 6 0 6 32 0 0 > DMPlexDistLabels 6 1.0 7.9111e+00 1.0 0.00e+00 0.0 4.0e+04 9.8e+05 6.0e+00 7 0 16 43 2 7 0 21 44 2 0 > DMPlexDistribOL 3 1.0 2.2335e+01 1.0 0.00e+00 0.0 1.3e+05 6.6e+05 7.8e+01 19 0 53 94 30 21 0 66 95 30 0 > DMPlexDistField 9 1.0 7.2773e-01 1.0 0.00e+00 0.0 1.7e+04 2.0e+05 6.0e+00 1 0 7 4 2 1 0 9 4 2 0 > DMPlexDistData 6 1.0 8.0047e+00 9.4 0.00e+00 0.0 8.6e+04 1.2e+04 0.0e+00 6 0 35 1 0 6 0 45 1 0 0 > DMPlexStratify 19 1.0 1.8531e+01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.9e+01 15 0 0 0 7 16 0 0 0 7 0 > SFSetGraph 141 1.0 2.2412e+00 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 > SFBcastBegin 271 1.0 1.1975e+01 2.0 0.00e+00 0.0 1.8e+05 4.8e+05 0.0e+00 9 0 75 98 0 10 0 95100 0 0 > SFBcastEnd 271 1.0 6.4306e+00 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 > SFReduceBegin 12 1.0 1.7538e-0112.8 0.00e+00 0.0 4.8e+03 5.9e+04 0.0e+00 0 0 2 0 0 0 0 2 0 0 0 > SFReduceEnd 12 1.0 2.2638e-01 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > SFFetchOpBegin 3 1.0 9.9087e-0415.6 0.00e+00 0.0 6.3e+02 3.9e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > SFFetchOpEnd 3 1.0 3.6049e-02 6.4 0.00e+00 0.0 6.3e+02 3.9e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > CreateMesh 15 1.0 4.9047e+01 1.0 0.00e+00 0.0 1.8e+05 4.9e+05 1.2e+02 42 0 73 97 44 45 0 92 98 46 0 > CreateFunctionSpace 3 1.0 4.2819e+01 1.0 0.00e+00 0.0 1.4e+05 6.3e+05 1.2e+02 37 0 56 95 44 40 0 71 97 45 0 > Mesh: reorder 3 1.0 1.5455e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 1 0 0 0 2 1 0 0 0 2 0 > Mesh: numbering 3 1.0 1.0627e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 9 0 0 0 2 10 0 0 0 2 0 > CreateSparsity 3 1.0 2.0243e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 > MatZeroInitial 3 1.0 2.7938e+00 1.0 0.00e+00 0.0 2.5e+03 1.3e+04 2.7e+01 2 0 1 0 10 3 0 1 0 11 0 > ParLoopExecute 6 1.0 3.1709e+00 1.2 1.24e+10 1.2 0.0e+00 0.0e+00 0.0e+00 3 47 0 0 0 3100 0 0 0 175069 > ParLoopset_4 2 1.0 1.1100e-0222.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopHaloEnd 6 1.0 2.9564e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednBegin 6 1.0 7.0810e-05 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednEnd 6 1.0 6.5088e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopCells 9 1.0 2.9736e+00 1.2 1.24e+10 1.2 0.0e+00 0.0e+00 0.0e+00 2 47 0 0 0 3100 0 0 0 186686 > ParLoopset_10 2 1.0 1.1411e-03 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopset_16 2 1.0 1.1880e-03 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > --- Event Stage 1: P(1) aij matrix > > VecScatterBegin 40 1.0 1.1312e-02 8.5 0.00e+00 0.0 1.7e+04 1.4e+04 0.0e+00 0 0 7 0 0 0 0100100 0 0 > VecScatterEnd 40 1.0 2.6442e-0161.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 10 0 0 0 0 0 > VecSetRandom 40 1.0 4.4251e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 27 0 0 0 0 0 > MatMult 40 1.0 4.5745e-01 1.1 2.06e+08 1.1 1.7e+04 1.4e+04 0.0e+00 0 1 7 0 0 28 17100100 0 20423 > MatAssemblyBegin 3 1.0 2.3842e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 3 1.0 1.8371e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > MatZeroEntries 1 1.0 5.2531e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatView 2 1.0 1.8248e-012468.9 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 1 5 0 0 0100 0 > AssembleMat 1 1.0 7.0037e-01 1.0 1.01e+09 1.1 0.0e+00 0.0e+00 2.0e+00 1 4 0 0 1 45 83 0 0100 66009 > ParLoopExecute 1 1.0 6.7369e-01 1.4 1.01e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 38 83 0 0 0 68623 > ParLoopHaloEnd 1 1.0 1.3113e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednBegin 1 1.0 1.3113e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednEnd 1 1.0 1.0967e-05 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopCells 3 1.0 6.7352e-01 1.4 1.01e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 38 83 0 0 0 68641 > > --- Event Stage 2: P(2) aij matrix > > VecScatterBegin 40 1.0 1.2448e-02 6.3 0.00e+00 0.0 1.7e+04 2.1e+04 0.0e+00 0 0 7 0 0 0 0100100 0 0 > VecScatterEnd 40 1.0 4.3488e-0156.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 14 0 0 0 0 0 > VecSetRandom 40 1.0 4.4287e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 22 0 0 0 0 0 > MatMult 40 1.0 7.8413e-01 1.1 4.04e+08 1.1 1.7e+04 2.1e+04 0.0e+00 1 2 7 0 0 39 21100100 0 23192 > MatAssemblyBegin 3 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 3 1.0 2.4675e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > MatZeroEntries 1 1.0 9.4781e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatView 2 1.0 1.4482e-01344.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 1 3 0 0 0100 0 > AssembleMat 1 1.0 7.5959e-01 1.0 1.57e+09 1.2 0.0e+00 0.0e+00 2.0e+00 1 6 0 0 1 39 79 0 0100 92192 > ParLoopExecute 1 1.0 7.1835e-01 1.2 1.57e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 6 0 0 0 34 79 0 0 0 97484 > ParLoopHaloEnd 1 1.0 1.1921e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednBegin 1 1.0 1.7881e-05 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednEnd 1 1.0 1.4067e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopCells 3 1.0 7.1820e-01 1.2 1.57e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 6 0 0 0 34 79 0 0 0 97505 > > --- Event Stage 3: P(3) aij matrix > > VecScatterBegin 40 1.0 2.3520e-0210.9 0.00e+00 0.0 1.6e+04 4.2e+04 0.0e+00 0 0 7 1 0 0 0100100 0 0 > VecScatterEnd 40 1.0 6.6521e-0138.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 0 0 0 0 0 > VecSetRandom 40 1.0 7.5565e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 16 0 0 0 0 0 > MatMult 40 1.0 2.0981e+00 1.0 1.19e+09 1.1 1.6e+04 4.2e+04 0.0e+00 2 4 7 1 0 46 11100100 0 25439 > MatAssemblyBegin 3 1.0 2.8610e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 3 1.0 5.6094e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > MatZeroEntries 1 1.0 2.9610e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > MatView 2 1.0 2.8071e-01958.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 1 3 0 0 0100 0 > AssembleMat 1 1.0 1.7038e+00 1.0 9.94e+09 1.2 0.0e+00 0.0e+00 2.0e+00 1 37 0 0 1 38 89 0 0100 257591 > ParLoopExecute 1 1.0 1.6101e+00 1.2 9.94e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 37 0 0 0 32 89 0 0 0 272582 > ParLoopHaloEnd 1 1.0 1.4067e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednBegin 1 1.0 1.7166e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopRednEnd 1 1.0 1.4067e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ParLoopCells 3 1.0 1.6099e+00 1.2 9.94e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 37 0 0 0 32 89 0 0 0 272617 > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Container 12 11 6776 0. > Viewer 4 0 0 0. > PetscRandom 3 3 2058 0. > Index Set 1095 1085 1383616 0. > IS L to G Mapping 15 14 204830392 0. > Section 222 209 158840 0. > Vector 31 28 41441632 0. > Vector Scatter 3 2 2416 0. > Matrix 22 18 131705576 0. > Distributed Mesh 40 37 182200 0. > GraphPartitioner 19 18 11808 0. > Star Forest Bipartite Graph 206 200 178256 0. > Discrete System 40 37 34336 0. > > --- Event Stage 1: P(1) aij matrix > > PetscRandom 40 40 27440 0. > > --- Event Stage 2: P(2) aij matrix > > PetscRandom 40 40 27440 0. > > --- Event Stage 3: P(3) aij matrix > > PetscRandom 40 40 27440 0. > ======================================================================================================================== > Average time to get PetscTime(): 0. > Average time for MPI_Barrier(): 1.06335e-05 > Average time for zero size MPI_Send(): 1.41561e-06 > #PETSc Option Table entries: > --dimension 3 > --output-file poisson-matvecs.csv > --problem poisson > -log_view > -mat_view ::ascii_info > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8 > Configure options: --COPTFLAGS="-march=ivybridge -O3" --CXXOPTFLAGS="-march=ivybridge -O3" --FOPTFLAGS="-march=ivybridge -O3" --PETSC_ARCH=petsc-gnu51-ivybridge-int64 --download-exodusii --download-hypre --download-metis --download-netcdf --download-parmetis --download-sowing=1 --known-bits-per-byte=8 --known-has-attribute-aligned=1 --known-level1-dcache-assoc=8 --known-level1-dcache-linesize=64 --known-level1-dcache-size=32768 --known-memcmp-ok=1 --known-mpi-c-double-complex=1 --known-mpi-int64_t=1 --known-mpi-long-double=1 --known-mpi-shared-libraries=1 --known-sdot-returns-double=0 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8 --known-sizeof-void-p=8 --known-snrm2-returns-double=0 --prefix=/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64 --with-64-bit-indices=1 --with-batch=1 --with-blas-lapack-lib="-L/opt/cray/libsci/16.03.1/GNU/5.1/x86_64/lib -lsci_gnu_mp" --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn --with-fortranlib-autodetect=0 --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.14/GNU/5.1 --with-hdf5=1 --with-make-np=4 --with-pic=1 --with-shared-libraries=1 --with-x=0 --download-eigen > ----------------------------------------- > Libraries compiled on Tue Feb 14 12:07:09 2017 on eslogin003 > Machine characteristics: Linux-3.0.101-0.47.86.1.11753.0.PTF-default-x86_64-with-SuSE-11-x86_64 > Using PETSc directory: /home2/n01/n01/lmn01/src/petsc > Using PETSc arch: petsc-gnu51-ivybridge-int64 > ----------------------------------------- > > Using C compiler: cc -fPIC -march=ivybridge -O3 ${COPTFLAGS} ${CFLAGS} > Using Fortran compiler: ftn -fPIC -march=ivybridge -O3 ${FOPTFLAGS} ${FFLAGS} > ----------------------------------------- > > Using include paths: -I/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/include -I/home2/n01/n01/lmn01/src/petsc/include -I/home2/n01/n01/lmn01/src/petsc/include -I/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/include -I/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/include -I/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/include/eigen3 -I/opt/cray/hdf5-parallel/1.8.14/GNU/5.1/include > ----------------------------------------- > > Using C linker: cc > Using Fortran linker: ftn > Using libraries: -Wl,-rpath,/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/lib -L/home2/n01/n01/lmn01/src/petsc/petsc-gnu51-ivybridge-int64/lib -lpetsc -Wl,-rpath,/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/lib -L/work/n01/n01/lmn01/petsc-gnu51-ivybridge-int64/lib -lHYPRE -lparmetis -lmetis -lexoIIv2for -lexodus -lnetcdf -Wl,-rpath,/opt/cray/hdf5-parallel/1.8.14/GNU/5.1/lib -L/opt/cray/hdf5-parallel/1.8.14/GNU/5.1/lib -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lssl -lcrypto -ldl > ----------------------------------------- > > Application 28632506 resources: utime ~4100s, stime ~428s, Rss ~2685552, inblocks ~2935062, outblocks ~42464 > -------------------------------------------------------------------------------- > > Resources requested: ncpus=48,place=free,walltime=00:20:00 > Resources allocated: cpupercent=0,cput=00:00:02,mem=8980kb,ncpus=48,vmem=172968kb,walltime=00:02:20 > > *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** > *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** > *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** > *** lmn01 Job: 4820277.sdb ended: 29/09/17 11:58:22 queue: S4808886 *** > -------------------------------------------------------------------------------- > From khaipham at mit.edu Fri Sep 29 09:12:05 2017 From: khaipham at mit.edu (Khai Pham) Date: Fri, 29 Sep 2017 14:12:05 +0000 Subject: [petsc-users] Hypre PILUT preconditioner Message-ID: <69F29A8D72136041A5F665D423C355148B05835D@OC11EXPO29.exchange.mit.edu> Hi, I try to use the parallel ILU which is provided by hypre library. As from some discussion from other petsc users, it mentioned that euclid preconditioner from hypre was removed from petsc. So, I tried the ilu with threshold "pilut", it is okay as I run in sequential, but it has an error as I tried with parallel : parilut.c:274: hypre_ComputeCommInfo: Assertion `cinfo->incolind != ((void *)0)' failed. I tried with other preconditioner from hypre as boomeramg, and it works fine. I am wondering that causes this issue with "pilut". I am aware that other petsc developers mention about some scaling issue with parallel ILU, but I would like to try it for my problem. Please let me know if you have any idea about this error. Thanks! Best, Khai -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Sep 29 09:23:02 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 29 Sep 2017 07:23:02 -0700 Subject: [petsc-users] Hypre PILUT preconditioner In-Reply-To: <69F29A8D72136041A5F665D423C355148B05835D@OC11EXPO29.exchange.mit.edu> References: <69F29A8D72136041A5F665D423C355148B05835D@OC11EXPO29.exchange.mit.edu> Message-ID: <0769B140-DD82-4680-8413-EABA2889BFE6@mcs.anl.gov> You need to contact hypre team. We don't think much of parallel ILU and can't support it ourselves. Barry > On Sep 29, 2017, at 7:12 AM, Khai Pham wrote: > > Hi, > > I try to use the parallel ILU which is provided by hypre library. As from some discussion from other petsc users, it mentioned that euclid preconditioner from hypre was removed from petsc. So, I tried the ilu with threshold "pilut", it is okay as I run in sequential, but it has an error as I tried with parallel : > > parilut.c:274: hypre_ComputeCommInfo: Assertion `cinfo->incolind != ((void *)0)' failed. > > I tried with other preconditioner from hypre as boomeramg, and it works fine. I am wondering that causes this issue with "pilut". I am aware that other petsc developers mention about some scaling issue with parallel ILU, but I would like to try it for my problem. Please let me know if you have any idea about this error. Thanks! > > Best, > Khai From lawrence.mitchell at imperial.ac.uk Fri Sep 29 09:24:23 2017 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Fri, 29 Sep 2017 15:24:23 +0100 Subject: [petsc-users] Understanding matmult memory performance In-Reply-To: <20170929140501.ukincl2vdohl7mok@gatech.edu> References: <8D683AE4-75B9-40AF-97E6-6801657DE095@imperial.ac.uk> <20170929130447.kmvqn3yintmnqrcw@gatech.edu> <20170929140501.ukincl2vdohl7mok@gatech.edu> Message-ID: <53A3BC17-1C6C-4D44-931B-EDDECC2EA54D@imperial.ac.uk> > On 29 Sep 2017, at 15:05, Tobin Isaac wrote: > > On Fri, Sep 29, 2017 at 09:04:47AM -0400, Tobin Isaac wrote: >> On Fri, Sep 29, 2017 at 12:19:54PM +0100, Lawrence Mitchell wrote: >>> Dear all, >>> >>> I'm attempting to understand some results I'm getting for matmult performance. In particular, it looks like I'm obtaining timings that suggest that I'm getting more main memory bandwidth than I think is possible. >>> >>> The run setup is using 2 24 core (dual socket) ivybridge nodes (Xeon E5-2697 v2). The specced main memory bandwidth is 85.3 GB/s per node, and I measure a STREAM triad bandwidth using 48 MPI processes (two nodes) of 148.2 GB/s. The last level cache is 30MB (shared between 12 cores) >> >> One thought: triad has a 1:2 write:read ratio, but with your MatMult() >> for P3 you would have about 1:50. Unless triad used nontemporal >> stores, the reported bandwidth from triad will be about 3./4. of the >> bandwidth available to pure streaming reads, so maybe you actually >> have ~197 GB/s of read bandwidth available. MatMult() would still be >> doing suspiciously well, but it would be within the measurements. How >> confident are you in the specced bandwidth? I thought I was quite confident, but as you note: > Are you running on archer? I found one site [1] that lists the > bandwidth you gave, which corresponds to DDR3-1333, but other sites > [2] all say the nodes have DDR3-1833, in which case you would be > getting about 80% of spec bandwidth. > > [1]: https://www.archer.ac.uk/documentation/best-practice-guide/arch.php > [2]: https://www.epcc.ed.ac.uk/blog/2013/11/20/archer-next-national-hpc-service-academic-research I am, yes. I will write to them to confirm, and get them to change their website! Yeah, 85.3 was computed via the assumption of DDR3-1333, whereas DDR3-1866 gives me 102.4. Phew :). Karl wrote: > according to > https://ark.intel.com/products/75283/Intel-Xeon-Processor-E5-2697-v2-30M-Cache-2_70-GHz > you get 59.7 GB/sec of peak memory bandwidth per CPU, so you should get about 240 GB/sec for your two-node system. That's assuming I have DDR3-2133 RAM chips, but as noted above, it looks like the nodes probably have 1866 RAM. Giving 204.8 GB/s. > If you use PETSc's `make streams`, then processor placement may - unfortunately - not be ideal and hence underestimating the achievable performance. Have a look at the new PETSc 3.8 manual [1], Chapter 14, where Richard and I nailed down some of these performance aspects. Thanks, I am using make streams (or rather, just running the MPIVersion by hand). I have long been led to believe that Cray's aprun does a reasonable job of process placement (and pinning) for pure MPI jobs, so I have just trusted that. On another note, I modified the MPIVersion to not just report the TRIAD number, and got: Copy: 91467.3281 Rate (MB/s) Scale: 63774.9615 Rate (MB/s) Add: 73994.6106 Rate (MB/s) Triad: 73564.8991 Rate (MB/s) Inspecting the assembly, none of these are using non-temporal stores, but the copy has been turned into a call to memcpy (which might be). In any case, those numbers would have led me to question the website-stated spec sheet numbers sooner. Thanks all, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: Message signed with OpenPGP URL: From bsmith at mcs.anl.gov Fri Sep 29 09:37:04 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 29 Sep 2017 07:37:04 -0700 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> Message-ID: <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> 1) The test is definitely in the wrong place. If we are testing and erroring if using Cholesky and mat is not marked as symmetric or hermitian the test should be in MatGetFactor() not in a particular implementation. 2) It is a tough call if we should do this check or not. There are good arguments in both directions. One thing we could do is if the matrix is not marked as symmetric/hermitian is we could just check at that point (but expensive) or we could just check in debug mode. But I think we should require the user to set the flag (note for SBAIJ the flag for symmetric (or hermitian? which one) should be automatically set at creation). Hong can you move the test up to the MatGetFactor() level? Thanks Barry > On Sep 28, 2017, at 11:35 PM, Stefano Zampini wrote: > > Hong, > > I personally believe that commit https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 should be reverted. > I agree on the fact that when the user sets an option (the hermitian one in this case) and that feature is not supported we should throw an error (https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7) , but I don't like the fact that the user is forced to set on option to use a feature that he already requested (as in https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171). > > Barry, what do you think? > > 2017-09-29 5:28 GMT+03:00 Hong : > Greg: > Thanks so much for the detailed response. I am glad to know the reason behind it--hopefully we eventually figure out why the solvers have this behavior! Hong, I really appreciate you working on a patch to throw an error in this case. It definitely bit me and some people using my code... Hopefully it doesn't happen to anyone else! :) > > I added an error flag for using MUMPS Cholesky factorization on Hermitian matrix. See branch hzhang/mumps-HermitianCholesky-errflag > https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7 > > Jose, > PETSc does not support Cholesky for Hermitian matrix. > > The linear solver table probably needs to be updated. I have tried several Cholesky solvers. With mkl_pardiso I get an explicit error message that it does not support Cholesky with complex scalars. The rest (PETSc, MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is not related to your matrix, nor to shift-and-invert in SLEPc. I tried with ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works in complex scalars, but the matrix is real. As soon as you add complex entries Cholesky fails, for instance adding this: > ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > In this case, you must call > MatSetOption(A,MAT_HERMITIAN,PETSC_TRUE); > > Then, petsc will throw an error for '-pc_type cholesky'. > > I don't know if it is a bug or that the algorithm cannot support complex Hermitian matrices. Maybe Hong can confirm any of these. In the latter case, I agree that all packages should give an error message, as mkl_pardiso does. > > I also add an error flag for Cholesky/ICC if user does not set > MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE) for aij matrix. > See https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 > > Let me know if you have any comments about this fix. > > Hong > > > El 25 sept 2017, a las 7:17, Greg Meyer escribi?: > > > > Hi all, > > > > Hong--looking at your link, there may be no special algorithm for Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like it would any matrix. Furthermore it appears that Cholesky of complex matrices is supported from this link: https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html > > > > So do you or anyone have any idea why I get incorrect eigenvalues? > > > > Thanks, > > Greg > > > > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer wrote: > > Ok, thanks. It seems that PETSc clearly should throw an error in this case instead of just giving incorrect answers? I am surprised that it does not throw an error... > > > > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: > > Greg : > > Yes, they are Hermitian. > > > > PETSc does not support Cholesky factorization for Hermitian. > > It seems mumps does not support Hermitian either > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-November/027541.html > > > > Hong > > > > > > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: > > Greg: > > > > OK, the difference is whether LU or Cholesky factorization is used. But I would hope that neither one should give incorrect eigenvalues, and when I run with the latter it does! > > Are your matrices symmetric/Hermitian? > > Hong > > > > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: > > Gregory : > > Use '-eps_view' for both runs to check the algorithms being used. > > Hong > > > > Hi all, > > > > I'm using shift-invert with EPS to solve for eigenvalues. I find that if I do only > > > > ... > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > ... > > > > in my code I get correct eigenvalues. But if I do > > > > ... > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); > > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); > > ... > > > > the eigenvalues found by EPS are completely wrong! Somehow I thought I was supposed to do the latter, from the examples etc, but I guess that was not correct? I attach the full piece of test code and a test matrix. > > > > Best, > > Greg > > > > > > > > > -- > Stefano From gregory.meyer at gmail.com Fri Sep 29 09:55:09 2017 From: gregory.meyer at gmail.com (Greg Meyer) Date: Fri, 29 Sep 2017 14:55:09 +0000 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> Message-ID: FYI my perspective as a user--something that really tricked me was that after setting the solver to Hermitian problem, the algorithm returned real eigenvalues so they seemed OK. When I turned off Hermitian as I was trying to debug, the eigenvalues were not at all just real, and thus it was clear that they were wrong. So I think the check at least when Hermitian is set is very important, since by construction real eigenvalues are returned. On Fri, Sep 29, 2017, 7:37 AM Barry Smith wrote: > > 1) The test is definitely in the wrong place. If we are testing and > erroring if using Cholesky and mat is not marked as symmetric or hermitian > the test should be in MatGetFactor() not in a particular implementation. > > 2) It is a tough call if we should do this check or not. There are good > arguments in both directions. > > One thing we could do is if the matrix is not marked as > symmetric/hermitian is we could just check at that point (but expensive) or > we could just check in debug mode. > > But I think we should require the user to set the flag (note for SBAIJ > the flag for symmetric (or hermitian? which one) should be automatically > set at creation). > > Hong can you move the test up to the MatGetFactor() level? > > Thanks > Barry > > > > On Sep 28, 2017, at 11:35 PM, Stefano Zampini > wrote: > > > > Hong, > > > > I personally believe that commit > https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 > should be reverted. > > I agree on the fact that when the user sets an option (the hermitian one > in this case) and that feature is not supported we should throw an error ( > https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7) > , but I don't like the fact that the user is forced to set on option to use > a feature that he already requested (as in > https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 > ). > > > > Barry, what do you think? > > > > 2017-09-29 5:28 GMT+03:00 Hong : > > Greg: > > Thanks so much for the detailed response. I am glad to know the reason > behind it--hopefully we eventually figure out why the solvers have this > behavior! Hong, I really appreciate you working on a patch to throw an > error in this case. It definitely bit me and some people using my code... > Hopefully it doesn't happen to anyone else! :) > > > > I added an error flag for using MUMPS Cholesky factorization on > Hermitian matrix. See branch hzhang/mumps-HermitianCholesky-errflag > > > https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7 > > > > Jose, > > PETSc does not support Cholesky for Hermitian matrix. > > > > The linear solver table probably needs to be updated. I have tried > several Cholesky solvers. With mkl_pardiso I get an explicit error message > that it does not support Cholesky with complex scalars. The rest (PETSc, > MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is > not related to your matrix, nor to shift-and-invert in SLEPc. I tried with > ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works in > complex scalars, but the matrix is real. As soon as you add complex entries > Cholesky fails, for instance adding this: > > ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > > > In this case, you must call > > MatSetOption(A,MAT_HERMITIAN,PETSC_TRUE); > > > > Then, petsc will throw an error for '-pc_type cholesky'. > > > > I don't know if it is a bug or that the algorithm cannot support complex > Hermitian matrices. Maybe Hong can confirm any of these. In the latter > case, I agree that all packages should give an error message, as > mkl_pardiso does. > > > > I also add an error flag for Cholesky/ICC if user does not set > > MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE) for aij matrix. > > See > https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 > > > > Let me know if you have any comments about this fix. > > > > Hong > > > > > El 25 sept 2017, a las 7:17, Greg Meyer > escribi?: > > > > > > Hi all, > > > > > > Hong--looking at your link, there may be no special algorithm for > Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like > it would any matrix. Furthermore it appears that Cholesky of complex > matrices is supported from this link: > https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html > > > > > > So do you or anyone have any idea why I get incorrect eigenvalues? > > > > > > Thanks, > > > Greg > > > > > > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer > wrote: > > > Ok, thanks. It seems that PETSc clearly should throw an error in this > case instead of just giving incorrect answers? I am surprised that it does > not throw an error... > > > > > > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: > > > Greg : > > > Yes, they are Hermitian. > > > > > > PETSc does not support Cholesky factorization for Hermitian. > > > It seems mumps does not support Hermitian either > > > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-November/027541.html > > > > > > Hong > > > > > > > > > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: > > > Greg: > > > > > > OK, the difference is whether LU or Cholesky factorization is used. > But I would hope that neither one should give incorrect eigenvalues, and > when I run with the latter it does! > > > Are your matrices symmetric/Hermitian? > > > Hong > > > > > > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: > > > Gregory : > > > Use '-eps_view' for both runs to check the algorithms being used. > > > Hong > > > > > > Hi all, > > > > > > I'm using shift-invert with EPS to solve for eigenvalues. I find that > if I do only > > > > > > ... > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > > ... > > > > > > in my code I get correct eigenvalues. But if I do > > > > > > ... > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); > > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > > > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); > > > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); > > > ... > > > > > > the eigenvalues found by EPS are completely wrong! Somehow I thought I > was supposed to do the latter, from the examples etc, but I guess that was > not correct? I attach the full piece of test code and a test matrix. > > > > > > Best, > > > Greg > > > > > > > > > > > > > > > > > -- > > Stefano > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ling.zou at inl.gov Fri Sep 29 10:06:30 2017 From: ling.zou at inl.gov (Zou, Ling) Date: Fri, 29 Sep 2017 09:06:30 -0600 Subject: [petsc-users] Good recommendation on meshing package? Message-ID: Hi all, I know this is a bit off topic on PETSc email list. I would like to try some finite volume type of CFD algorithm with PETSc, but I found it quite troublesome to manage mesh by myself. I wonder if there is any good existing meshing package that works well with PTESc. My expectation on such a package would be: 1) I create the mesh with some tool. 2) Read this mesh with the meshing package, so I have things like node set, edge set, cell set, etc. to play with 3) discretize my PDE with the mesh 4) solve it I also understand many people here use PETSc solve their CFD problem. I would appreciate it if you could also point me to some good examples. Best, Ling -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Fri Sep 29 10:19:01 2017 From: epscodes at gmail.com (Xiangdong) Date: Fri, 29 Sep 2017 11:19:01 -0400 Subject: [petsc-users] questions about pc_comosite Message-ID: Hello everyone, I have a questions about residuals reported in pc_composite. For the examples in petsc-3.7.6/src/ksp/ksp/examples/tutorials/ex1.c, I found that when I use these three options: 1) -pc_type none 2) -pc_type composite -pc_composite_pcs none,none -pc_composite_type multiplicative 3) -pc_type composite -pc_composite_pcs none,none -pc_composite_type multiplicative -ksp_norm_type unpreconditioned as shown below, it reported different KSP residuals. Given that I use the identity preconditioner (with multiplicative), why does the residual vary for different options? Thanks. Best, Xiangdong mpirun -np 1 ./extest -ksp_monitor -pc_type none 0 KSP Residual norm 1.414213562373e+00 1 KSP Residual norm 6.324555320337e-01 2 KSP Residual norm 3.779644730092e-01 3 KSP Residual norm 2.581988897472e-01 4 KSP Residual norm 1.906925178491e-01 5 KSP Residual norm 1.616509124176e-15 mpirun -np 1 ./extest -ksp_monitor -pc_type composite -pc_composite_pcs none,none -pc_composite_type multiplicative 0 KSP Residual norm 1.414213562373e+00 1 KSP Residual norm 1.176696810829e+00 2 KSP Residual norm 1.096908636191e+00 3 KSP Residual norm 4.389821446437e-01 4 KSP Residual norm 2.088364906576e-01 5 KSP Residual norm 2.725851091482e-13 mpirun -np 1 ./extest -ksp_monitor -pc_type composite -pc_composite_pcs none,none -pc_composite_type multiplicative -ksp_norm_type unpreconditioned 0 KSP Residual norm 1.414213562373e+00 1 KSP Residual norm 1.290994448736e+00 2 KSP Residual norm 1.249516035344e+00 3 KSP Residual norm 4.836894336642e-01 4 KSP Residual norm 1.467306491390e-01 5 KSP Residual norm 9.881776494390e-14 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Fri Sep 29 10:19:27 2017 From: hzhang at mcs.anl.gov (Hong) Date: Fri, 29 Sep 2017 10:19:27 -0500 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> Message-ID: Thanks for all the input. I can do following: 1) Move test to MatGetFactor() - If there is sufficient requests from user, we are able to add Hermitian support to petsc sequential Cholesky. 2) Tests: a) complex build && ftype==CHOLESKY: if (mat->hermitian) throw an "not supported" error b) all builds: if (!sbaij && (CHOLESKY || ICC)) if (!mat->symmetric) call MatIsSymmetric(mat,0.0,&symm) if (!symm) throw an error Hong On Fri, Sep 29, 2017 at 9:55 AM, Greg Meyer wrote: > FYI my perspective as a user--something that really tricked me was that > after setting the solver to Hermitian problem, the algorithm returned real > eigenvalues so they seemed OK. When I turned off Hermitian as I was trying > to debug, the eigenvalues were not at all just real, and thus it was clear > that they were wrong. So I think the check at least when Hermitian is set > is very important, since by construction real eigenvalues are returned. > > On Fri, Sep 29, 2017, 7:37 AM Barry Smith wrote: > >> >> 1) The test is definitely in the wrong place. If we are testing and >> erroring if using Cholesky and mat is not marked as symmetric or hermitian >> the test should be in MatGetFactor() not in a particular implementation. >> >> 2) It is a tough call if we should do this check or not. There are good >> arguments in both directions. >> >> One thing we could do is if the matrix is not marked as >> symmetric/hermitian is we could just check at that point (but expensive) or >> we could just check in debug mode. >> >> But I think we should require the user to set the flag (note for SBAIJ >> the flag for symmetric (or hermitian? which one) should be automatically >> set at creation). >> >> Hong can you move the test up to the MatGetFactor() level? >> >> Thanks >> Barry >> >> >> > On Sep 28, 2017, at 11:35 PM, Stefano Zampini < >> stefano.zampini at gmail.com> wrote: >> > >> > Hong, >> > >> > I personally believe that commit https://bitbucket.org/petsc/ >> petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 should be >> reverted. >> > I agree on the fact that when the user sets an option (the hermitian >> one in this case) and that feature is not supported we should throw an >> error (https://bitbucket.org/petsc/petsc/commits/ >> 8f21f52c465b775a76cda90fe9c51d0c742078c7) , but I don't like the fact >> that the user is forced to set on option to use a feature that he already >> requested (as in https://bitbucket.org/petsc/petsc/commits/ >> 966c94c9cf4fa13d455726ec36800a577e00b171). >> > >> > Barry, what do you think? >> > >> > 2017-09-29 5:28 GMT+03:00 Hong : >> > Greg: >> > Thanks so much for the detailed response. I am glad to know the reason >> behind it--hopefully we eventually figure out why the solvers have this >> behavior! Hong, I really appreciate you working on a patch to throw an >> error in this case. It definitely bit me and some people using my code... >> Hopefully it doesn't happen to anyone else! :) >> > >> > I added an error flag for using MUMPS Cholesky factorization on >> Hermitian matrix. See branch hzhang/mumps-HermitianCholesky-errflag >> > https://bitbucket.org/petsc/petsc/commits/ >> 8f21f52c465b775a76cda90fe9c51d0c742078c7 >> > >> > Jose, >> > PETSc does not support Cholesky for Hermitian matrix. >> > >> > The linear solver table probably needs to be updated. I have tried >> several Cholesky solvers. With mkl_pardiso I get an explicit error message >> that it does not support Cholesky with complex scalars. The rest (PETSc, >> MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is >> not related to your matrix, nor to shift-and-invert in SLEPc. I tried with >> ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works >> in complex scalars, but the matrix is real. As soon as you add complex >> entries Cholesky fails, for instance adding this: >> > ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); >> > ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); >> > >> > In this case, you must call >> > MatSetOption(A,MAT_HERMITIAN,PETSC_TRUE); >> > >> > Then, petsc will throw an error for '-pc_type cholesky'. >> > >> > I don't know if it is a bug or that the algorithm cannot support >> complex Hermitian matrices. Maybe Hong can confirm any of these. In the >> latter case, I agree that all packages should give an error message, as >> mkl_pardiso does. >> > >> > I also add an error flag for Cholesky/ICC if user does not set >> > MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE) for aij matrix. >> > See https://bitbucket.org/petsc/petsc/commits/ >> 966c94c9cf4fa13d455726ec36800a577e00b171 >> > >> > Let me know if you have any comments about this fix. >> > >> > Hong >> > >> > > El 25 sept 2017, a las 7:17, Greg Meyer >> escribi?: >> > > >> > > Hi all, >> > > >> > > Hong--looking at your link, there may be no special algorithm for >> Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like >> it would any matrix. Furthermore it appears that Cholesky of complex >> matrices is supported from this link: https://www.mcs.anl.gov/petsc/ >> documentation/linearsolvertable.html >> > > >> > > So do you or anyone have any idea why I get incorrect eigenvalues? >> > > >> > > Thanks, >> > > Greg >> > > >> > > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer >> wrote: >> > > Ok, thanks. It seems that PETSc clearly should throw an error in this >> case instead of just giving incorrect answers? I am surprised that it does >> not throw an error... >> > > >> > > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: >> > > Greg : >> > > Yes, they are Hermitian. >> > > >> > > PETSc does not support Cholesky factorization for Hermitian. >> > > It seems mumps does not support Hermitian either >> > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/ >> 2015-November/027541.html >> > > >> > > Hong >> > > >> > > >> > > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: >> > > Greg: >> > > >> > > OK, the difference is whether LU or Cholesky factorization is used. >> But I would hope that neither one should give incorrect eigenvalues, and >> when I run with the latter it does! >> > > Are your matrices symmetric/Hermitian? >> > > Hong >> > > >> > > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: >> > > Gregory : >> > > Use '-eps_view' for both runs to check the algorithms being used. >> > > Hong >> > > >> > > Hi all, >> > > >> > > I'm using shift-invert with EPS to solve for eigenvalues. I find that >> if I do only >> > > >> > > ... >> > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >> > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >> > > ... >> > > >> > > in my code I get correct eigenvalues. But if I do >> > > >> > > ... >> > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >> > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >> > > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); >> > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >> > > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); >> > > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); >> > > ... >> > > >> > > the eigenvalues found by EPS are completely wrong! Somehow I thought >> I was supposed to do the latter, from the examples etc, but I guess that >> was not correct? I attach the full piece of test code and a test matrix. >> > > >> > > Best, >> > > Greg >> > > >> > >> > >> > >> > >> > >> > >> > -- >> > Stefano >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitse at lmt.ens-cachan.fr Fri Sep 29 10:23:17 2017 From: vitse at lmt.ens-cachan.fr (Matthieu Vitse) Date: Fri, 29 Sep 2017 17:23:17 +0200 Subject: [petsc-users] Load distributed matrices from directory Message-ID: <56F37582-36CD-4862-8E63-E14A90704517@lmt.ens-cachan.fr> Hi all, I wanted to have some advices from advanced users about what I want to do. I would like to load in parallel a set of distributed matrices into petsc to build an ASM preconditioner, of course without having to handle the full matrix. Those matrices are generated by my FE software, exported using petsc4py. Is there a simple way to load all the matrices at the same time ? (by targeting a given folder for example, using MatLoad ? I haven?t been able to succeed in doing that yet). I guess it then be directly fed to PCASM? (maybe by provided user-defined subdomains). Thanks, ? Matt From bsmith at mcs.anl.gov Fri Sep 29 10:24:03 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 29 Sep 2017 08:24:03 -0700 Subject: [petsc-users] questions about pc_comosite In-Reply-To: References: Message-ID: <0414BE26-F9DF-4543-AFB7-010628BAD57C@mcs.anl.gov> They are all different iteration patterns. They definitely should present different residual results > On Sep 29, 2017, at 8:19 AM, Xiangdong wrote: > > Hello everyone, > > I have a questions about residuals reported in pc_composite. For the examples in petsc-3.7.6/src/ksp/ksp/examples/tutorials/ex1.c, I found that when I use these three options: > > 1) -pc_type none > 2) -pc_type composite -pc_composite_pcs none,none -pc_composite_type multiplicative > 3) -pc_type composite -pc_composite_pcs none,none -pc_composite_type multiplicative -ksp_norm_type unpreconditioned > > as shown below, it reported different KSP residuals. Given that I use the identity preconditioner (with multiplicative), why does the residual vary for different options? > > Thanks. > > Best, > Xiangdong > > mpirun -np 1 ./extest -ksp_monitor -pc_type none > > 0 KSP Residual norm 1.414213562373e+00 > 1 KSP Residual norm 6.324555320337e-01 > 2 KSP Residual norm 3.779644730092e-01 > 3 KSP Residual norm 2.581988897472e-01 > 4 KSP Residual norm 1.906925178491e-01 > 5 KSP Residual norm 1.616509124176e-15 > > mpirun -np 1 ./extest -ksp_monitor -pc_type composite -pc_composite_pcs none,none -pc_composite_type multiplicative > > 0 KSP Residual norm 1.414213562373e+00 > 1 KSP Residual norm 1.176696810829e+00 > 2 KSP Residual norm 1.096908636191e+00 > 3 KSP Residual norm 4.389821446437e-01 > 4 KSP Residual norm 2.088364906576e-01 > 5 KSP Residual norm 2.725851091482e-13 > > mpirun -np 1 ./extest -ksp_monitor -pc_type composite -pc_composite_pcs none,none -pc_composite_type multiplicative -ksp_norm_type unpreconditioned > 0 KSP Residual norm 1.414213562373e+00 > 1 KSP Residual norm 1.290994448736e+00 > 2 KSP Residual norm 1.249516035344e+00 > 3 KSP Residual norm 4.836894336642e-01 > 4 KSP Residual norm 1.467306491390e-01 > 5 KSP Residual norm 9.881776494390e-14 From stefano.zampini at gmail.com Fri Sep 29 10:25:23 2017 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Fri, 29 Sep 2017 18:25:23 +0300 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> Message-ID: <5FB7EA98-0C19-446B-B327-248DD15B47B5@gmail.com> > On Sep 29, 2017, at 6:19 PM, Hong > wrote: > > Thanks for all the input. I can do following: > 1) Move test to MatGetFactor() > - If there is sufficient requests from user, we are able to add Hermitian support to petsc sequential Cholesky. > > 2) Tests: > a) complex build && ftype==CHOLESKY: > if (mat->hermitian) throw an "not supported" error > > b) all builds: > if (!sbaij && (CHOLESKY || ICC)) > if (!mat->symmetric) I guess checking for mat->symmetric is redundant as you are calling MatIsSymmetric that already checks for it. Also, I?d rather prefer to guard this code branch with PETSC_USE_DEBUG > call MatIsSymmetric(mat,0.0,&symm) > if (!symm) throw an error > > Hong > > > On Fri, Sep 29, 2017 at 9:55 AM, Greg Meyer > wrote: > FYI my perspective as a user--something that really tricked me was that after setting the solver to Hermitian problem, the algorithm returned real eigenvalues so they seemed OK. When I turned off Hermitian as I was trying to debug, the eigenvalues were not at all just real, and thus it was clear that they were wrong. So I think the check at least when Hermitian is set is very important, since by construction real eigenvalues are returned. > > > On Fri, Sep 29, 2017, 7:37 AM Barry Smith > wrote: > > 1) The test is definitely in the wrong place. If we are testing and erroring if using Cholesky and mat is not marked as symmetric or hermitian the test should be in MatGetFactor() not in a particular implementation. > > 2) It is a tough call if we should do this check or not. There are good arguments in both directions. > > One thing we could do is if the matrix is not marked as symmetric/hermitian is we could just check at that point (but expensive) or we could just check in debug mode. > > But I think we should require the user to set the flag (note for SBAIJ the flag for symmetric (or hermitian? which one) should be automatically set at creation). > > Hong can you move the test up to the MatGetFactor() level? > > Thanks > Barry > > > > On Sep 28, 2017, at 11:35 PM, Stefano Zampini > wrote: > > > > Hong, > > > > I personally believe that commit https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 should be reverted. > > I agree on the fact that when the user sets an option (the hermitian one in this case) and that feature is not supported we should throw an error (https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7 ) , but I don't like the fact that the user is forced to set on option to use a feature that he already requested (as in https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 ). > > > > Barry, what do you think? > > > > 2017-09-29 5:28 GMT+03:00 Hong >: > > Greg: > > Thanks so much for the detailed response. I am glad to know the reason behind it--hopefully we eventually figure out why the solvers have this behavior! Hong, I really appreciate you working on a patch to throw an error in this case. It definitely bit me and some people using my code... Hopefully it doesn't happen to anyone else! :) > > > > I added an error flag for using MUMPS Cholesky factorization on Hermitian matrix. See branch hzhang/mumps-HermitianCholesky-errflag > > https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7 > > > > Jose, > > PETSc does not support Cholesky for Hermitian matrix. > > > > The linear solver table probably needs to be updated. I have tried several Cholesky solvers. With mkl_pardiso I get an explicit error message that it does not support Cholesky with complex scalars. The rest (PETSc, MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is not related to your matrix, nor to shift-and-invert in SLEPc. I tried with ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works in complex scalars, but the matrix is real. As soon as you add complex entries Cholesky fails, for instance adding this: > > ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > > > In this case, you must call > > MatSetOption(A,MAT_HERMITIAN,PETSC_TRUE); > > > > Then, petsc will throw an error for '-pc_type cholesky'. > > > > I don't know if it is a bug or that the algorithm cannot support complex Hermitian matrices. Maybe Hong can confirm any of these. In the latter case, I agree that all packages should give an error message, as mkl_pardiso does. > > > > I also add an error flag for Cholesky/ICC if user does not set > > MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE) for aij matrix. > > See https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 > > > > Let me know if you have any comments about this fix. > > > > Hong > > > > > El 25 sept 2017, a las 7:17, Greg Meyer > escribi?: > > > > > > Hi all, > > > > > > Hong--looking at your link, there may be no special algorithm for Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like it would any matrix. Furthermore it appears that Cholesky of complex matrices is supported from this link: https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html > > > > > > So do you or anyone have any idea why I get incorrect eigenvalues? > > > > > > Thanks, > > > Greg > > > > > > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer > wrote: > > > Ok, thanks. It seems that PETSc clearly should throw an error in this case instead of just giving incorrect answers? I am surprised that it does not throw an error... > > > > > > On Thu, Sep 21, 2017 at 5:24 PM Hong > wrote: > > > Greg : > > > Yes, they are Hermitian. > > > > > > PETSc does not support Cholesky factorization for Hermitian. > > > It seems mumps does not support Hermitian either > > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-November/027541.html > > > > > > Hong > > > > > > > > > On Thu, Sep 21, 2017 at 3:43 PM Hong > wrote: > > > Greg: > > > > > > OK, the difference is whether LU or Cholesky factorization is used. But I would hope that neither one should give incorrect eigenvalues, and when I run with the latter it does! > > > Are your matrices symmetric/Hermitian? > > > Hong > > > > > > On Thu, Sep 21, 2017 at 2:05 PM Hong > wrote: > > > Gregory : > > > Use '-eps_view' for both runs to check the algorithms being used. > > > Hong > > > > > > Hi all, > > > > > > I'm using shift-invert with EPS to solve for eigenvalues. I find that if I do only > > > > > > ... > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > > ... > > > > > > in my code I get correct eigenvalues. But if I do > > > > > > ... > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); > > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > > > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); > > > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); > > > ... > > > > > > the eigenvalues found by EPS are completely wrong! Somehow I thought I was supposed to do the latter, from the examples etc, but I guess that was not correct? I attach the full piece of test code and a test matrix. > > > > > > Best, > > > Greg > > > > > > > > > > > > > > > > > -- > > Stefano > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Sep 29 10:25:35 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 29 Sep 2017 08:25:35 -0700 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> Message-ID: No I don't want you to actually check if the matrix is symmetric (too expensive) just throw an error if the user has not indicated the appropriate properties of the matrix > On Sep 29, 2017, at 8:19 AM, Hong wrote: > > Thanks for all the input. I can do following: > 1) Move test to MatGetFactor() > - If there is sufficient requests from user, we are able to add Hermitian support to petsc sequential Cholesky. > > 2) Tests: > a) complex build && ftype==CHOLESKY: > if (mat->hermitian) throw an "not supported" error > > b) all builds: > if (!sbaij && (CHOLESKY || ICC)) > if (!mat->symmetric) > call MatIsSymmetric(mat,0.0,&symm) > if (!symm) throw an error > > Hong > > > On Fri, Sep 29, 2017 at 9:55 AM, Greg Meyer wrote: > FYI my perspective as a user--something that really tricked me was that after setting the solver to Hermitian problem, the algorithm returned real eigenvalues so they seemed OK. When I turned off Hermitian as I was trying to debug, the eigenvalues were not at all just real, and thus it was clear that they were wrong. So I think the check at least when Hermitian is set is very important, since by construction real eigenvalues are returned. > > > On Fri, Sep 29, 2017, 7:37 AM Barry Smith wrote: > > 1) The test is definitely in the wrong place. If we are testing and erroring if using Cholesky and mat is not marked as symmetric or hermitian the test should be in MatGetFactor() not in a particular implementation. > > 2) It is a tough call if we should do this check or not. There are good arguments in both directions. > > One thing we could do is if the matrix is not marked as symmetric/hermitian is we could just check at that point (but expensive) or we could just check in debug mode. > > But I think we should require the user to set the flag (note for SBAIJ the flag for symmetric (or hermitian? which one) should be automatically set at creation). > > Hong can you move the test up to the MatGetFactor() level? > > Thanks > Barry > > > > On Sep 28, 2017, at 11:35 PM, Stefano Zampini wrote: > > > > Hong, > > > > I personally believe that commit https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 should be reverted. > > I agree on the fact that when the user sets an option (the hermitian one in this case) and that feature is not supported we should throw an error (https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7) , but I don't like the fact that the user is forced to set on option to use a feature that he already requested (as in https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171). > > > > Barry, what do you think? > > > > 2017-09-29 5:28 GMT+03:00 Hong : > > Greg: > > Thanks so much for the detailed response. I am glad to know the reason behind it--hopefully we eventually figure out why the solvers have this behavior! Hong, I really appreciate you working on a patch to throw an error in this case. It definitely bit me and some people using my code... Hopefully it doesn't happen to anyone else! :) > > > > I added an error flag for using MUMPS Cholesky factorization on Hermitian matrix. See branch hzhang/mumps-HermitianCholesky-errflag > > https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7 > > > > Jose, > > PETSc does not support Cholesky for Hermitian matrix. > > > > The linear solver table probably needs to be updated. I have tried several Cholesky solvers. With mkl_pardiso I get an explicit error message that it does not support Cholesky with complex scalars. The rest (PETSc, MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is not related to your matrix, nor to shift-and-invert in SLEPc. I tried with ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works in complex scalars, but the matrix is real. As soon as you add complex entries Cholesky fails, for instance adding this: > > ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > > > In this case, you must call > > MatSetOption(A,MAT_HERMITIAN,PETSC_TRUE); > > > > Then, petsc will throw an error for '-pc_type cholesky'. > > > > I don't know if it is a bug or that the algorithm cannot support complex Hermitian matrices. Maybe Hong can confirm any of these. In the latter case, I agree that all packages should give an error message, as mkl_pardiso does. > > > > I also add an error flag for Cholesky/ICC if user does not set > > MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE) for aij matrix. > > See https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 > > > > Let me know if you have any comments about this fix. > > > > Hong > > > > > El 25 sept 2017, a las 7:17, Greg Meyer escribi?: > > > > > > Hi all, > > > > > > Hong--looking at your link, there may be no special algorithm for Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like it would any matrix. Furthermore it appears that Cholesky of complex matrices is supported from this link: https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html > > > > > > So do you or anyone have any idea why I get incorrect eigenvalues? > > > > > > Thanks, > > > Greg > > > > > > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer wrote: > > > Ok, thanks. It seems that PETSc clearly should throw an error in this case instead of just giving incorrect answers? I am surprised that it does not throw an error... > > > > > > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: > > > Greg : > > > Yes, they are Hermitian. > > > > > > PETSc does not support Cholesky factorization for Hermitian. > > > It seems mumps does not support Hermitian either > > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-November/027541.html > > > > > > Hong > > > > > > > > > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: > > > Greg: > > > > > > OK, the difference is whether LU or Cholesky factorization is used. But I would hope that neither one should give incorrect eigenvalues, and when I run with the latter it does! > > > Are your matrices symmetric/Hermitian? > > > Hong > > > > > > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: > > > Gregory : > > > Use '-eps_view' for both runs to check the algorithms being used. > > > Hong > > > > > > Hi all, > > > > > > I'm using shift-invert with EPS to solve for eigenvalues. I find that if I do only > > > > > > ... > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > > ... > > > > > > in my code I get correct eigenvalues. But if I do > > > > > > ... > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); > > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > > > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); > > > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); > > > ... > > > > > > the eigenvalues found by EPS are completely wrong! Somehow I thought I was supposed to do the latter, from the examples etc, but I guess that was not correct? I attach the full piece of test code and a test matrix. > > > > > > Best, > > > Greg > > > > > > > > > > > > > > > > > -- > > Stefano > > From epscodes at gmail.com Fri Sep 29 10:26:44 2017 From: epscodes at gmail.com (Xiangdong) Date: Fri, 29 Sep 2017 11:26:44 -0400 Subject: [petsc-users] questions about pc_comosite In-Reply-To: <0414BE26-F9DF-4543-AFB7-010628BAD57C@mcs.anl.gov> References: <0414BE26-F9DF-4543-AFB7-010628BAD57C@mcs.anl.gov> Message-ID: Hi Barry, Can you point me the documentation for understanding these iteration patterns? Thanks a lot. Best, Xiangdong On Fri, Sep 29, 2017 at 11:24 AM, Barry Smith wrote: > > They are all different iteration patterns. They definitely should > present different residual results > > > > On Sep 29, 2017, at 8:19 AM, Xiangdong wrote: > > > > Hello everyone, > > > > I have a questions about residuals reported in pc_composite. For the > examples in petsc-3.7.6/src/ksp/ksp/examples/tutorials/ex1.c, I found > that when I use these three options: > > > > 1) -pc_type none > > 2) -pc_type composite -pc_composite_pcs none,none -pc_composite_type > multiplicative > > 3) -pc_type composite -pc_composite_pcs none,none -pc_composite_type > multiplicative -ksp_norm_type unpreconditioned > > > > as shown below, it reported different KSP residuals. Given that I use > the identity preconditioner (with multiplicative), why does the residual > vary for different options? > > > > Thanks. > > > > Best, > > Xiangdong > > > > mpirun -np 1 ./extest -ksp_monitor -pc_type none > > > > 0 KSP Residual norm 1.414213562373e+00 > > 1 KSP Residual norm 6.324555320337e-01 > > 2 KSP Residual norm 3.779644730092e-01 > > 3 KSP Residual norm 2.581988897472e-01 > > 4 KSP Residual norm 1.906925178491e-01 > > 5 KSP Residual norm 1.616509124176e-15 > > > > mpirun -np 1 ./extest -ksp_monitor -pc_type composite -pc_composite_pcs > none,none -pc_composite_type multiplicative > > > > 0 KSP Residual norm 1.414213562373e+00 > > 1 KSP Residual norm 1.176696810829e+00 > > 2 KSP Residual norm 1.096908636191e+00 > > 3 KSP Residual norm 4.389821446437e-01 > > 4 KSP Residual norm 2.088364906576e-01 > > 5 KSP Residual norm 2.725851091482e-13 > > > > mpirun -np 1 ./extest -ksp_monitor -pc_type composite -pc_composite_pcs > none,none -pc_composite_type multiplicative -ksp_norm_type unpreconditioned > > 0 KSP Residual norm 1.414213562373e+00 > > 1 KSP Residual norm 1.290994448736e+00 > > 2 KSP Residual norm 1.249516035344e+00 > > 3 KSP Residual norm 4.836894336642e-01 > > 4 KSP Residual norm 1.467306491390e-01 > > 5 KSP Residual norm 9.881776494390e-14 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Fri Sep 29 10:33:07 2017 From: hzhang at mcs.anl.gov (Hong) Date: Fri, 29 Sep 2017 10:33:07 -0500 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> Message-ID: Taking both suggestions, how about 2) Tests: a) complex build && ftype==CHOLESKY: if (mat->hermitian) throw an "not supported" error b) all builds: if (!mat->sbaij && (CHOLESKY || ICC)) if (!mat->symmetric) //user does not indicate mat is symmetric #PETSC_USE_DEBUG call MatIsSymmetric(mat,0.0,&symm) if (!symm) throw an error #else throw an error #endif Hong On Fri, Sep 29, 2017 at 10:25 AM, Barry Smith wrote: > > No I don't want you to actually check if the matrix is symmetric (too > expensive) just throw an error if the user has not indicated the > appropriate properties of the matrix > > > > On Sep 29, 2017, at 8:19 AM, Hong wrote: > > > > Thanks for all the input. I can do following: > > 1) Move test to MatGetFactor() > > - If there is sufficient requests from user, we are able to add > Hermitian support to petsc sequential Cholesky. > > > > 2) Tests: > > a) complex build && ftype==CHOLESKY: > > if (mat->hermitian) throw an "not supported" error > > > > b) all builds: > > if (!sbaij && (CHOLESKY || ICC)) > > if (!mat->symmetric) > > call MatIsSymmetric(mat,0.0,&symm) > > if (!symm) throw an error > > > > Hong > > > > > > On Fri, Sep 29, 2017 at 9:55 AM, Greg Meyer > wrote: > > FYI my perspective as a user--something that really tricked me was that > after setting the solver to Hermitian problem, the algorithm returned real > eigenvalues so they seemed OK. When I turned off Hermitian as I was trying > to debug, the eigenvalues were not at all just real, and thus it was clear > that they were wrong. So I think the check at least when Hermitian is set > is very important, since by construction real eigenvalues are returned. > > > > > > On Fri, Sep 29, 2017, 7:37 AM Barry Smith wrote: > > > > 1) The test is definitely in the wrong place. If we are testing and > erroring if using Cholesky and mat is not marked as symmetric or hermitian > the test should be in MatGetFactor() not in a particular implementation. > > > > 2) It is a tough call if we should do this check or not. There are > good arguments in both directions. > > > > One thing we could do is if the matrix is not marked as > symmetric/hermitian is we could just check at that point (but expensive) or > we could just check in debug mode. > > > > But I think we should require the user to set the flag (note for SBAIJ > the flag for symmetric (or hermitian? which one) should be automatically > set at creation). > > > > Hong can you move the test up to the MatGetFactor() level? > > > > Thanks > > Barry > > > > > > > On Sep 28, 2017, at 11:35 PM, Stefano Zampini < > stefano.zampini at gmail.com> wrote: > > > > > > Hong, > > > > > > I personally believe that commit https://bitbucket.org/petsc/ > petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 should be reverted. > > > I agree on the fact that when the user sets an option (the hermitian > one in this case) and that feature is not supported we should throw an > error (https://bitbucket.org/petsc/petsc/commits/ > 8f21f52c465b775a76cda90fe9c51d0c742078c7) , but I don't like the fact > that the user is forced to set on option to use a feature that he already > requested (as in https://bitbucket.org/petsc/petsc/commits/ > 966c94c9cf4fa13d455726ec36800a577e00b171). > > > > > > Barry, what do you think? > > > > > > 2017-09-29 5:28 GMT+03:00 Hong : > > > Greg: > > > Thanks so much for the detailed response. I am glad to know the reason > behind it--hopefully we eventually figure out why the solvers have this > behavior! Hong, I really appreciate you working on a patch to throw an > error in this case. It definitely bit me and some people using my code... > Hopefully it doesn't happen to anyone else! :) > > > > > > I added an error flag for using MUMPS Cholesky factorization on > Hermitian matrix. See branch hzhang/mumps-HermitianCholesky-errflag > > > https://bitbucket.org/petsc/petsc/commits/ > 8f21f52c465b775a76cda90fe9c51d0c742078c7 > > > > > > Jose, > > > PETSc does not support Cholesky for Hermitian matrix. > > > > > > The linear solver table probably needs to be updated. I have tried > several Cholesky solvers. With mkl_pardiso I get an explicit error message > that it does not support Cholesky with complex scalars. The rest (PETSc, > MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is > not related to your matrix, nor to shift-and-invert in SLEPc. I tried with > ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works > in complex scalars, but the matrix is real. As soon as you add complex > entries Cholesky fails, for instance adding this: > > > ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > > ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > > > > > In this case, you must call > > > MatSetOption(A,MAT_HERMITIAN,PETSC_TRUE); > > > > > > Then, petsc will throw an error for '-pc_type cholesky'. > > > > > > I don't know if it is a bug or that the algorithm cannot support > complex Hermitian matrices. Maybe Hong can confirm any of these. In the > latter case, I agree that all packages should give an error message, as > mkl_pardiso does. > > > > > > I also add an error flag for Cholesky/ICC if user does not set > > > MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE) for aij matrix. > > > See https://bitbucket.org/petsc/petsc/commits/ > 966c94c9cf4fa13d455726ec36800a577e00b171 > > > > > > Let me know if you have any comments about this fix. > > > > > > Hong > > > > > > > El 25 sept 2017, a las 7:17, Greg Meyer > escribi?: > > > > > > > > Hi all, > > > > > > > > Hong--looking at your link, there may be no special algorithm for > Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like > it would any matrix. Furthermore it appears that Cholesky of complex > matrices is supported from this link: https://www.mcs.anl.gov/petsc/ > documentation/linearsolvertable.html > > > > > > > > So do you or anyone have any idea why I get incorrect eigenvalues? > > > > > > > > Thanks, > > > > Greg > > > > > > > > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer > wrote: > > > > Ok, thanks. It seems that PETSc clearly should throw an error in > this case instead of just giving incorrect answers? I am surprised that it > does not throw an error... > > > > > > > > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: > > > > Greg : > > > > Yes, they are Hermitian. > > > > > > > > PETSc does not support Cholesky factorization for Hermitian. > > > > It seems mumps does not support Hermitian either > > > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/ > 2015-November/027541.html > > > > > > > > Hong > > > > > > > > > > > > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: > > > > Greg: > > > > > > > > OK, the difference is whether LU or Cholesky factorization is used. > But I would hope that neither one should give incorrect eigenvalues, and > when I run with the latter it does! > > > > Are your matrices symmetric/Hermitian? > > > > Hong > > > > > > > > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: > > > > Gregory : > > > > Use '-eps_view' for both runs to check the algorithms being used. > > > > Hong > > > > > > > > Hi all, > > > > > > > > I'm using shift-invert with EPS to solve for eigenvalues. I find > that if I do only > > > > > > > > ... > > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > > > ... > > > > > > > > in my code I get correct eigenvalues. But if I do > > > > > > > > ... > > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > > > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); > > > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > > > > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); > > > > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); > > > > ... > > > > > > > > the eigenvalues found by EPS are completely wrong! Somehow I thought > I was supposed to do the latter, from the examples etc, but I guess that > was not correct? I attach the full piece of test code and a test matrix. > > > > > > > > Best, > > > > Greg > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Stefano > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Sep 29 10:43:06 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 29 Sep 2017 08:43:06 -0700 Subject: [petsc-users] Load distributed matrices from directory In-Reply-To: <56F37582-36CD-4862-8E63-E14A90704517@lmt.ens-cachan.fr> References: <56F37582-36CD-4862-8E63-E14A90704517@lmt.ens-cachan.fr> Message-ID: > On Sep 29, 2017, at 8:23 AM, Matthieu Vitse wrote: > > Hi all, > > I wanted to have some advices from advanced users about what I want to do. I would like to load in parallel a set of distributed matrices into petsc to build an ASM preconditioner, of course without having to handle the full matrix. Why, this is likely a painful, yet fruitless optimization. Just generate the entire matrix in parallel and use -pc_type asm Or if it is some other program generating the matrix save it to disk with MatView() and then load it with MatLoad(). Or is your matrix generator code sequential and cannot generate the full matrix so you want to generate chunks at a time and save to disk then load them? Better for you to refactor your code to work in parallel in generating the whole thing (since you can already generate parts the refactoring shouldn't be terribly difficult). Barry > Those matrices are generated by my FE software, exported using petsc4py. Is there a simple way to load all the matrices at the same time ? (by targeting a given folder for example, using MatLoad ? I haven?t been able to succeed in doing that yet). I guess it then be directly fed to PCASM? (maybe by provided user-defined subdomains). > > Thanks, > > ? > Matt From bsmith at mcs.anl.gov Fri Sep 29 10:53:59 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 29 Sep 2017 08:53:59 -0700 Subject: [petsc-users] questions about pc_comosite In-Reply-To: References: <0414BE26-F9DF-4543-AFB7-010628BAD57C@mcs.anl.gov> Message-ID: <0FD5B2C8-201F-4B78-BE51-3ABDA0239437@mcs.anl.gov> > On Sep 29, 2017, at 8:26 AM, Xiangdong wrote: > > Hi Barry, > > Can you point me the documentation for understanding these iteration patterns? Thanks a lot. They are explained implicitly in the documentation for the pieces but I admit it may not be clear. Case 1: Plain GMRES, prints the preconditioned residual at each iteration by default. The operator is just application by A Case 2: The operator is multiply by A, the preconditioner is apply the identity, compute the residual, update the solution by the residual (because the second inner preconditioner is identity). The preconditioner is not the same as in case 1 because of the computation of the residual inside the preconditioner and the update of the solution. Case 3: Same preconditioner as 2 but since the preconditioner is not the identity the unpreconditioned norm is different than the preconditioned norm hence it prints something different when you request the unpreconditioned norm. Barry > > Best, > Xiangdong > > On Fri, Sep 29, 2017 at 11:24 AM, Barry Smith wrote: > > They are all different iteration patterns. They definitely should present different residual results > > > > On Sep 29, 2017, at 8:19 AM, Xiangdong wrote: > > > > Hello everyone, > > > > I have a questions about residuals reported in pc_composite. For the examples in petsc-3.7.6/src/ksp/ksp/examples/tutorials/ex1.c, I found that when I use these three options: > > > > 1) -pc_type none > > 2) -pc_type composite -pc_composite_pcs none,none -pc_composite_type multiplicative > > 3) -pc_type composite -pc_composite_pcs none,none -pc_composite_type multiplicative -ksp_norm_type unpreconditioned > > > > as shown below, it reported different KSP residuals. Given that I use the identity preconditioner (with multiplicative), why does the residual vary for different options? > > > > Thanks. > > > > Best, > > Xiangdong > > > > mpirun -np 1 ./extest -ksp_monitor -pc_type none > > > > 0 KSP Residual norm 1.414213562373e+00 > > 1 KSP Residual norm 6.324555320337e-01 > > 2 KSP Residual norm 3.779644730092e-01 > > 3 KSP Residual norm 2.581988897472e-01 > > 4 KSP Residual norm 1.906925178491e-01 > > 5 KSP Residual norm 1.616509124176e-15 > > > > mpirun -np 1 ./extest -ksp_monitor -pc_type composite -pc_composite_pcs none,none -pc_composite_type multiplicative > > > > 0 KSP Residual norm 1.414213562373e+00 > > 1 KSP Residual norm 1.176696810829e+00 > > 2 KSP Residual norm 1.096908636191e+00 > > 3 KSP Residual norm 4.389821446437e-01 > > 4 KSP Residual norm 2.088364906576e-01 > > 5 KSP Residual norm 2.725851091482e-13 > > > > mpirun -np 1 ./extest -ksp_monitor -pc_type composite -pc_composite_pcs none,none -pc_composite_type multiplicative -ksp_norm_type unpreconditioned > > 0 KSP Residual norm 1.414213562373e+00 > > 1 KSP Residual norm 1.290994448736e+00 > > 2 KSP Residual norm 1.249516035344e+00 > > 3 KSP Residual norm 4.836894336642e-01 > > 4 KSP Residual norm 1.467306491390e-01 > > 5 KSP Residual norm 9.881776494390e-14 > > From bsmith at mcs.anl.gov Fri Sep 29 10:57:14 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 29 Sep 2017 08:57:14 -0700 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> Message-ID: No, never do the MatIsSymmetric() that was me just "thinking aloud" in my first mail Barry > On Sep 29, 2017, at 8:33 AM, Hong wrote: > > Taking both suggestions, how about > > 2) Tests: > a) complex build && ftype==CHOLESKY: > if (mat->hermitian) throw an "not supported" error > > b) all builds: > if (!mat->sbaij && (CHOLESKY || ICC)) > if (!mat->symmetric) //user does not indicate mat is symmetric > #PETSC_USE_DEBUG > call MatIsSymmetric(mat,0.0,&symm) > if (!symm) throw an error > #else > throw an error > #endif > > Hong > > On Fri, Sep 29, 2017 at 10:25 AM, Barry Smith wrote: > > No I don't want you to actually check if the matrix is symmetric (too expensive) just throw an error if the user has not indicated the appropriate properties of the matrix > > > > On Sep 29, 2017, at 8:19 AM, Hong wrote: > > > > Thanks for all the input. I can do following: > > 1) Move test to MatGetFactor() > > - If there is sufficient requests from user, we are able to add Hermitian support to petsc sequential Cholesky. > > > > 2) Tests: > > a) complex build && ftype==CHOLESKY: > > if (mat->hermitian) throw an "not supported" error > > > > b) all builds: > > if (!sbaij && (CHOLESKY || ICC)) > > if (!mat->symmetric) > > call MatIsSymmetric(mat,0.0,&symm) > > if (!symm) throw an error > > > > Hong > > > > > > On Fri, Sep 29, 2017 at 9:55 AM, Greg Meyer wrote: > > FYI my perspective as a user--something that really tricked me was that after setting the solver to Hermitian problem, the algorithm returned real eigenvalues so they seemed OK. When I turned off Hermitian as I was trying to debug, the eigenvalues were not at all just real, and thus it was clear that they were wrong. So I think the check at least when Hermitian is set is very important, since by construction real eigenvalues are returned. > > > > > > On Fri, Sep 29, 2017, 7:37 AM Barry Smith wrote: > > > > 1) The test is definitely in the wrong place. If we are testing and erroring if using Cholesky and mat is not marked as symmetric or hermitian the test should be in MatGetFactor() not in a particular implementation. > > > > 2) It is a tough call if we should do this check or not. There are good arguments in both directions. > > > > One thing we could do is if the matrix is not marked as symmetric/hermitian is we could just check at that point (but expensive) or we could just check in debug mode. > > > > But I think we should require the user to set the flag (note for SBAIJ the flag for symmetric (or hermitian? which one) should be automatically set at creation). > > > > Hong can you move the test up to the MatGetFactor() level? > > > > Thanks > > Barry > > > > > > > On Sep 28, 2017, at 11:35 PM, Stefano Zampini wrote: > > > > > > Hong, > > > > > > I personally believe that commit https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 should be reverted. > > > I agree on the fact that when the user sets an option (the hermitian one in this case) and that feature is not supported we should throw an error (https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7) , but I don't like the fact that the user is forced to set on option to use a feature that he already requested (as in https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171). > > > > > > Barry, what do you think? > > > > > > 2017-09-29 5:28 GMT+03:00 Hong : > > > Greg: > > > Thanks so much for the detailed response. I am glad to know the reason behind it--hopefully we eventually figure out why the solvers have this behavior! Hong, I really appreciate you working on a patch to throw an error in this case. It definitely bit me and some people using my code... Hopefully it doesn't happen to anyone else! :) > > > > > > I added an error flag for using MUMPS Cholesky factorization on Hermitian matrix. See branch hzhang/mumps-HermitianCholesky-errflag > > > https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7 > > > > > > Jose, > > > PETSc does not support Cholesky for Hermitian matrix. > > > > > > The linear solver table probably needs to be updated. I have tried several Cholesky solvers. With mkl_pardiso I get an explicit error message that it does not support Cholesky with complex scalars. The rest (PETSc, MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is not related to your matrix, nor to shift-and-invert in SLEPc. I tried with ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works in complex scalars, but the matrix is real. As soon as you add complex entries Cholesky fails, for instance adding this: > > > ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > > ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > > > > > In this case, you must call > > > MatSetOption(A,MAT_HERMITIAN,PETSC_TRUE); > > > > > > Then, petsc will throw an error for '-pc_type cholesky'. > > > > > > I don't know if it is a bug or that the algorithm cannot support complex Hermitian matrices. Maybe Hong can confirm any of these. In the latter case, I agree that all packages should give an error message, as mkl_pardiso does. > > > > > > I also add an error flag for Cholesky/ICC if user does not set > > > MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE) for aij matrix. > > > See https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 > > > > > > Let me know if you have any comments about this fix. > > > > > > Hong > > > > > > > El 25 sept 2017, a las 7:17, Greg Meyer escribi?: > > > > > > > > Hi all, > > > > > > > > Hong--looking at your link, there may be no special algorithm for Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like it would any matrix. Furthermore it appears that Cholesky of complex matrices is supported from this link: https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html > > > > > > > > So do you or anyone have any idea why I get incorrect eigenvalues? > > > > > > > > Thanks, > > > > Greg > > > > > > > > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer wrote: > > > > Ok, thanks. It seems that PETSc clearly should throw an error in this case instead of just giving incorrect answers? I am surprised that it does not throw an error... > > > > > > > > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: > > > > Greg : > > > > Yes, they are Hermitian. > > > > > > > > PETSc does not support Cholesky factorization for Hermitian. > > > > It seems mumps does not support Hermitian either > > > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015-November/027541.html > > > > > > > > Hong > > > > > > > > > > > > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: > > > > Greg: > > > > > > > > OK, the difference is whether LU or Cholesky factorization is used. But I would hope that neither one should give incorrect eigenvalues, and when I run with the latter it does! > > > > Are your matrices symmetric/Hermitian? > > > > Hong > > > > > > > > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: > > > > Gregory : > > > > Use '-eps_view' for both runs to check the algorithms being used. > > > > Hong > > > > > > > > Hi all, > > > > > > > > I'm using shift-invert with EPS to solve for eigenvalues. I find that if I do only > > > > > > > > ... > > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > > > ... > > > > > > > > in my code I get correct eigenvalues. But if I do > > > > > > > > ... > > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > > > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); > > > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > > > > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); > > > > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); > > > > ... > > > > > > > > the eigenvalues found by EPS are completely wrong! Somehow I thought I was supposed to do the latter, from the examples etc, but I guess that was not correct? I attach the full piece of test code and a test matrix. > > > > > > > > Best, > > > > Greg > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Stefano > > > > > > From hzhang at mcs.anl.gov Fri Sep 29 11:46:59 2017 From: hzhang at mcs.anl.gov (Hong) Date: Fri, 29 Sep 2017 11:46:59 -0500 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> Message-ID: Barry, When users apply Cholesky to a non-symmetric matrix, petsc uses his upper half matrix and would produce incorrect solutions without user's knowledge. Adding such check under "#PETSC_USE_DEBUG", user sees 1) his code crashes when matrix is non-symmetric or 2) too slow due to checking, which prompts user to set symmetric flag. I do not see any harm to call MatIsSymmetric() in this situation. Hong On Fri, Sep 29, 2017 at 10:57 AM, Barry Smith wrote: > > No, never do the MatIsSymmetric() that was me just "thinking aloud" in > my first mail > > Barry > > > On Sep 29, 2017, at 8:33 AM, Hong wrote: > > > > Taking both suggestions, how about > > > > 2) Tests: > > a) complex build && ftype==CHOLESKY: > > if (mat->hermitian) throw an "not supported" error > > > > b) all builds: > > if (!mat->sbaij && (CHOLESKY || ICC)) > > if (!mat->symmetric) //user does not indicate mat is symmetric > > #PETSC_USE_DEBUG > > call MatIsSymmetric(mat,0.0,&symm) > > if (!symm) throw an error > > #else > > throw an error > > #endif > > > > Hong > > > > On Fri, Sep 29, 2017 at 10:25 AM, Barry Smith > wrote: > > > > No I don't want you to actually check if the matrix is symmetric (too > expensive) just throw an error if the user has not indicated the > appropriate properties of the matrix > > > > > > > On Sep 29, 2017, at 8:19 AM, Hong wrote: > > > > > > Thanks for all the input. I can do following: > > > 1) Move test to MatGetFactor() > > > - If there is sufficient requests from user, we are able to add > Hermitian support to petsc sequential Cholesky. > > > > > > 2) Tests: > > > a) complex build && ftype==CHOLESKY: > > > if (mat->hermitian) throw an "not supported" error > > > > > > b) all builds: > > > if (!sbaij && (CHOLESKY || ICC)) > > > if (!mat->symmetric) > > > call MatIsSymmetric(mat,0.0,&symm) > > > if (!symm) throw an error > > > > > > Hong > > > > > > > > > On Fri, Sep 29, 2017 at 9:55 AM, Greg Meyer > wrote: > > > FYI my perspective as a user--something that really tricked me was > that after setting the solver to Hermitian problem, the algorithm returned > real eigenvalues so they seemed OK. When I turned off Hermitian as I was > trying to debug, the eigenvalues were not at all just real, and thus it was > clear that they were wrong. So I think the check at least when Hermitian is > set is very important, since by construction real eigenvalues are returned. > > > > > > > > > On Fri, Sep 29, 2017, 7:37 AM Barry Smith wrote: > > > > > > 1) The test is definitely in the wrong place. If we are testing and > erroring if using Cholesky and mat is not marked as symmetric or hermitian > the test should be in MatGetFactor() not in a particular implementation. > > > > > > 2) It is a tough call if we should do this check or not. There are > good arguments in both directions. > > > > > > One thing we could do is if the matrix is not marked as > symmetric/hermitian is we could just check at that point (but expensive) or > we could just check in debug mode. > > > > > > But I think we should require the user to set the flag (note for > SBAIJ the flag for symmetric (or hermitian? which one) should be > automatically set at creation). > > > > > > Hong can you move the test up to the MatGetFactor() level? > > > > > > Thanks > > > Barry > > > > > > > > > > On Sep 28, 2017, at 11:35 PM, Stefano Zampini < > stefano.zampini at gmail.com> wrote: > > > > > > > > Hong, > > > > > > > > I personally believe that commit https://bitbucket.org/petsc/ > petsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 should be reverted. > > > > I agree on the fact that when the user sets an option (the hermitian > one in this case) and that feature is not supported we should throw an > error (https://bitbucket.org/petsc/petsc/commits/ > 8f21f52c465b775a76cda90fe9c51d0c742078c7) , but I don't like the fact > that the user is forced to set on option to use a feature that he already > requested (as in https://bitbucket.org/petsc/petsc/commits/ > 966c94c9cf4fa13d455726ec36800a577e00b171). > > > > > > > > Barry, what do you think? > > > > > > > > 2017-09-29 5:28 GMT+03:00 Hong : > > > > Greg: > > > > Thanks so much for the detailed response. I am glad to know the > reason behind it--hopefully we eventually figure out why the solvers have > this behavior! Hong, I really appreciate you working on a patch to throw an > error in this case. It definitely bit me and some people using my code... > Hopefully it doesn't happen to anyone else! :) > > > > > > > > I added an error flag for using MUMPS Cholesky factorization on > Hermitian matrix. See branch hzhang/mumps-HermitianCholesky-errflag > > > > https://bitbucket.org/petsc/petsc/commits/ > 8f21f52c465b775a76cda90fe9c51d0c742078c7 > > > > > > > > Jose, > > > > PETSc does not support Cholesky for Hermitian matrix. > > > > > > > > The linear solver table probably needs to be updated. I have tried > several Cholesky solvers. With mkl_pardiso I get an explicit error message > that it does not support Cholesky with complex scalars. The rest (PETSc, > MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is > not related to your matrix, nor to shift-and-invert in SLEPc. I tried with > ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works > in complex scalars, but the matrix is real. As soon as you add complex > entries Cholesky fails, for instance adding this: > > > > ierr = MatSetValue(A,0,1,1.0+PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > > > ierr = MatSetValue(A,1,0,1.0-PETSC_i,INSERT_VALUES);CHKERRQ(ierr); > > > > > > > > In this case, you must call > > > > MatSetOption(A,MAT_HERMITIAN,PETSC_TRUE); > > > > > > > > Then, petsc will throw an error for '-pc_type cholesky'. > > > > > > > > I don't know if it is a bug or that the algorithm cannot support > complex Hermitian matrices. Maybe Hong can confirm any of these. In the > latter case, I agree that all packages should give an error message, as > mkl_pardiso does. > > > > > > > > I also add an error flag for Cholesky/ICC if user does not set > > > > MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE) for aij matrix. > > > > See https://bitbucket.org/petsc/petsc/commits/ > 966c94c9cf4fa13d455726ec36800a577e00b171 > > > > > > > > Let me know if you have any comments about this fix. > > > > > > > > Hong > > > > > > > > > El 25 sept 2017, a las 7:17, Greg Meyer > escribi?: > > > > > > > > > > Hi all, > > > > > > > > > > Hong--looking at your link, there may be no special algorithm for > Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like > it would any matrix. Furthermore it appears that Cholesky of complex > matrices is supported from this link: https://www.mcs.anl.gov/petsc/ > documentation/linearsolvertable.html > > > > > > > > > > So do you or anyone have any idea why I get incorrect eigenvalues? > > > > > > > > > > Thanks, > > > > > Greg > > > > > > > > > > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer < > gregory.meyer at gmail.com> wrote: > > > > > Ok, thanks. It seems that PETSc clearly should throw an error in > this case instead of just giving incorrect answers? I am surprised that it > does not throw an error... > > > > > > > > > > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: > > > > > Greg : > > > > > Yes, they are Hermitian. > > > > > > > > > > PETSc does not support Cholesky factorization for Hermitian. > > > > > It seems mumps does not support Hermitian either > > > > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/ > 2015-November/027541.html > > > > > > > > > > Hong > > > > > > > > > > > > > > > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: > > > > > Greg: > > > > > > > > > > OK, the difference is whether LU or Cholesky factorization is > used. But I would hope that neither one should give incorrect eigenvalues, > and when I run with the latter it does! > > > > > Are your matrices symmetric/Hermitian? > > > > > Hong > > > > > > > > > > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: > > > > > Gregory : > > > > > Use '-eps_view' for both runs to check the algorithms being used. > > > > > Hong > > > > > > > > > > Hi all, > > > > > > > > > > I'm using shift-invert with EPS to solve for eigenvalues. I find > that if I do only > > > > > > > > > > ... > > > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > > > > ... > > > > > > > > > > in my code I get correct eigenvalues. But if I do > > > > > > > > > > ... > > > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); > > > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); > > > > > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); > > > > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > > > > > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); > > > > > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); > > > > > ... > > > > > > > > > > the eigenvalues found by EPS are completely wrong! Somehow I > thought I was supposed to do the latter, from the examples etc, but I guess > that was not correct? I attach the full piece of test code and a test > matrix. > > > > > > > > > > Best, > > > > > Greg > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Stefano > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Fri Sep 29 12:22:22 2017 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Fri, 29 Sep 2017 18:22:22 +0100 Subject: [petsc-users] Understanding matmult memory performance In-Reply-To: <53A3BC17-1C6C-4D44-931B-EDDECC2EA54D@imperial.ac.uk> References: <8D683AE4-75B9-40AF-97E6-6801657DE095@imperial.ac.uk> <20170929130447.kmvqn3yintmnqrcw@gatech.edu> <20170929140501.ukincl2vdohl7mok@gatech.edu> <53A3BC17-1C6C-4D44-931B-EDDECC2EA54D@imperial.ac.uk> Message-ID: <0F4FA504-6685-4CB9-A6C0-20AF2E936EE7@imperial.ac.uk> > On 29 Sep 2017, at 15:24, Lawrence Mitchell wrote: > >> according to >> https://ark.intel.com/products/75283/Intel-Xeon-Processor-E5-2697-v2-30M-Cache-2_70-GHz >> you get 59.7 GB/sec of peak memory bandwidth per CPU, so you should get about 240 GB/sec for your two-node system. > > That's assuming I have DDR3-2133 RAM chips, but as noted above, it looks like the nodes probably have 1866 RAM. Giving 204.8 GB/s. Ugh, I can't multiply numbers together sensibly. Karl is right, and I am dumb. Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: Message signed with OpenPGP URL: From stefano.zampini at gmail.com Fri Sep 29 12:57:17 2017 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Fri, 29 Sep 2017 20:57:17 +0300 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> Message-ID: Should we add a basic fallback that tests for matmult and matmulttranspose to MatIsSymmetric if ops->issymmetric is not set? Il 29 Set 2017 7:47 PM, "Hong" ha scritto: > Barry, > > When users apply Cholesky to a non-symmetric matrix, > petsc uses his upper half matrix and would produce incorrect solutions > without user's knowledge. > > Adding such check under "#PETSC_USE_DEBUG", user sees > 1) his code crashes when matrix is non-symmetric > or > 2) too slow due to checking, which prompts user to set symmetric flag. > I do not see any harm to call MatIsSymmetric() in this situation. > > Hong > > On Fri, Sep 29, 2017 at 10:57 AM, Barry Smith wrote: > >> >> No, never do the MatIsSymmetric() that was me just "thinking aloud" in >> my first mail >> >> Barry >> >> > On Sep 29, 2017, at 8:33 AM, Hong wrote: >> > >> > Taking both suggestions, how about >> > >> > 2) Tests: >> > a) complex build && ftype==CHOLESKY: >> > if (mat->hermitian) throw an "not supported" error >> > >> > b) all builds: >> > if (!mat->sbaij && (CHOLESKY || ICC)) >> > if (!mat->symmetric) //user does not indicate mat is symmetric >> > #PETSC_USE_DEBUG >> > call MatIsSymmetric(mat,0.0,&symm) >> > if (!symm) throw an error >> > #else >> > throw an error >> > #endif >> > >> > Hong >> > >> > On Fri, Sep 29, 2017 at 10:25 AM, Barry Smith >> wrote: >> > >> > No I don't want you to actually check if the matrix is symmetric (too >> expensive) just throw an error if the user has not indicated the >> appropriate properties of the matrix >> > >> > >> > > On Sep 29, 2017, at 8:19 AM, Hong wrote: >> > > >> > > Thanks for all the input. I can do following: >> > > 1) Move test to MatGetFactor() >> > > - If there is sufficient requests from user, we are able to add >> Hermitian support to petsc sequential Cholesky. >> > > >> > > 2) Tests: >> > > a) complex build && ftype==CHOLESKY: >> > > if (mat->hermitian) throw an "not supported" error >> > > >> > > b) all builds: >> > > if (!sbaij && (CHOLESKY || ICC)) >> > > if (!mat->symmetric) >> > > call MatIsSymmetric(mat,0.0,&symm) >> > > if (!symm) throw an error >> > > >> > > Hong >> > > >> > > >> > > On Fri, Sep 29, 2017 at 9:55 AM, Greg Meyer >> wrote: >> > > FYI my perspective as a user--something that really tricked me was >> that after setting the solver to Hermitian problem, the algorithm returned >> real eigenvalues so they seemed OK. When I turned off Hermitian as I was >> trying to debug, the eigenvalues were not at all just real, and thus it was >> clear that they were wrong. So I think the check at least when Hermitian is >> set is very important, since by construction real eigenvalues are returned. >> > > >> > > >> > > On Fri, Sep 29, 2017, 7:37 AM Barry Smith wrote: >> > > >> > > 1) The test is definitely in the wrong place. If we are testing and >> erroring if using Cholesky and mat is not marked as symmetric or hermitian >> the test should be in MatGetFactor() not in a particular implementation. >> > > >> > > 2) It is a tough call if we should do this check or not. There are >> good arguments in both directions. >> > > >> > > One thing we could do is if the matrix is not marked as >> symmetric/hermitian is we could just check at that point (but expensive) or >> we could just check in debug mode. >> > > >> > > But I think we should require the user to set the flag (note for >> SBAIJ the flag for symmetric (or hermitian? which one) should be >> automatically set at creation). >> > > >> > > Hong can you move the test up to the MatGetFactor() level? >> > > >> > > Thanks >> > > Barry >> > > >> > > >> > > > On Sep 28, 2017, at 11:35 PM, Stefano Zampini < >> stefano.zampini at gmail.com> wrote: >> > > > >> > > > Hong, >> > > > >> > > > I personally believe that commit https://bitbucket.org/petsc/pe >> tsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171 should be reverted. >> > > > I agree on the fact that when the user sets an option (the >> hermitian one in this case) and that feature is not supported we should >> throw an error (https://bitbucket.org/petsc/p >> etsc/commits/8f21f52c465b775a76cda90fe9c51d0c742078c7) , but I don't >> like the fact that the user is forced to set on option to use a feature >> that he already requested (as in https://bitbucket.org/petsc/pe >> tsc/commits/966c94c9cf4fa13d455726ec36800a577e00b171). >> > > > >> > > > Barry, what do you think? >> > > > >> > > > 2017-09-29 5:28 GMT+03:00 Hong : >> > > > Greg: >> > > > Thanks so much for the detailed response. I am glad to know the >> reason behind it--hopefully we eventually figure out why the solvers have >> this behavior! Hong, I really appreciate you working on a patch to throw an >> error in this case. It definitely bit me and some people using my code... >> Hopefully it doesn't happen to anyone else! :) >> > > > >> > > > I added an error flag for using MUMPS Cholesky factorization on >> Hermitian matrix. See branch hzhang/mumps-HermitianCholesky-errflag >> > > > https://bitbucket.org/petsc/petsc/commits/8f21f52c465b775a76 >> cda90fe9c51d0c742078c7 >> > > > >> > > > Jose, >> > > > PETSc does not support Cholesky for Hermitian matrix. >> > > > >> > > > The linear solver table probably needs to be updated. I have tried >> several Cholesky solvers. With mkl_pardiso I get an explicit error message >> that it does not support Cholesky with complex scalars. The rest (PETSc, >> MUMPS, CHOLMOD) give a wrong answer (without error message). The problem is >> not related to your matrix, nor to shift-and-invert in SLEPc. I tried with >> ex1.c under PETSC_DIR/src/ksp/ksp/examples/tutorials. The example works >> in complex scalars, but the matrix is real. As soon as you add complex >> entries Cholesky fails, for instance adding this: >> > > > ierr = MatSetValue(A,0,1,1.0+PETSC_i, >> INSERT_VALUES);CHKERRQ(ierr); >> > > > ierr = MatSetValue(A,1,0,1.0-PETSC_i, >> INSERT_VALUES);CHKERRQ(ierr); >> > > > >> > > > In this case, you must call >> > > > MatSetOption(A,MAT_HERMITIAN,PETSC_TRUE); >> > > > >> > > > Then, petsc will throw an error for '-pc_type cholesky'. >> > > > >> > > > I don't know if it is a bug or that the algorithm cannot support >> complex Hermitian matrices. Maybe Hong can confirm any of these. In the >> latter case, I agree that all packages should give an error message, as >> mkl_pardiso does. >> > > > >> > > > I also add an error flag for Cholesky/ICC if user does not set >> > > > MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE) for aij matrix. >> > > > See https://bitbucket.org/petsc/petsc/commits/966c94c9cf4fa13d45 >> 5726ec36800a577e00b171 >> > > > >> > > > Let me know if you have any comments about this fix. >> > > > >> > > > Hong >> > > > >> > > > > El 25 sept 2017, a las 7:17, Greg Meyer >> escribi?: >> > > > > >> > > > > Hi all, >> > > > > >> > > > > Hong--looking at your link, there may be no special algorithm for >> Hermitian matrices in MUMPS, but that doesn't mean it can't solve them like >> it would any matrix. Furthermore it appears that Cholesky of complex >> matrices is supported from this link: https://www.mcs.anl.gov/petsc/ >> documentation/linearsolvertable.html >> > > > > >> > > > > So do you or anyone have any idea why I get incorrect eigenvalues? >> > > > > >> > > > > Thanks, >> > > > > Greg >> > > > > >> > > > > On Thu, Sep 21, 2017 at 5:51 PM Greg Meyer < >> gregory.meyer at gmail.com> wrote: >> > > > > Ok, thanks. It seems that PETSc clearly should throw an error in >> this case instead of just giving incorrect answers? I am surprised that it >> does not throw an error... >> > > > > >> > > > > On Thu, Sep 21, 2017 at 5:24 PM Hong wrote: >> > > > > Greg : >> > > > > Yes, they are Hermitian. >> > > > > >> > > > > PETSc does not support Cholesky factorization for Hermitian. >> > > > > It seems mumps does not support Hermitian either >> > > > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2015- >> November/027541.html >> > > > > >> > > > > Hong >> > > > > >> > > > > >> > > > > On Thu, Sep 21, 2017 at 3:43 PM Hong wrote: >> > > > > Greg: >> > > > > >> > > > > OK, the difference is whether LU or Cholesky factorization is >> used. But I would hope that neither one should give incorrect eigenvalues, >> and when I run with the latter it does! >> > > > > Are your matrices symmetric/Hermitian? >> > > > > Hong >> > > > > >> > > > > On Thu, Sep 21, 2017 at 2:05 PM Hong wrote: >> > > > > Gregory : >> > > > > Use '-eps_view' for both runs to check the algorithms being used. >> > > > > Hong >> > > > > >> > > > > Hi all, >> > > > > >> > > > > I'm using shift-invert with EPS to solve for eigenvalues. I find >> that if I do only >> > > > > >> > > > > ... >> > > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >> > > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >> > > > > ... >> > > > > >> > > > > in my code I get correct eigenvalues. But if I do >> > > > > >> > > > > ... >> > > > > ierr = EPSGetST(eps,&st);CHKERRQ(ierr); >> > > > > ierr = STSetType(st,STSINVERT);CHKERRQ(ierr); >> > > > > ierr = STGetKSP(st,&ksp);CHKERRQ(ierr); >> > > > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >> > > > > ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); >> > > > > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); >> > > > > ... >> > > > > >> > > > > the eigenvalues found by EPS are completely wrong! Somehow I >> thought I was supposed to do the latter, from the examples etc, but I guess >> that was not correct? I attach the full piece of test code and a test >> matrix. >> > > > > >> > > > > Best, >> > > > > Greg >> > > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > Stefano >> > > >> > > >> > >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Sep 29 13:48:10 2017 From: jed at jedbrown.org (Jed Brown) Date: Fri, 29 Sep 2017 12:48:10 -0600 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> Message-ID: <87a81dwdcl.fsf@jedbrown.org> Barry Smith writes: > 1) The test is definitely in the wrong place. If we are testing and erroring if using Cholesky and mat is not marked as symmetric or hermitian the test should be in MatGetFactor() not in a particular implementation. > > 2) It is a tough call if we should do this check or not. There are good arguments in both directions. > > One thing we could do is if the matrix is not marked as symmetric/hermitian is we could just check at that point (but expensive) or we could just check in debug mode. > > But I think we should require the user to set the flag (note for SBAIJ the flag for symmetric (or hermitian? which one) should be automatically set at creation). So what's the approach if the user wants to do Cholesky on the nearby symmetric matrix defined by the upper triangular part? Now they'll need to create that matrix explicitly? It's certainly relevant for a nearly-symmetric matrix, either due to a weak transport process or implementation of boundary conditions. From hzhang at mcs.anl.gov Fri Sep 29 16:13:07 2017 From: hzhang at mcs.anl.gov (Hong) Date: Fri, 29 Sep 2017 16:13:07 -0500 Subject: [petsc-users] Incorrect Eigenvalues when Setting KSP and PC types In-Reply-To: <87a81dwdcl.fsf@jedbrown.org> References: <9DB5C630-CC9E-4B4D-904F-E4F6190EFBED@dsic.upv.es> <62BCE2B5-2B6F-447F-9BEF-9E7AEB404184@mcs.anl.gov> <87a81dwdcl.fsf@jedbrown.org> Message-ID: OK, making things easier and flexible, I only did 1) Move test to MatGetFactor() 2) Test: complex build && ftype==CHOLESKY||ICC: if (mat->hermitian) throw an "not supported" error MatIsSymmetric() is either expensive or not supported, and users may apply Cholesky or ICC to almost symmetric matrix, so I will not change existing way for real build. Hong On Fri, Sep 29, 2017 at 1:48 PM, Jed Brown wrote: > Barry Smith writes: > > > 1) The test is definitely in the wrong place. If we are testing and > erroring if using Cholesky and mat is not marked as symmetric or hermitian > the test should be in MatGetFactor() not in a particular implementation. > > > > 2) It is a tough call if we should do this check or not. There are > good arguments in both directions. > > > > One thing we could do is if the matrix is not marked as > symmetric/hermitian is we could just check at that point (but expensive) or > we could just check in debug mode. > > > > But I think we should require the user to set the flag (note for SBAIJ > the flag for symmetric (or hermitian? which one) should be automatically > set at creation). > > So what's the approach if the user wants to do Cholesky on the nearby > symmetric matrix defined by the upper triangular part? Now they'll need > to create that matrix explicitly? It's certainly relevant for a > nearly-symmetric matrix, either due to a weak transport process or > implementation of boundary conditions. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbcbh1999 at gmail.com Fri Sep 29 19:15:29 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Fri, 29 Sep 2017 20:15:29 -0400 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: I'm using -fp_trap as I run the simulation. the output only gives floating point exception It doesn't matter coarse grid or medium grid or more refined grid. the program crashed right away. On Fri, Sep 29, 2017 at 12:19 AM, Barry Smith wrote: > > Please upgrade to the latest 3.8 version of PETSc. and then run with > -fp_trap > > This message is usually an indication that a NaN or Inf got into the > numerical computation. Using the latest PETSc will make it much easier to > track down. > > Barry > > > On Sep 28, 2017, at 8:57 PM, Hao Zhang wrote: > > > > I'm experiencing error when doing mesh refinement(mesh x2 for X, Y and Z > direction) for 3D poisson problem. > > PETSc produces no errors when no mesh refinement. I've pasted some > debugging info here. Please advise. > > > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Invalid argument > > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 > > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid on a > arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 23:47:39 2017 > > [0]PETSC ERROR: Configure options --with-mpi-dir=/cm/shared/apps/openmpi/gcc/64/1.8.5/ > --download-fblaslapack=/home/haozhang/tmp/fblaslapack-3.4.2.tar.gz > --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz > --with-valgrind-include=/home/haozhang/include/valgrind/ > > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in > /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/vec/vec/interface/rvector.c > > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in > /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/impls/bcgs/bcgs.c > > [0]PETSC ERROR: #3 KSPSolve() line 460 in /home/haozhang/tmp/petsc-3.5. > 4_HYPRE/src/ksp/ksp/interface/itfunc.c > > [vn18:99838] *** Process received signal *** > > [vn18:99838] Signal: Aborted (6) > > [vn18:99838] Signal code: (-6) > > [vn18:99848] *** Process received signal *** > > [vn18:99849] *** Process received signal *** > > [vn18:99849] Signal: Aborted (6) > > [vn18:99849] Signal code: (-6) > > [vn18:99848] Signal: Aborted (6) > > [vn18:99848] Signal code: (-6) > > [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790)[0x2aaaadbb4790] > > [vn18:99838] [ 1] [vn18:99848] [ 0] /lib64/libpthread.so.0(+ > 0xf790)[0x2aaaadbb4790] > > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > [vn18:99838] [ 3] [vn18:99849] [ 0] /lib64/libpthread.so.0(+ > 0xf790)[0x2aaaadbb4790] > > [vn18:99849] [ 1] [vn15:122106] *** Process received signal *** > > [vn15:122106] Signal: Aborted (6) > > [vn15:122106] Signal code: (-6) > > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5. > 4_HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5( > PetscTraceBackErrorHandler+0x503)[0x2aaaaae1d7e1] > > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > [vn18:99849] [ 3] [vn15:122107] *** Process received signal *** > > [vn15:122107] Signal: Aborted (6) > > [vn15:122107] Signal code: (-6) > > [vn16:86295] *** Process received signal *** > > [vn16:86295] Signal: Aborted (6) > > [vn16:86295] Signal code: (-6) > > [vn18:99824] *** Process received signal *** > > [vn18:99824] Signal: Aborted (6) > > [vn18:99824] Signal code: (-6) > > [vn18:99844] *** Process received signal *** > > [vn18:99844] Signal: Aborted (6) > > [vn18:99844] Signal code: (-6) > > /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/ > lib/libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] > > > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbcbh1999 at gmail.com Fri Sep 29 19:28:42 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Fri, 29 Sep 2017 20:28:42 -0400 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: I've modified my source code accordingly using PETSc 3.8.0. It seems no more useful info got printed out. On Fri, Sep 29, 2017 at 8:15 PM, Hao Zhang wrote: > I'm using -fp_trap as I run the simulation. the output only gives > > floating point exception > > It doesn't matter coarse grid or medium grid or more refined grid. the > program crashed right away. > > On Fri, Sep 29, 2017 at 12:19 AM, Barry Smith wrote: > >> >> Please upgrade to the latest 3.8 version of PETSc. and then run with >> -fp_trap >> >> This message is usually an indication that a NaN or Inf got into the >> numerical computation. Using the latest PETSc will make it much easier to >> track down. >> >> Barry >> >> > On Sep 28, 2017, at 8:57 PM, Hao Zhang wrote: >> > >> > I'm experiencing error when doing mesh refinement(mesh x2 for X, Y and >> Z direction) for 3D poisson problem. >> > PETSc produces no errors when no mesh refinement. I've pasted some >> debugging info here. Please advise. >> > >> > >> > [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > [0]PETSC ERROR: Invalid argument >> > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 >> > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid on a >> arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 23:47:39 2017 >> > [0]PETSC ERROR: Configure options --with-mpi-dir=/cm/shared/apps/openmpi/gcc/64/1.8.5/ >> --download-fblaslapack=/home/haozhang/tmp/fblaslapack-3.4.2.tar.gz >> --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz >> --with-valgrind-include=/home/haozhang/include/valgrind/ >> > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/vec/vec/interface/rvector.c >> > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/impls/bcgs/bcgs.c >> > [0]PETSC ERROR: #3 KSPSolve() line 460 in /home/haozhang/tmp/petsc-3.5.4 >> _HYPRE/src/ksp/ksp/interface/itfunc.c >> > [vn18:99838] *** Process received signal *** >> > [vn18:99838] Signal: Aborted (6) >> > [vn18:99838] Signal code: (-6) >> > [vn18:99848] *** Process received signal *** >> > [vn18:99849] *** Process received signal *** >> > [vn18:99849] Signal: Aborted (6) >> > [vn18:99849] Signal code: (-6) >> > [vn18:99848] Signal: Aborted (6) >> > [vn18:99848] Signal code: (-6) >> > [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790)[0x2aaaadbb4790] >> > [vn18:99838] [ 1] [vn18:99848] [ 0] /lib64/libpthread.so.0(+0xf790 >> )[0x2aaaadbb4790] >> > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] >> > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] >> > [vn18:99838] [ 3] [vn18:99849] [ 0] /lib64/libpthread.so.0(+0xf790 >> )[0x2aaaadbb4790] >> > [vn18:99849] [ 1] [vn15:122106] *** Process received signal *** >> > [vn15:122106] Signal: Aborted (6) >> > [vn15:122106] Signal code: (-6) >> > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] >> > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] >> > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4 >> _HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscTraceBac >> kErrorHandler+0x503)[0x2aaaaae1d7e1] >> > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] >> > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] >> > [vn18:99849] [ 3] [vn15:122107] *** Process received signal *** >> > [vn15:122107] Signal: Aborted (6) >> > [vn15:122107] Signal code: (-6) >> > [vn16:86295] *** Process received signal *** >> > [vn16:86295] Signal: Aborted (6) >> > [vn16:86295] Signal code: (-6) >> > [vn18:99824] *** Process received signal *** >> > [vn18:99824] Signal: Aborted (6) >> > [vn18:99824] Signal code: (-6) >> > [vn18:99844] *** Process received signal *** >> > [vn18:99844] Signal: Aborted (6) >> > [vn18:99844] Signal code: (-6) >> > /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/lib >> /libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] >> > >> > >> > >> > -- >> > Hao Zhang >> > Dept. of Applid Mathematics and Statistics, >> > Stony Brook University, >> > Stony Brook, New York, 11790 >> >> > > > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Sep 29 19:45:44 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 29 Sep 2017 19:45:44 -0500 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: Can you run this code with valgrind and send the log? http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Satish On Sat, 30 Sep 2017, Hao Zhang wrote: > I've modified my source code accordingly using PETSc 3.8.0. It seems no > more useful info got printed out. > > On Fri, Sep 29, 2017 at 8:15 PM, Hao Zhang wrote: > > > I'm using -fp_trap as I run the simulation. the output only gives > > > > floating point exception > > > > It doesn't matter coarse grid or medium grid or more refined grid. the > > program crashed right away. > > > > On Fri, Sep 29, 2017 at 12:19 AM, Barry Smith wrote: > > > >> > >> Please upgrade to the latest 3.8 version of PETSc. and then run with > >> -fp_trap > >> > >> This message is usually an indication that a NaN or Inf got into the > >> numerical computation. Using the latest PETSc will make it much easier to > >> track down. > >> > >> Barry > >> > >> > On Sep 28, 2017, at 8:57 PM, Hao Zhang wrote: > >> > > >> > I'm experiencing error when doing mesh refinement(mesh x2 for X, Y and > >> Z direction) for 3D poisson problem. > >> > PETSc produces no errors when no mesh refinement. I've pasted some > >> debugging info here. Please advise. > >> > > >> > > >> > [0]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> > [0]PETSC ERROR: Invalid argument > >> > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 > >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > >> for trouble shooting. > >> > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 > >> > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid on a > >> arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 23:47:39 2017 > >> > [0]PETSC ERROR: Configure options --with-mpi-dir=/cm/shared/apps/openmpi/gcc/64/1.8.5/ > >> --download-fblaslapack=/home/haozhang/tmp/fblaslapack-3.4.2.tar.gz > >> --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz > >> --with-valgrind-include=/home/haozhang/include/valgrind/ > >> > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/vec/vec/interface/rvector.c > >> > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/impls/bcgs/bcgs.c > >> > [0]PETSC ERROR: #3 KSPSolve() line 460 in /home/haozhang/tmp/petsc-3.5.4 > >> _HYPRE/src/ksp/ksp/interface/itfunc.c > >> > [vn18:99838] *** Process received signal *** > >> > [vn18:99838] Signal: Aborted (6) > >> > [vn18:99838] Signal code: (-6) > >> > [vn18:99848] *** Process received signal *** > >> > [vn18:99849] *** Process received signal *** > >> > [vn18:99849] Signal: Aborted (6) > >> > [vn18:99849] Signal code: (-6) > >> > [vn18:99848] Signal: Aborted (6) > >> > [vn18:99848] Signal code: (-6) > >> > [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790)[0x2aaaadbb4790] > >> > [vn18:99838] [ 1] [vn18:99848] [ 0] /lib64/libpthread.so.0(+0xf790 > >> )[0x2aaaadbb4790] > >> > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > >> > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > >> > [vn18:99838] [ 3] [vn18:99849] [ 0] /lib64/libpthread.so.0(+0xf790 > >> )[0x2aaaadbb4790] > >> > [vn18:99849] [ 1] [vn15:122106] *** Process received signal *** > >> > [vn15:122106] Signal: Aborted (6) > >> > [vn15:122106] Signal code: (-6) > >> > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > >> > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > >> > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4 > >> _HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscTraceBac > >> kErrorHandler+0x503)[0x2aaaaae1d7e1] > >> > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > >> > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > >> > [vn18:99849] [ 3] [vn15:122107] *** Process received signal *** > >> > [vn15:122107] Signal: Aborted (6) > >> > [vn15:122107] Signal code: (-6) > >> > [vn16:86295] *** Process received signal *** > >> > [vn16:86295] Signal: Aborted (6) > >> > [vn16:86295] Signal code: (-6) > >> > [vn18:99824] *** Process received signal *** > >> > [vn18:99824] Signal: Aborted (6) > >> > [vn18:99824] Signal code: (-6) > >> > [vn18:99844] *** Process received signal *** > >> > [vn18:99844] Signal: Aborted (6) > >> > [vn18:99844] Signal code: (-6) > >> > /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/lib > >> /libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] > >> > > >> > > >> > > >> > -- > >> > Hao Zhang > >> > Dept. of Applid Mathematics and Statistics, > >> > Stony Brook University, > >> > Stony Brook, New York, 11790 > >> > >> > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > > > > > From hbcbh1999 at gmail.com Fri Sep 29 19:48:29 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Fri, 29 Sep 2017 20:48:29 -0400 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: Ok. Thanks. I will send the log as soon as the program finishes. On Fri, Sep 29, 2017 at 8:45 PM, Satish Balay wrote: > Can you run this code with valgrind and send the log? > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > Satish > > On Sat, 30 Sep 2017, Hao Zhang wrote: > > > I've modified my source code accordingly using PETSc 3.8.0. It seems no > > more useful info got printed out. > > > > On Fri, Sep 29, 2017 at 8:15 PM, Hao Zhang wrote: > > > > > I'm using -fp_trap as I run the simulation. the output only gives > > > > > > floating point exception > > > > > > It doesn't matter coarse grid or medium grid or more refined grid. the > > > program crashed right away. > > > > > > On Fri, Sep 29, 2017 at 12:19 AM, Barry Smith > wrote: > > > > > >> > > >> Please upgrade to the latest 3.8 version of PETSc. and then run with > > >> -fp_trap > > >> > > >> This message is usually an indication that a NaN or Inf got into > the > > >> numerical computation. Using the latest PETSc will make it much > easier to > > >> track down. > > >> > > >> Barry > > >> > > >> > On Sep 28, 2017, at 8:57 PM, Hao Zhang wrote: > > >> > > > >> > I'm experiencing error when doing mesh refinement(mesh x2 for X, Y > and > > >> Z direction) for 3D poisson problem. > > >> > PETSc produces no errors when no mesh refinement. I've pasted some > > >> debugging info here. Please advise. > > >> > > > >> > > > >> > [0]PETSC ERROR: --------------------- Error Message > > >> -------------------------------------------------------------- > > >> > [0]PETSC ERROR: Invalid argument > > >> > [0]PETSC ERROR: Scalar value must be same on all processes, > argument # 3 > > >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ > documentation/faq.html > > >> for trouble shooting. > > >> > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 > > >> > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid on a > > >> arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 23:47:39 2017 > > >> > [0]PETSC ERROR: Configure options --with-mpi-dir=/cm/shared/ > apps/openmpi/gcc/64/1.8.5/ > > >> --download-fblaslapack=/home/haozhang/tmp/fblaslapack-3.4.2.tar.gz > > >> --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz > > >> --with-valgrind-include=/home/haozhang/include/valgrind/ > > >> > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/vec/vec/interface/rvector.c > > >> > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/impls/bcgs/bcgs.c > > >> > [0]PETSC ERROR: #3 KSPSolve() line 460 in > /home/haozhang/tmp/petsc-3.5.4 > > >> _HYPRE/src/ksp/ksp/interface/itfunc.c > > >> > [vn18:99838] *** Process received signal *** > > >> > [vn18:99838] Signal: Aborted (6) > > >> > [vn18:99838] Signal code: (-6) > > >> > [vn18:99848] *** Process received signal *** > > >> > [vn18:99849] *** Process received signal *** > > >> > [vn18:99849] Signal: Aborted (6) > > >> > [vn18:99849] Signal code: (-6) > > >> > [vn18:99848] Signal: Aborted (6) > > >> > [vn18:99848] Signal code: (-6) > > >> > [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790)[0x2aaaadbb4790] > > >> > [vn18:99838] [ 1] [vn18:99848] [ 0] /lib64/libpthread.so.0(+0xf790 > > >> )[0x2aaaadbb4790] > > >> > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > >> > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > >> > [vn18:99838] [ 3] [vn18:99849] [ 0] /lib64/libpthread.so.0(+0xf790 > > >> )[0x2aaaadbb4790] > > >> > [vn18:99849] [ 1] [vn15:122106] *** Process received signal *** > > >> > [vn15:122106] Signal: Aborted (6) > > >> > [vn15:122106] Signal code: (-6) > > >> > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > >> > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > >> > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4 > > >> _HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscTraceBac > > >> kErrorHandler+0x503)[0x2aaaaae1d7e1] > > >> > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > >> > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > >> > [vn18:99849] [ 3] [vn15:122107] *** Process received signal *** > > >> > [vn15:122107] Signal: Aborted (6) > > >> > [vn15:122107] Signal code: (-6) > > >> > [vn16:86295] *** Process received signal *** > > >> > [vn16:86295] Signal: Aborted (6) > > >> > [vn16:86295] Signal code: (-6) > > >> > [vn18:99824] *** Process received signal *** > > >> > [vn18:99824] Signal: Aborted (6) > > >> > [vn18:99824] Signal code: (-6) > > >> > [vn18:99844] *** Process received signal *** > > >> > [vn18:99844] Signal: Aborted (6) > > >> > [vn18:99844] Signal code: (-6) > > >> > /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/lib > > >> /libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] > > >> > > > >> > > > >> > > > >> > -- > > >> > Hao Zhang > > >> > Dept. of Applid Mathematics and Statistics, > > >> > Stony Brook University, > > >> > Stony Brook, New York, 11790 > > >> > > >> > > > > > > > > > -- > > > Hao Zhang > > > Dept. of Applid Mathematics and Statistics, > > > Stony Brook University, > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbcbh1999 at gmail.com Fri Sep 29 20:19:35 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Fri, 29 Sep 2017 21:19:35 -0400 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: ==31497== Use of uninitialised value of size 8 ==31497== at 0x85AEA9B: _itoa_word (in /lib64/libc-2.12.so) ==31497== by 0x85B1652: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x866AB4F: __vsnprintf_chk (in /lib64/libc-2.12.so) ==31497== by 0x866AA89: __snprintf_chk (in /lib64/libc-2.12.so) ==31497== by 0x10A847F6: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xD6A7D18: ompi_common_verbs_find_ports (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xD48BC1F: usnic_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_usnic.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85AEAA5: _itoa_word (in /lib64/libc-2.12.so) ==31497== by 0x85B1652: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x866AB4F: __vsnprintf_chk (in /lib64/libc-2.12.so) ==31497== by 0x866AA89: __snprintf_chk (in /lib64/libc-2.12.so) ==31497== by 0x10A847F6: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xD6A7D18: ompi_common_verbs_find_ports (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xD48BC1F: usnic_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_usnic.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xD6A7D2E: ompi_common_verbs_find_ports (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xD48BC1F: usnic_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_usnic.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xD6A7D59: ompi_common_verbs_find_ports (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xD48BC1F: usnic_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_usnic.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xD6A7D2E: ompi_common_verbs_find_ports (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xD48C752: usnic_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_usnic.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xD6A7D59: ompi_common_verbs_find_ports (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xD48C752: usnic_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_usnic.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE316F5E: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309719: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B6C1: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B203: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0xE316F6A: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309719: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B6C1: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B203: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0xE316FC9: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309719: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B6C1: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B203: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE316FD7: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309719: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B6C1: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B203: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0xE316F7C: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309719: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B6C1: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B203: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xE30A677: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B254: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffe9b8 is on thread 1's stack ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xE30A472: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B254: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffe860 is on thread 1's stack ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4C29B6A: memcpy (mc_replace_strmem.c:119) ==31497== by 0xE30A48B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B254: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4C29B8C: memcpy (mc_replace_strmem.c:882) ==31497== by 0xE30A48B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B254: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4C29C18: memcpy (mc_replace_strmem.c:882) ==31497== by 0xE30A48B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B254: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4C29C3E: memcpy (mc_replace_strmem.c:882) ==31497== by 0xE30A48B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B254: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4C29C5B: memcpy (mc_replace_strmem.c:882) ==31497== by 0xE30A48B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B254: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4C29B6A: memcpy (mc_replace_strmem.c:119) ==31497== by 0xE30A4A3: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B254: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE30A4F8: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B254: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xE30A592: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A6B7: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B254: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffe860 is on thread 1's stack ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xE30A05E: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A0AD: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B269: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffea70 is on thread 1's stack ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE30A0B2: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B269: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE30A0DA: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B269: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE309DA5: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A10C: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B269: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0xE316F31: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309E8B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A10C: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B269: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0xE316F35: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309E8B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A10C: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B269: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B467: rdma_create_event_channel (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DBE: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE316F5E: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309719: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B6C1: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DE1: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0xE316F6A: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309719: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B6C1: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DE1: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0xE316F7C: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309719: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B6C1: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1DE1: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xE30A677: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1E9E: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffea48 is on thread 1's stack ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xE30A05E: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A0AD: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1F71: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffeb00 is on thread 1's stack ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE30A0B2: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1F71: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE30A0DA: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1F71: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE309DA5: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A10C: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1F71: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0xE316F31: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309E8B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A10C: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1F71: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0xE316F35: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309E8B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A10C: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1F71: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE30A4F8: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1E9E: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE30976C: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A501: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1E9E: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE3095DF: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE3097C3: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A501: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1E9E: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE309601: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE3097C3: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A501: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1E9E: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xDABACE1: ibv_cmd_dealloc_pd (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A846AC: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE309E40: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A10C: rdma_destroy_id (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F1F71: mca_btl_openib_build_rdma_addr_list (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F35A9: rdmacm_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2940: ompi_btl_openib_connect_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E6184: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffead8 is on thread 1's stack ==31497== ==31497== Syscall param mmap(offset) contains uninitialised byte(s) ==31497== at 0x865038A: mmap (in /lib64/libc-2.12.so) ==31497== by 0x10A8454B: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0D8E: ibv_create_cq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xD6A8AB4: ompi_common_verbs_qp_test (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2DEF: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xDABCABC: ibv_cmd_create_qp (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A8406D: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0B71: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xD6A8A1C: make_qp (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xD6A8AF2: ompi_common_verbs_qp_test (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2DEF: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffe968 is on thread 1's stack ==31497== ==31497== Syscall param mmap(length) contains uninitialised byte(s) ==31497== at 0x865038A: mmap (in /lib64/libc-2.12.so) ==31497== by 0x10A8412C: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0B71: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xD6A8A1C: make_qp (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xD6A8AF2: ompi_common_verbs_qp_test (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2DEF: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param mmap(offset) contains uninitialised byte(s) ==31497== at 0x865038A: mmap (in /lib64/libc-2.12.so) ==31497== by 0x10A8412C: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0B71: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xD6A8A1C: make_qp (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xD6A8AF2: ompi_common_verbs_qp_test (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2DEF: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xDABC45A: ibv_cmd_destroy_qp (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A83EB5: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xD6A8A2B: make_qp (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xD6A8AF2: ompi_common_verbs_qp_test (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2DEF: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffe9b0 is on thread 1's stack ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xDABC48E: ibv_cmd_destroy_qp (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A83EB5: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xD6A8A2B: make_qp (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xD6A8AF2: ompi_common_verbs_qp_test (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2DEF: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param munmap(length) contains uninitialised byte(s) ==31497== at 0x86503B7: munmap (in /lib64/libc-2.12.so) ==31497== by 0x10A83EEB: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xD6A8A2B: make_qp (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xD6A8AF2: ompi_common_verbs_qp_test (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2DEF: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xDABACE1: ibv_cmd_dealloc_pd (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A846AC: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xD6A8AD8: ompi_common_verbs_qp_test (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2DEF: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffea28 is on thread 1's stack ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xDABC5EA: ibv_cmd_destroy_cq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A84275: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xD6A8AE0: ompi_common_verbs_qp_test (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2DEF: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffea00 is on thread 1's stack ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xDABC632: ibv_cmd_destroy_cq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A84275: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xD6A8AE0: ompi_common_verbs_qp_test (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2DEF: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xDABC63B: ibv_cmd_destroy_cq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A84275: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xD6A8AE0: ompi_common_verbs_qp_test (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2DEF: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param munmap(length) contains uninitialised byte(s) ==31497== at 0x86503B7: munmap (in /lib64/libc-2.12.so) ==31497== by 0x10A84297: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xD6A8AE0: ompi_common_verbs_qp_test (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2DEF: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0E3173: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0E318B: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0E351C: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0EF658: ompi_btl_openib_ini_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E353A: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0EF65E: ompi_btl_openib_ini_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E353A: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0x85AEA9B: _itoa_word (in /lib64/libc-2.12.so) ==31497== by 0x85B1652: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85DA3D9: vasprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA347: asprintf (in /lib64/libc-2.12.so) ==31497== by 0xE0E3872: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85AEAA5: _itoa_word (in /lib64/libc-2.12.so) ==31497== by 0x85B1652: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85DA3D9: vasprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA347: asprintf (in /lib64/libc-2.12.so) ==31497== by 0xE0E3872: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0E3930: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0E393F: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0E258A: init_one_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E3CB0: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0E2638: init_one_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E3CB0: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0E2642: init_one_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E3CB0: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0E268A: init_one_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E3CB0: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0x85AEA9B: _itoa_word (in /lib64/libc-2.12.so) ==31497== by 0x85B1652: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85D4608: vsprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA2B7: sprintf (in /lib64/libc-2.12.so) ==31497== by 0xE0E2925: init_one_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E3CB0: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85AEAA5: _itoa_word (in /lib64/libc-2.12.so) ==31497== by 0x85B1652: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85D4608: vsprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA2B7: sprintf (in /lib64/libc-2.12.so) ==31497== by 0xE0E2925: init_one_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E3CB0: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xD6A88FA: ompi_common_verbs_port_bw (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2B9C: init_one_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E3CB0: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0xD6A8913: ompi_common_verbs_port_bw (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2B9C: init_one_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E3CB0: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xD6A892D: ompi_common_verbs_port_bw (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmca_common_verbs.so.0.2.3) ==31497== by 0xE0E2B9C: init_one_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E3CB0: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0E2B14: init_one_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E3CB0: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0E38FB: init_one_device (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E66B8: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE316F5E: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309719: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B6C1: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F5C9D: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0xE316F6A: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309719: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B6C1: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F5C9D: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0xE316F7C: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE309719: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30B6C1: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F5C9D: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xE30A677: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F5DD8: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffea78 is on thread 1's stack ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE30A4F8: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30A688: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AB7A: rdma_bind_addr (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F5DD8: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0F1B05: mca_btl_openib_rdma_get_ipv4addr (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F5E08: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xE30AA40: rdma_listen (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F6032: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffeb78 is on thread 1's stack ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xE30A472: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AA6F: rdma_listen (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F6032: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffea20 is on thread 1's stack ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4C29B6A: memcpy (mc_replace_strmem.c:119) ==31497== by 0xE30A48B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AA6F: rdma_listen (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F6032: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4C29B8C: memcpy (mc_replace_strmem.c:882) ==31497== by 0xE30A48B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AA6F: rdma_listen (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F6032: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4C29C18: memcpy (mc_replace_strmem.c:882) ==31497== by 0xE30A48B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AA6F: rdma_listen (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F6032: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4C29C3E: memcpy (mc_replace_strmem.c:882) ==31497== by 0xE30A48B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AA6F: rdma_listen (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F6032: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4C29C5B: memcpy (mc_replace_strmem.c:882) ==31497== by 0xE30A48B: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AA6F: rdma_listen (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F6032: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4C29B6A: memcpy (mc_replace_strmem.c:119) ==31497== by 0xE30A4A3: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AA6F: rdma_listen (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F6032: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE30A4F8: ??? (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE30AA6F: rdma_listen (in /usr/lib64/librdmacm.so.1.0.0) ==31497== by 0xE0F6032: rdmacm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param mmap(offset) contains uninitialised byte(s) ==31497== at 0x865038A: mmap (in /lib64/libc-2.12.so) ==31497== by 0x10A8454B: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0D8E: ibv_create_cq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xE0F763C: udcm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param mmap(offset) contains uninitialised byte(s) ==31497== at 0x865038A: mmap (in /lib64/libc-2.12.so) ==31497== by 0x10A8454B: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0D8E: ibv_create_cq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xE0F7668: udcm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xDABAAE6: ibv_cmd_reg_mr (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A84659: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC10C2: ibv_reg_mr (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xE0F770C: udcm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffea78 is on thread 1's stack ==31497== ==31497== Syscall param mmap(length) contains uninitialised byte(s) ==31497== at 0x865038A: mmap (in /lib64/libc-2.12.so) ==31497== by 0x10A8412C: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0B71: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xE0F77C4: udcm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param mmap(offset) contains uninitialised byte(s) ==31497== at 0x865038A: mmap (in /lib64/libc-2.12.so) ==31497== by 0x10A8412C: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0B71: ibv_create_qp (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xE0F77C4: udcm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xDABBF83: ibv_cmd_modify_qp (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A83F34: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0913: ibv_modify_qp (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xE0F7825: udcm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffea88 is on thread 1's stack ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x10A838FD: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0F7931: udcm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x10A83920: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0F7931: udcm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0x10A8393F: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0F7931: udcm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0x10A83966: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0F7931: udcm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0x10A83975: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0F7931: udcm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== ==31497== More than 100 errors detected. Subsequent errors ==31497== will still be recorded, but in less detail than before. ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xDABA90C: ibv_cmd_req_notify_cq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xE0F7A91: udcm_component_query (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0F2B63: ompi_btl_openib_connect_base_select_for_local_port (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E7216: btl_openib_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0x7959D5E: mca_btl_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xD079751: mca_bml_r2_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0x79595B8: mca_bml_base_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0xE9B41B4: mca_pml_ob1_component_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x796A07E: mca_pml_base_select (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x791E253: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffead8 is on thread 1's stack ==31497== ==31497== Thread 2: ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xB83D2A6: send_bytes (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_oob_tcp.so) ==31497== by 0xB83D6D4: mca_oob_tcp_send_handler (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_oob_tcp.so) ==31497== by 0x97A69EB: opal_libevent2021_event_base_loop (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libopen-pal.so.6.2.1) ==31497== by 0x9507DAD: orte_progress_thread_engine (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libopen-rte.so.7.0.5) ==31497== by 0x8355A50: start_thread (in /lib64/libpthread-2.12.so) ==31497== by 0x86539AC: clone (in /lib64/libc-2.12.so) ==31497== Address 0xf43d11f is 239 bytes inside a block of size 512 alloc'd ==31497== at 0x4C27C20: realloc (vg_replace_malloc.c:662) ==31497== by 0x9774F72: opal_dss_buffer_extend (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libopen-pal.so.6.2.1) ==31497== by 0x9775BDC: opal_dss_copy_payload (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libopen-pal.so.6.2.1) ==31497== by 0x950C035: orte_grpcomm_base_pack_collective (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libopen-rte.so.7.0.5) ==31497== by 0xB62E613: process_allgather (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_grpcomm_bad.so) ==31497== by 0x97A69EB: opal_libevent2021_event_base_loop (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libopen-pal.so.6.2.1) ==31497== by 0x9507DAD: orte_progress_thread_engine (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libopen-rte.so.7.0.5) ==31497== by 0x8355A50: start_thread (in /lib64/libpthread-2.12.so) ==31497== by 0x86539AC: clone (in /lib64/libc-2.12.so) ==31497== ==31497== Thread 1: ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xE0E0205: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffee14 is on thread 1's stack ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0EF977: btl_openib_async_command_done (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0E021D: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xE0E0824: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xDABA9BF: ibv_cmd_create_cq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A84517: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0D8E: ibv_create_cq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xE0DEC77: adjust_cq (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0DFBF9: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffec18 is on thread 1's stack ==31497== ==31497== Syscall param mmap(length) contains uninitialised byte(s) ==31497== at 0x865038A: mmap (in /lib64/libc-2.12.so) ==31497== by 0x10A8454B: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0D8E: ibv_create_cq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xE0DEC77: adjust_cq (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xE0DFBF9: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xDABCEE1: ibv_cmd_create_srq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A83D16: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0C1B: ibv_create_srq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xE0E028B: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffec58 is on thread 1's stack ==31497== ==31497== Syscall param mmap(length) contains uninitialised byte(s) ==31497== at 0x865038A: mmap (in /lib64/libc-2.12.so) ==31497== by 0x10A83D67: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0C1B: ibv_create_srq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xE0E028B: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param mmap(offset) contains uninitialised byte(s) ==31497== at 0x865038A: mmap (in /lib64/libc-2.12.so) ==31497== by 0x10A83D67: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xDAC0C1B: ibv_create_srq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0xE0E028B: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x10A838FD: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0E031C: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x10A83920: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0E031C: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0x10A8393F: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0E031C: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0x10A83966: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0E031C: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Use of uninitialised value of size 8 ==31497== at 0x10A83975: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0E031C: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x10A83997: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0E031C: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xDABC3D6: ibv_cmd_modify_srq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A83B22: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0E0356: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffec78 is on thread 1's stack ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x835C77D: ??? (in /lib64/libpthread-2.12.so) ==31497== by 0xDABC52A: ibv_cmd_destroy_srq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A83A45: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0E036B: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== Address 0x7feffec70 is on thread 1's stack ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0xDABC55B: ibv_cmd_destroy_srq (in /usr/lib64/libibverbs.so.1.0.0) ==31497== by 0x10A83A45: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0E036B: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Syscall param munmap(length) contains uninitialised byte(s) ==31497== at 0x86503B7: munmap (in /lib64/libc-2.12.so) ==31497== by 0x10A83A76: ??? (in /usr/lib64/libipathverbs-rdmav2.so) ==31497== by 0xE0E036B: mca_btl_openib_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_btl_openib.so) ==31497== by 0xD078381: mca_bml_r2_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_bml_r2.so) ==31497== by 0xE9B33F9: mca_pml_ob1_add_procs (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/openmpi/mca_pml_ob1.so) ==31497== by 0x791E61F: ompi_mpi_init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x793C26F: PMPI_Init (in /cm/shared/apps/openmpi/gcc/64/1.8.5/lib64/libmpi.so.1.6.0) ==31497== by 0x6B0652: pp_init (ppsub.c:178) ==31497== by 0x54EAB2: FT_Init (fmap.c:385) ==31497== by 0x40BD5E: main (iFluid.cpp:96) ==31497== ==31497== Invalid write of size 8 ==31497== at 0x643685: level_wave_func_Meniscus (imkcurve.c:1826) ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) ==31497== by 0x64623B: make_level_surface (imksurf.c:245) ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) ==31497== by 0x40C072: main (iFluid.cpp:218) ==31497== Address 0x7fe859c38 is on thread 1's stack ==31497== ==31497== Invalid write of size 8 ==31497== at 0x64368C: level_wave_func_Meniscus (imkcurve.c:1826) ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) ==31497== by 0x64623B: make_level_surface (imksurf.c:245) ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) ==31497== by 0x40C072: main (iFluid.cpp:218) ==31497== Address 0x7fe859c30 is on thread 1's stack ==31497== ==31497== Invalid read of size 8 ==31497== at 0x643693: level_wave_func_Meniscus (imkcurve.c:1831) ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) ==31497== by 0x64623B: make_level_surface (imksurf.c:245) ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) ==31497== by 0x40C072: main (iFluid.cpp:218) ==31497== Address 0x7fe859c38 is on thread 1's stack ==31497== ==31497== Invalid write of size 8 ==31497== at 0x64372D: level_wave_func_Meniscus (imkcurve.c:1848) ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) ==31497== by 0x64623B: make_level_surface (imksurf.c:245) ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) ==31497== by 0x40C072: main (iFluid.cpp:218) ==31497== Address 0x7fe859c18 is on thread 1's stack ==31497== ==31497== Invalid read of size 8 ==31497== at 0x7ED3127: tan (in /lib64/libm-2.12.so) ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) ==31497== by 0x64623B: make_level_surface (imksurf.c:245) ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) ==31497== by 0x40C072: main (iFluid.cpp:218) ==31497== Address 0x7fe859c18 is on thread 1's stack ==31497== ==31497== Invalid read of size 8 ==31497== at 0x643755: level_wave_func_Meniscus (imkcurve.c:1853) ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) ==31497== by 0x64623B: make_level_surface (imksurf.c:245) ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) ==31497== by 0x40C072: main (iFluid.cpp:218) ==31497== Address 0x7fe859c30 is on thread 1's stack ==31497== ==31497== Invalid read of size 8 ==31497== at 0x643765: level_wave_func_Meniscus (imkcurve.c:1854) ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) ==31497== by 0x64623B: make_level_surface (imksurf.c:245) ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) ==31497== by 0x40C072: main (iFluid.cpp:218) ==31497== Address 0x7fe859c30 is on thread 1's stack ==31497== ==31497== Invalid read of size 8 ==31497== at 0x643A7F: level_wave_func_Meniscus (imkcurve.c:1944) ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) ==31497== by 0x64623B: make_level_surface (imksurf.c:245) ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) ==31497== by 0x40C072: main (iFluid.cpp:218) ==31497== Address 0x7fe859c30 is on thread 1's stack ==31497== ==31497== Invalid write of size 8 ==31497== at 0x643A92: level_wave_func_Meniscus (imkcurve.c:1944) ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) ==31497== by 0x64623B: make_level_surface (imksurf.c:245) ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) ==31497== by 0x40C072: main (iFluid.cpp:218) ==31497== Address 0x7feffecf0 is on thread 1's stack ==31497== ==31497== Invalid read of size 8 ==31497== at 0x643AEB: level_wave_func_Meniscus (imkcurve.c:1955) ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) ==31497== by 0x64623B: make_level_surface (imksurf.c:245) ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) ==31497== by 0x40C072: main (iFluid.cpp:218) ==31497== Address 0x7feffecf0 is on thread 1's stack ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x428B9F: Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() (iFbasic.cpp:2799) ==31497== by 0x4B8BB9: Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_RSSY_vd(_LEVEL_FUNC_PACK*) (iFcartsn3d.cpp:15253) ==31497== by 0x40C485: main (iFluid.cpp:349) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x428BA9: Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() (iFbasic.cpp:2799) ==31497== by 0x4B8BB9: Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_RSSY_vd(_LEVEL_FUNC_PACK*) (iFcartsn3d.cpp:15253) ==31497== by 0x40C485: main (iFluid.cpp:349) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85B00E3: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) ==31497== by 0x486108: Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothProperty_velocity_MAC_coupled_vd() (iFcartsn3d.cpp:6101) ==31497== by 0x4CBF0D: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18574) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85B0101: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) ==31497== by 0x486108: Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothProperty_velocity_MAC_coupled_vd() (iFcartsn3d.cpp:6101) ==31497== by 0x4CBF0D: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18574) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4ED737: Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() (iFcartsn3d.cpp:24519) ==31497== by 0x4CBFBC: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18594) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x4ED754: Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() (iFcartsn3d.cpp:24519) ==31497== by 0x4CBFBC: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18594) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85B4CF0: __printf_fp (in /lib64/libc-2.12.so) ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) ==31497== by 0x4ED9F7: Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() (iFcartsn3d.cpp:24549) ==31497== by 0x4CBFBC: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18594) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85B4F91: __printf_fp (in /lib64/libc-2.12.so) ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) ==31497== by 0x4ED9F7: Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() (iFcartsn3d.cpp:24549) ==31497== by 0x4CBFBC: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18594) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85AE396: __mpn_extract_double (in /lib64/libc-2.12.so) ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) ==31497== by 0x4ED9F7: Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() (iFcartsn3d.cpp:24549) ==31497== by 0x4CBFBC: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18594) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85AE39B: __mpn_extract_double (in /lib64/libc-2.12.so) ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) ==31497== by 0x4ED9F7: Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() (iFcartsn3d.cpp:24549) ==31497== by 0x4CBFBC: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18594) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85B59EC: __printf_fp (in /lib64/libc-2.12.so) ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) ==31497== by 0x4ED9F7: Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() (iFcartsn3d.cpp:24549) ==31497== by 0x4CBFBC: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18594) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85B5A9D: __printf_fp (in /lib64/libc-2.12.so) ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) ==31497== by 0x4ED9F7: Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() (iFcartsn3d.cpp:24549) ==31497== by 0x4CBFBC: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18594) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85B5C13: __printf_fp (in /lib64/libc-2.12.so) ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) ==31497== by 0x4ED9F7: Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() (iFcartsn3d.cpp:24549) ==31497== by 0x4CBFBC: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18594) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85B5CA4: __printf_fp (in /lib64/libc-2.12.so) ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) ==31497== by 0x4ED9F7: Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() (iFcartsn3d.cpp:24549) ==31497== by 0x4CBFBC: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18594) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85B625E: __printf_fp (in /lib64/libc-2.12.so) ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) ==31497== by 0x4ED9F7: Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() (iFcartsn3d.cpp:24549) ==31497== by 0x4CBFBC: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18594) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Conditional jump or move depends on uninitialised value(s) ==31497== at 0x85B6242: __printf_fp (in /lib64/libc-2.12.so) ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) ==31497== by 0x4ED9F7: Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() (iFcartsn3d.cpp:24549) ==31497== by 0x4CBFBC: Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) (iFcartsn3d.cpp:18594) ==31497== by 0x40C72E: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== ==31497== Syscall param write(buf) points to uninitialised byte(s) ==31497== at 0x86465ED: ??? (in /lib64/libc-2.12.so) ==31497== by 0x85DCAD2: _IO_file_write@@GLIBC_2.2.5 (in /lib64/ libc-2.12.so) ==31497== by 0x85DE084: _IO_do_write@@GLIBC_2.2.5 (in /lib64/libc-2.12.so ) ==31497== by 0x85DD287: _IO_file_sync@@GLIBC_2.2.5 (in /lib64/ libc-2.12.so) ==31497== by 0x85D1999: fflush (in /lib64/libc-2.12.so) ==31497== by 0x6FACFA: preserve_front_advance_front3d (fadv.c:2255) ==31497== by 0x6FA14E: mixed_advance_front3d (fadv.c:1895) ==31497== by 0x6F9B2D: advance_front3d_tracking_control (fadv.c:1703) ==31497== by 0x54E119: FrontAdvance (fmap.c:142) ==31497== by 0x54DF27: FT_Propagate (fmap.c:80) ==31497== by 0x40CA37: ifluid_driver(_Front*, Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:512) ==31497== by 0x40C4F3: main (iFluid.cpp:369) ==31497== Address 0x412308a is not stack'd, malloc'd or (recently) free'd ==31497== On Fri, Sep 29, 2017 at 8:45 PM, Satish Balay wrote: > Can you run this code with valgrind and send the log? > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > Satish > > On Sat, 30 Sep 2017, Hao Zhang wrote: > > > I've modified my source code accordingly using PETSc 3.8.0. It seems no > > more useful info got printed out. > > > > On Fri, Sep 29, 2017 at 8:15 PM, Hao Zhang wrote: > > > > > I'm using -fp_trap as I run the simulation. the output only gives > > > > > > floating point exception > > > > > > It doesn't matter coarse grid or medium grid or more refined grid. the > > > program crashed right away. > > > > > > On Fri, Sep 29, 2017 at 12:19 AM, Barry Smith > wrote: > > > > > >> > > >> Please upgrade to the latest 3.8 version of PETSc. and then run with > > >> -fp_trap > > >> > > >> This message is usually an indication that a NaN or Inf got into > the > > >> numerical computation. Using the latest PETSc will make it much > easier to > > >> track down. > > >> > > >> Barry > > >> > > >> > On Sep 28, 2017, at 8:57 PM, Hao Zhang wrote: > > >> > > > >> > I'm experiencing error when doing mesh refinement(mesh x2 for X, Y > and > > >> Z direction) for 3D poisson problem. > > >> > PETSc produces no errors when no mesh refinement. I've pasted some > > >> debugging info here. Please advise. > > >> > > > >> > > > >> > [0]PETSC ERROR: --------------------- Error Message > > >> -------------------------------------------------------------- > > >> > [0]PETSC ERROR: Invalid argument > > >> > [0]PETSC ERROR: Scalar value must be same on all processes, > argument # 3 > > >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ > documentation/faq.html > > >> for trouble shooting. > > >> > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 > > >> > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid on a > > >> arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 23:47:39 2017 > > >> > [0]PETSC ERROR: Configure options --with-mpi-dir=/cm/shared/ > apps/openmpi/gcc/64/1.8.5/ > > >> --download-fblaslapack=/home/haozhang/tmp/fblaslapack-3.4.2.tar.gz > > >> --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz > > >> --with-valgrind-include=/home/haozhang/include/valgrind/ > > >> > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/vec/vec/interface/rvector.c > > >> > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/impls/bcgs/bcgs.c > > >> > [0]PETSC ERROR: #3 KSPSolve() line 460 in > /home/haozhang/tmp/petsc-3.5.4 > > >> _HYPRE/src/ksp/ksp/interface/itfunc.c > > >> > [vn18:99838] *** Process received signal *** > > >> > [vn18:99838] Signal: Aborted (6) > > >> > [vn18:99838] Signal code: (-6) > > >> > [vn18:99848] *** Process received signal *** > > >> > [vn18:99849] *** Process received signal *** > > >> > [vn18:99849] Signal: Aborted (6) > > >> > [vn18:99849] Signal code: (-6) > > >> > [vn18:99848] Signal: Aborted (6) > > >> > [vn18:99848] Signal code: (-6) > > >> > [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790)[0x2aaaadbb4790] > > >> > [vn18:99838] [ 1] [vn18:99848] [ 0] /lib64/libpthread.so.0(+0xf790 > > >> )[0x2aaaadbb4790] > > >> > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > >> > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > >> > [vn18:99838] [ 3] [vn18:99849] [ 0] /lib64/libpthread.so.0(+0xf790 > > >> )[0x2aaaadbb4790] > > >> > [vn18:99849] [ 1] [vn15:122106] *** Process received signal *** > > >> > [vn15:122106] Signal: Aborted (6) > > >> > [vn15:122106] Signal code: (-6) > > >> > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > >> > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > >> > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4 > > >> _HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscTraceBac > > >> kErrorHandler+0x503)[0x2aaaaae1d7e1] > > >> > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > >> > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > >> > [vn18:99849] [ 3] [vn15:122107] *** Process received signal *** > > >> > [vn15:122107] Signal: Aborted (6) > > >> > [vn15:122107] Signal code: (-6) > > >> > [vn16:86295] *** Process received signal *** > > >> > [vn16:86295] Signal: Aborted (6) > > >> > [vn16:86295] Signal code: (-6) > > >> > [vn18:99824] *** Process received signal *** > > >> > [vn18:99824] Signal: Aborted (6) > > >> > [vn18:99824] Signal code: (-6) > > >> > [vn18:99844] *** Process received signal *** > > >> > [vn18:99844] Signal: Aborted (6) > > >> > [vn18:99844] Signal code: (-6) > > >> > /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/lib > > >> /libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] > > >> > > > >> > > > >> > > > >> > -- > > >> > Hao Zhang > > >> > Dept. of Applid Mathematics and Statistics, > > >> > Stony Brook University, > > >> > Stony Brook, New York, 11790 > > >> > > >> > > > > > > > > > -- > > > Hao Zhang > > > Dept. of Applid Mathematics and Statistics, > > > Stony Brook University, > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Sep 29 20:31:21 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 29 Sep 2017 20:31:21 -0500 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: Most of this is from openmpi. Its best to use --download-mpich for valgrind clean mpi as mentioned below. http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [attempting to remove openmpi messages] - all messages below appear to be from your appliation code. And I don't see any messages from PETSc code in this log. Once there is memory corruption we get random behavior from application. So its best to fix all these issues and make this application valgrind clean. Satish On Sat, 30 Sep 2017, Hao Zhang wrote: > ==31497== Invalid write of size 8 > ==31497== at 0x643685: level_wave_func_Meniscus (imkcurve.c:1826) > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > ==31497== by 0x40C072: main (iFluid.cpp:218) > ==31497== Address 0x7fe859c38 is on thread 1's stack > ==31497== > ==31497== Invalid write of size 8 > ==31497== at 0x64368C: level_wave_func_Meniscus (imkcurve.c:1826) > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > ==31497== by 0x40C072: main (iFluid.cpp:218) > ==31497== Address 0x7fe859c30 is on thread 1's stack > ==31497== > ==31497== Invalid read of size 8 > ==31497== at 0x643693: level_wave_func_Meniscus (imkcurve.c:1831) > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > ==31497== by 0x40C072: main (iFluid.cpp:218) > ==31497== Address 0x7fe859c38 is on thread 1's stack > ==31497== > ==31497== Invalid write of size 8 > ==31497== at 0x64372D: level_wave_func_Meniscus (imkcurve.c:1848) > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > ==31497== by 0x40C072: main (iFluid.cpp:218) > ==31497== Address 0x7fe859c18 is on thread 1's stack > ==31497== > ==31497== Invalid read of size 8 > ==31497== at 0x7ED3127: tan (in /lib64/libm-2.12.so) > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > ==31497== by 0x40C072: main (iFluid.cpp:218) > ==31497== Address 0x7fe859c18 is on thread 1's stack > ==31497== > ==31497== Invalid read of size 8 > ==31497== at 0x643755: level_wave_func_Meniscus (imkcurve.c:1853) > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > ==31497== by 0x40C072: main (iFluid.cpp:218) > ==31497== Address 0x7fe859c30 is on thread 1's stack > ==31497== > ==31497== Invalid read of size 8 > ==31497== at 0x643765: level_wave_func_Meniscus (imkcurve.c:1854) > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > ==31497== by 0x40C072: main (iFluid.cpp:218) > ==31497== Address 0x7fe859c30 is on thread 1's stack > ==31497== > ==31497== Invalid read of size 8 > ==31497== at 0x643A7F: level_wave_func_Meniscus (imkcurve.c:1944) > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > ==31497== by 0x40C072: main (iFluid.cpp:218) > ==31497== Address 0x7fe859c30 is on thread 1's stack > ==31497== > ==31497== Invalid write of size 8 > ==31497== at 0x643A92: level_wave_func_Meniscus (imkcurve.c:1944) > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > ==31497== by 0x40C072: main (iFluid.cpp:218) > ==31497== Address 0x7feffecf0 is on thread 1's stack > ==31497== > ==31497== Invalid read of size 8 > ==31497== at 0x643AEB: level_wave_func_Meniscus (imkcurve.c:1955) > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > ==31497== by 0x40C072: main (iFluid.cpp:218) > ==31497== Address 0x7feffecf0 is on thread 1's stack > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x428B9F: > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() (iFbasic.cpp:2799) > ==31497== by 0x4B8BB9: > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_RSSY_vd(_LEVEL_FUNC_PACK*) > (iFcartsn3d.cpp:15253) > ==31497== by 0x40C485: main (iFluid.cpp:349) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x428BA9: > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() (iFbasic.cpp:2799) > ==31497== by 0x4B8BB9: > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_RSSY_vd(_LEVEL_FUNC_PACK*) > (iFcartsn3d.cpp:15253) > ==31497== by 0x40C485: main (iFluid.cpp:349) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x85B00E3: vfprintf (in /lib64/libc-2.12.so) > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > ==31497== by 0x486108: > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothProperty_velocity_MAC_coupled_vd() > (iFcartsn3d.cpp:6101) > ==31497== by 0x4CBF0D: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18574) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x85B0101: vfprintf (in /lib64/libc-2.12.so) > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > ==31497== by 0x486108: > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothProperty_velocity_MAC_coupled_vd() > (iFcartsn3d.cpp:6101) > ==31497== by 0x4CBF0D: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18574) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x4ED737: > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > (iFcartsn3d.cpp:24519) > ==31497== by 0x4CBFBC: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18594) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x4ED754: > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > (iFcartsn3d.cpp:24519) > ==31497== by 0x4CBFBC: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18594) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x85B4CF0: __printf_fp (in /lib64/libc-2.12.so) > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > ==31497== by 0x4ED9F7: > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > (iFcartsn3d.cpp:24549) > ==31497== by 0x4CBFBC: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18594) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x85B4F91: __printf_fp (in /lib64/libc-2.12.so) > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > ==31497== by 0x4ED9F7: > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > (iFcartsn3d.cpp:24549) > ==31497== by 0x4CBFBC: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18594) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x85AE396: __mpn_extract_double (in /lib64/libc-2.12.so) > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > ==31497== by 0x4ED9F7: > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > (iFcartsn3d.cpp:24549) > ==31497== by 0x4CBFBC: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18594) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x85AE39B: __mpn_extract_double (in /lib64/libc-2.12.so) > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > ==31497== by 0x4ED9F7: > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > (iFcartsn3d.cpp:24549) > ==31497== by 0x4CBFBC: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18594) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x85B59EC: __printf_fp (in /lib64/libc-2.12.so) > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > ==31497== by 0x4ED9F7: > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > (iFcartsn3d.cpp:24549) > ==31497== by 0x4CBFBC: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18594) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x85B5A9D: __printf_fp (in /lib64/libc-2.12.so) > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > ==31497== by 0x4ED9F7: > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > (iFcartsn3d.cpp:24549) > ==31497== by 0x4CBFBC: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18594) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x85B5C13: __printf_fp (in /lib64/libc-2.12.so) > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > ==31497== by 0x4ED9F7: > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > (iFcartsn3d.cpp:24549) > ==31497== by 0x4CBFBC: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18594) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x85B5CA4: __printf_fp (in /lib64/libc-2.12.so) > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > ==31497== by 0x4ED9F7: > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > (iFcartsn3d.cpp:24549) > ==31497== by 0x4CBFBC: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18594) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x85B625E: __printf_fp (in /lib64/libc-2.12.so) > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > ==31497== by 0x4ED9F7: > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > (iFcartsn3d.cpp:24549) > ==31497== by 0x4CBFBC: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18594) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Conditional jump or move depends on uninitialised value(s) > ==31497== at 0x85B6242: __printf_fp (in /lib64/libc-2.12.so) > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > ==31497== by 0x4ED9F7: > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > (iFcartsn3d.cpp:24549) > ==31497== by 0x4CBFBC: > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > (iFcartsn3d.cpp:18594) > ==31497== by 0x40C72E: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== > ==31497== Syscall param write(buf) points to uninitialised byte(s) > ==31497== at 0x86465ED: ??? (in /lib64/libc-2.12.so) > ==31497== by 0x85DCAD2: _IO_file_write@@GLIBC_2.2.5 (in /lib64/ > libc-2.12.so) > ==31497== by 0x85DE084: _IO_do_write@@GLIBC_2.2.5 (in /lib64/libc-2.12.so > ) > ==31497== by 0x85DD287: _IO_file_sync@@GLIBC_2.2.5 (in /lib64/ > libc-2.12.so) > ==31497== by 0x85D1999: fflush (in /lib64/libc-2.12.so) > ==31497== by 0x6FACFA: preserve_front_advance_front3d (fadv.c:2255) > ==31497== by 0x6FA14E: mixed_advance_front3d (fadv.c:1895) > ==31497== by 0x6F9B2D: advance_front3d_tracking_control (fadv.c:1703) > ==31497== by 0x54E119: FrontAdvance (fmap.c:142) > ==31497== by 0x54DF27: FT_Propagate (fmap.c:80) > ==31497== by 0x40CA37: ifluid_driver(_Front*, > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:512) > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > ==31497== Address 0x412308a is not stack'd, malloc'd or (recently) free'd > ==31497== > > On Fri, Sep 29, 2017 at 8:45 PM, Satish Balay wrote: > > > Can you run this code with valgrind and send the log? > > > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > Satish > > > > On Sat, 30 Sep 2017, Hao Zhang wrote: > > > > > I've modified my source code accordingly using PETSc 3.8.0. It seems no > > > more useful info got printed out. > > > > > > On Fri, Sep 29, 2017 at 8:15 PM, Hao Zhang wrote: > > > > > > > I'm using -fp_trap as I run the simulation. the output only gives > > > > > > > > floating point exception > > > > > > > > It doesn't matter coarse grid or medium grid or more refined grid. the > > > > program crashed right away. > > > > > > > > On Fri, Sep 29, 2017 at 12:19 AM, Barry Smith > > wrote: > > > > > > > >> > > > >> Please upgrade to the latest 3.8 version of PETSc. and then run with > > > >> -fp_trap > > > >> > > > >> This message is usually an indication that a NaN or Inf got into > > the > > > >> numerical computation. Using the latest PETSc will make it much > > easier to > > > >> track down. > > > >> > > > >> Barry > > > >> > > > >> > On Sep 28, 2017, at 8:57 PM, Hao Zhang wrote: > > > >> > > > > >> > I'm experiencing error when doing mesh refinement(mesh x2 for X, Y > > and > > > >> Z direction) for 3D poisson problem. > > > >> > PETSc produces no errors when no mesh refinement. I've pasted some > > > >> debugging info here. Please advise. > > > >> > > > > >> > > > > >> > [0]PETSC ERROR: --------------------- Error Message > > > >> -------------------------------------------------------------- > > > >> > [0]PETSC ERROR: Invalid argument > > > >> > [0]PETSC ERROR: Scalar value must be same on all processes, > > argument # 3 > > > >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ > > documentation/faq.html > > > >> for trouble shooting. > > > >> > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 > > > >> > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid on a > > > >> arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 23:47:39 2017 > > > >> > [0]PETSC ERROR: Configure options --with-mpi-dir=/cm/shared/ > > apps/openmpi/gcc/64/1.8.5/ > > > >> --download-fblaslapack=/home/haozhang/tmp/fblaslapack-3.4.2.tar.gz > > > >> --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz > > > >> --with-valgrind-include=/home/haozhang/include/valgrind/ > > > >> > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in > > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/vec/vec/interface/rvector.c > > > >> > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in > > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/impls/bcgs/bcgs.c > > > >> > [0]PETSC ERROR: #3 KSPSolve() line 460 in > > /home/haozhang/tmp/petsc-3.5.4 > > > >> _HYPRE/src/ksp/ksp/interface/itfunc.c > > > >> > [vn18:99838] *** Process received signal *** > > > >> > [vn18:99838] Signal: Aborted (6) > > > >> > [vn18:99838] Signal code: (-6) > > > >> > [vn18:99848] *** Process received signal *** > > > >> > [vn18:99849] *** Process received signal *** > > > >> > [vn18:99849] Signal: Aborted (6) > > > >> > [vn18:99849] Signal code: (-6) > > > >> > [vn18:99848] Signal: Aborted (6) > > > >> > [vn18:99848] Signal code: (-6) > > > >> > [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790)[0x2aaaadbb4790] > > > >> > [vn18:99838] [ 1] [vn18:99848] [ 0] /lib64/libpthread.so.0(+0xf790 > > > >> )[0x2aaaadbb4790] > > > >> > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > > >> > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > > >> > [vn18:99838] [ 3] [vn18:99849] [ 0] /lib64/libpthread.so.0(+0xf790 > > > >> )[0x2aaaadbb4790] > > > >> > [vn18:99849] [ 1] [vn15:122106] *** Process received signal *** > > > >> > [vn15:122106] Signal: Aborted (6) > > > >> > [vn15:122106] Signal code: (-6) > > > >> > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > > >> > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > > >> > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4 > > > >> _HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscTraceBac > > > >> kErrorHandler+0x503)[0x2aaaaae1d7e1] > > > >> > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > > >> > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > > >> > [vn18:99849] [ 3] [vn15:122107] *** Process received signal *** > > > >> > [vn15:122107] Signal: Aborted (6) > > > >> > [vn15:122107] Signal code: (-6) > > > >> > [vn16:86295] *** Process received signal *** > > > >> > [vn16:86295] Signal: Aborted (6) > > > >> > [vn16:86295] Signal code: (-6) > > > >> > [vn18:99824] *** Process received signal *** > > > >> > [vn18:99824] Signal: Aborted (6) > > > >> > [vn18:99824] Signal code: (-6) > > > >> > [vn18:99844] *** Process received signal *** > > > >> > [vn18:99844] Signal: Aborted (6) > > > >> > [vn18:99844] Signal code: (-6) > > > >> > /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/lib > > > >> /libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] > > > >> > > > > >> > > > > >> > > > > >> > -- > > > >> > Hao Zhang > > > >> > Dept. of Applid Mathematics and Statistics, > > > >> > Stony Brook University, > > > >> > Stony Brook, New York, 11790 > > > >> > > > >> > > > > > > > > > > > > -- > > > > Hao Zhang > > > > Dept. of Applid Mathematics and Statistics, > > > > Stony Brook University, > > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > > > > > > > > > > > From hbcbh1999 at gmail.com Fri Sep 29 20:33:55 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Fri, 29 Sep 2017 21:33:55 -0400 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: Thanks. I will recompile and submit a new debug log On Fri, Sep 29, 2017 at 9:31 PM, Satish Balay wrote: > Most of this is from openmpi. Its best to use --download-mpich > for valgrind clean mpi as mentioned below. > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [attempting to remove openmpi messages] - all messages below appear to > be from your appliation code. And I don't see any messages from PETSc > code in this log. > > Once there is memory corruption we get random behavior from > application. So its best to fix all these issues and make this > application valgrind clean. > > Satish > > On Sat, 30 Sep 2017, Hao Zhang wrote: > > > > > ==31497== Invalid write of size 8 > > ==31497== at 0x643685: level_wave_func_Meniscus (imkcurve.c:1826) > > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > ==31497== by 0x40C072: main (iFluid.cpp:218) > > ==31497== Address 0x7fe859c38 is on thread 1's stack > > ==31497== > > ==31497== Invalid write of size 8 > > ==31497== at 0x64368C: level_wave_func_Meniscus (imkcurve.c:1826) > > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > ==31497== by 0x40C072: main (iFluid.cpp:218) > > ==31497== Address 0x7fe859c30 is on thread 1's stack > > ==31497== > > ==31497== Invalid read of size 8 > > ==31497== at 0x643693: level_wave_func_Meniscus (imkcurve.c:1831) > > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > ==31497== by 0x40C072: main (iFluid.cpp:218) > > ==31497== Address 0x7fe859c38 is on thread 1's stack > > ==31497== > > ==31497== Invalid write of size 8 > > ==31497== at 0x64372D: level_wave_func_Meniscus (imkcurve.c:1848) > > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > ==31497== by 0x40C072: main (iFluid.cpp:218) > > ==31497== Address 0x7fe859c18 is on thread 1's stack > > ==31497== > > ==31497== Invalid read of size 8 > > ==31497== at 0x7ED3127: tan (in /lib64/libm-2.12.so) > > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > ==31497== by 0x40C072: main (iFluid.cpp:218) > > ==31497== Address 0x7fe859c18 is on thread 1's stack > > ==31497== > > ==31497== Invalid read of size 8 > > ==31497== at 0x643755: level_wave_func_Meniscus (imkcurve.c:1853) > > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > ==31497== by 0x40C072: main (iFluid.cpp:218) > > ==31497== Address 0x7fe859c30 is on thread 1's stack > > ==31497== > > ==31497== Invalid read of size 8 > > ==31497== at 0x643765: level_wave_func_Meniscus (imkcurve.c:1854) > > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > ==31497== by 0x40C072: main (iFluid.cpp:218) > > ==31497== Address 0x7fe859c30 is on thread 1's stack > > ==31497== > > ==31497== Invalid read of size 8 > > ==31497== at 0x643A7F: level_wave_func_Meniscus (imkcurve.c:1944) > > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > ==31497== by 0x40C072: main (iFluid.cpp:218) > > ==31497== Address 0x7fe859c30 is on thread 1's stack > > ==31497== > > ==31497== Invalid write of size 8 > > ==31497== at 0x643A92: level_wave_func_Meniscus (imkcurve.c:1944) > > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > ==31497== by 0x40C072: main (iFluid.cpp:218) > > ==31497== Address 0x7feffecf0 is on thread 1's stack > > ==31497== > > ==31497== Invalid read of size 8 > > ==31497== at 0x643AEB: level_wave_func_Meniscus (imkcurve.c:1955) > > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > ==31497== by 0x40C072: main (iFluid.cpp:218) > > ==31497== Address 0x7feffecf0 is on thread 1's stack > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x428B9F: > > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() (iFbasic.cpp:2799) > > ==31497== by 0x4B8BB9: > > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_RSSY_vd(_ > LEVEL_FUNC_PACK*) > > (iFcartsn3d.cpp:15253) > > ==31497== by 0x40C485: main (iFluid.cpp:349) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x428BA9: > > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() (iFbasic.cpp:2799) > > ==31497== by 0x4B8BB9: > > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_RSSY_vd(_ > LEVEL_FUNC_PACK*) > > (iFcartsn3d.cpp:15253) > > ==31497== by 0x40C485: main (iFluid.cpp:349) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x85B00E3: vfprintf (in /lib64/libc-2.12.so) > > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > ==31497== by 0x486108: > > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothProperty_ > velocity_MAC_coupled_vd() > > (iFcartsn3d.cpp:6101) > > ==31497== by 0x4CBF0D: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18574) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x85B0101: vfprintf (in /lib64/libc-2.12.so) > > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > ==31497== by 0x486108: > > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothProperty_ > velocity_MAC_coupled_vd() > > (iFcartsn3d.cpp:6101) > > ==31497== by 0x4CBF0D: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18574) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x4ED737: > > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > (iFcartsn3d.cpp:24519) > > ==31497== by 0x4CBFBC: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18594) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x4ED754: > > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > (iFcartsn3d.cpp:24519) > > ==31497== by 0x4CBFBC: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18594) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x85B4CF0: __printf_fp (in /lib64/libc-2.12.so) > > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > ==31497== by 0x4ED9F7: > > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > (iFcartsn3d.cpp:24549) > > ==31497== by 0x4CBFBC: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18594) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x85B4F91: __printf_fp (in /lib64/libc-2.12.so) > > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > ==31497== by 0x4ED9F7: > > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > (iFcartsn3d.cpp:24549) > > ==31497== by 0x4CBFBC: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18594) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x85AE396: __mpn_extract_double (in /lib64/libc-2.12.so) > > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) > > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > ==31497== by 0x4ED9F7: > > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > (iFcartsn3d.cpp:24549) > > ==31497== by 0x4CBFBC: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18594) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x85AE39B: __mpn_extract_double (in /lib64/libc-2.12.so) > > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) > > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > ==31497== by 0x4ED9F7: > > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > (iFcartsn3d.cpp:24549) > > ==31497== by 0x4CBFBC: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18594) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x85B59EC: __printf_fp (in /lib64/libc-2.12.so) > > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > ==31497== by 0x4ED9F7: > > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > (iFcartsn3d.cpp:24549) > > ==31497== by 0x4CBFBC: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18594) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x85B5A9D: __printf_fp (in /lib64/libc-2.12.so) > > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > ==31497== by 0x4ED9F7: > > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > (iFcartsn3d.cpp:24549) > > ==31497== by 0x4CBFBC: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18594) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x85B5C13: __printf_fp (in /lib64/libc-2.12.so) > > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > ==31497== by 0x4ED9F7: > > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > (iFcartsn3d.cpp:24549) > > ==31497== by 0x4CBFBC: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18594) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x85B5CA4: __printf_fp (in /lib64/libc-2.12.so) > > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > ==31497== by 0x4ED9F7: > > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > (iFcartsn3d.cpp:24549) > > ==31497== by 0x4CBFBC: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18594) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x85B625E: __printf_fp (in /lib64/libc-2.12.so) > > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > ==31497== by 0x4ED9F7: > > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > (iFcartsn3d.cpp:24549) > > ==31497== by 0x4CBFBC: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18594) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Conditional jump or move depends on uninitialised value(s) > > ==31497== at 0x85B6242: __printf_fp (in /lib64/libc-2.12.so) > > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > ==31497== by 0x4ED9F7: > > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > (iFcartsn3d.cpp:24549) > > ==31497== by 0x4CBFBC: > > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > (iFcartsn3d.cpp:18594) > > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== > > ==31497== Syscall param write(buf) points to uninitialised byte(s) > > ==31497== at 0x86465ED: ??? (in /lib64/libc-2.12.so) > > ==31497== by 0x85DCAD2: _IO_file_write@@GLIBC_2.2.5 (in /lib64/ > > libc-2.12.so) > > ==31497== by 0x85DE084: _IO_do_write@@GLIBC_2.2.5 (in /lib64/ > libc-2.12.so > > ) > > ==31497== by 0x85DD287: _IO_file_sync@@GLIBC_2.2.5 (in /lib64/ > > libc-2.12.so) > > ==31497== by 0x85D1999: fflush (in /lib64/libc-2.12.so) > > ==31497== by 0x6FACFA: preserve_front_advance_front3d (fadv.c:2255) > > ==31497== by 0x6FA14E: mixed_advance_front3d (fadv.c:1895) > > ==31497== by 0x6F9B2D: advance_front3d_tracking_control (fadv.c:1703) > > ==31497== by 0x54E119: FrontAdvance (fmap.c:142) > > ==31497== by 0x54DF27: FT_Propagate (fmap.c:80) > > ==31497== by 0x40CA37: ifluid_driver(_Front*, > > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:512) > > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > ==31497== Address 0x412308a is not stack'd, malloc'd or (recently) > free'd > > ==31497== > > > > On Fri, Sep 29, 2017 at 8:45 PM, Satish Balay wrote: > > > > > Can you run this code with valgrind and send the log? > > > > > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > > > Satish > > > > > > On Sat, 30 Sep 2017, Hao Zhang wrote: > > > > > > > I've modified my source code accordingly using PETSc 3.8.0. It seems > no > > > > more useful info got printed out. > > > > > > > > On Fri, Sep 29, 2017 at 8:15 PM, Hao Zhang > wrote: > > > > > > > > > I'm using -fp_trap as I run the simulation. the output only gives > > > > > > > > > > floating point exception > > > > > > > > > > It doesn't matter coarse grid or medium grid or more refined grid. > the > > > > > program crashed right away. > > > > > > > > > > On Fri, Sep 29, 2017 at 12:19 AM, Barry Smith > > > wrote: > > > > > > > > > >> > > > > >> Please upgrade to the latest 3.8 version of PETSc. and then run > with > > > > >> -fp_trap > > > > >> > > > > >> This message is usually an indication that a NaN or Inf got > into > > > the > > > > >> numerical computation. Using the latest PETSc will make it much > > > easier to > > > > >> track down. > > > > >> > > > > >> Barry > > > > >> > > > > >> > On Sep 28, 2017, at 8:57 PM, Hao Zhang > wrote: > > > > >> > > > > > >> > I'm experiencing error when doing mesh refinement(mesh x2 for > X, Y > > > and > > > > >> Z direction) for 3D poisson problem. > > > > >> > PETSc produces no errors when no mesh refinement. I've pasted > some > > > > >> debugging info here. Please advise. > > > > >> > > > > > >> > > > > > >> > [0]PETSC ERROR: --------------------- Error Message > > > > >> -------------------------------------------------------------- > > > > >> > [0]PETSC ERROR: Invalid argument > > > > >> > [0]PETSC ERROR: Scalar value must be same on all processes, > > > argument # 3 > > > > >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ > > > documentation/faq.html > > > > >> for trouble shooting. > > > > >> > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 > > > > >> > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid on a > > > > >> arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 23:47:39 > 2017 > > > > >> > [0]PETSC ERROR: Configure options --with-mpi-dir=/cm/shared/ > > > apps/openmpi/gcc/64/1.8.5/ > > > > >> --download-fblaslapack=/home/haozhang/tmp/fblaslapack-3.4. > 2.tar.gz > > > > >> --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz > > > > >> --with-valgrind-include=/home/haozhang/include/valgrind/ > > > > >> > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in > > > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/vec/vec/interface/ > rvector.c > > > > >> > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in > > > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/impls/ > bcgs/bcgs.c > > > > >> > [0]PETSC ERROR: #3 KSPSolve() line 460 in > > > /home/haozhang/tmp/petsc-3.5.4 > > > > >> _HYPRE/src/ksp/ksp/interface/itfunc.c > > > > >> > [vn18:99838] *** Process received signal *** > > > > >> > [vn18:99838] Signal: Aborted (6) > > > > >> > [vn18:99838] Signal code: (-6) > > > > >> > [vn18:99848] *** Process received signal *** > > > > >> > [vn18:99849] *** Process received signal *** > > > > >> > [vn18:99849] Signal: Aborted (6) > > > > >> > [vn18:99849] Signal code: (-6) > > > > >> > [vn18:99848] Signal: Aborted (6) > > > > >> > [vn18:99848] Signal code: (-6) > > > > >> > [vn18:99838] [ 0] /lib64/libpthread.so.0(+ > 0xf790)[0x2aaaadbb4790] > > > > >> > [vn18:99838] [ 1] [vn18:99848] [ 0] > /lib64/libpthread.so.0(+0xf790 > > > > >> )[0x2aaaadbb4790] > > > > >> > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35) > [0x2aaaaddf5625] > > > > >> > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > > > >> > [vn18:99838] [ 3] [vn18:99849] [ 0] > /lib64/libpthread.so.0(+0xf790 > > > > >> )[0x2aaaadbb4790] > > > > >> > [vn18:99849] [ 1] [vn15:122106] *** Process received signal *** > > > > >> > [vn15:122106] Signal: Aborted (6) > > > > >> > [vn15:122106] Signal code: (-6) > > > > >> > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > > > >> > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > > > >> > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4 > > > > >> _HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscTraceBac > > > > >> kErrorHandler+0x503)[0x2aaaaae1d7e1] > > > > >> > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35) > [0x2aaaaddf5625] > > > > >> > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[0x2aaaaddf6e05] > > > > >> > [vn18:99849] [ 3] [vn15:122107] *** Process received signal *** > > > > >> > [vn15:122107] Signal: Aborted (6) > > > > >> > [vn15:122107] Signal code: (-6) > > > > >> > [vn16:86295] *** Process received signal *** > > > > >> > [vn16:86295] Signal: Aborted (6) > > > > >> > [vn16:86295] Signal code: (-6) > > > > >> > [vn18:99824] *** Process received signal *** > > > > >> > [vn18:99824] Signal: Aborted (6) > > > > >> > [vn18:99824] Signal code: (-6) > > > > >> > [vn18:99844] *** Process received signal *** > > > > >> > [vn18:99844] Signal: Aborted (6) > > > > >> > [vn18:99844] Signal code: (-6) > > > > >> > /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/lib > > > > >> /libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] > > > > >> > > > > > >> > > > > > >> > > > > > >> > -- > > > > >> > Hao Zhang > > > > >> > Dept. of Applid Mathematics and Statistics, > > > > >> > Stony Brook University, > > > > >> > Stony Brook, New York, 11790 > > > > >> > > > > >> > > > > > > > > > > > > > > > -- > > > > > Hao Zhang > > > > > Dept. of Applid Mathematics and Statistics, > > > > > Stony Brook University, > > > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbcbh1999 at gmail.com Fri Sep 29 22:02:02 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Fri, 29 Sep 2017 23:02:02 -0400 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: it seems that running petsc with download-mpich is a dead end for me. I've re-compiled PETSc with mpich options but I can't reinstall TORQUE for all users just for my own test. TORQUE needs root access. Do you have more suggestion on the issue? just a quick refresh. my PETSc complained floating point exception. for coarse and medium mesh grid without fp_trap option, my program runs. with fp_trap, program crashed right away. Thanks again! On Fri, Sep 29, 2017 at 9:33 PM, Hao Zhang wrote: > Thanks. I will recompile and submit a new debug log > > On Fri, Sep 29, 2017 at 9:31 PM, Satish Balay wrote: > >> Most of this is from openmpi. Its best to use --download-mpich >> for valgrind clean mpi as mentioned below. >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> >> [attempting to remove openmpi messages] - all messages below appear to >> be from your appliation code. And I don't see any messages from PETSc >> code in this log. >> >> Once there is memory corruption we get random behavior from >> application. So its best to fix all these issues and make this >> application valgrind clean. >> >> Satish >> >> On Sat, 30 Sep 2017, Hao Zhang wrote: >> >> >> >> > ==31497== Invalid write of size 8 >> > ==31497== at 0x643685: level_wave_func_Meniscus (imkcurve.c:1826) >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > ==31497== Address 0x7fe859c38 is on thread 1's stack >> > ==31497== >> > ==31497== Invalid write of size 8 >> > ==31497== at 0x64368C: level_wave_func_Meniscus (imkcurve.c:1826) >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > ==31497== Address 0x7fe859c30 is on thread 1's stack >> > ==31497== >> > ==31497== Invalid read of size 8 >> > ==31497== at 0x643693: level_wave_func_Meniscus (imkcurve.c:1831) >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > ==31497== Address 0x7fe859c38 is on thread 1's stack >> > ==31497== >> > ==31497== Invalid write of size 8 >> > ==31497== at 0x64372D: level_wave_func_Meniscus (imkcurve.c:1848) >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > ==31497== Address 0x7fe859c18 is on thread 1's stack >> > ==31497== >> > ==31497== Invalid read of size 8 >> > ==31497== at 0x7ED3127: tan (in /lib64/libm-2.12.so) >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > ==31497== Address 0x7fe859c18 is on thread 1's stack >> > ==31497== >> > ==31497== Invalid read of size 8 >> > ==31497== at 0x643755: level_wave_func_Meniscus (imkcurve.c:1853) >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > ==31497== Address 0x7fe859c30 is on thread 1's stack >> > ==31497== >> > ==31497== Invalid read of size 8 >> > ==31497== at 0x643765: level_wave_func_Meniscus (imkcurve.c:1854) >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > ==31497== Address 0x7fe859c30 is on thread 1's stack >> > ==31497== >> > ==31497== Invalid read of size 8 >> > ==31497== at 0x643A7F: level_wave_func_Meniscus (imkcurve.c:1944) >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > ==31497== Address 0x7fe859c30 is on thread 1's stack >> > ==31497== >> > ==31497== Invalid write of size 8 >> > ==31497== at 0x643A92: level_wave_func_Meniscus (imkcurve.c:1944) >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > ==31497== Address 0x7feffecf0 is on thread 1's stack >> > ==31497== >> > ==31497== Invalid read of size 8 >> > ==31497== at 0x643AEB: level_wave_func_Meniscus (imkcurve.c:1955) >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > ==31497== Address 0x7feffecf0 is on thread 1's stack >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x428B9F: >> > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() (iFbasic.cpp:2799) >> > ==31497== by 0x4B8BB9: >> > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_ >> RSSY_vd(_LEVEL_FUNC_PACK*) >> > (iFcartsn3d.cpp:15253) >> > ==31497== by 0x40C485: main (iFluid.cpp:349) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x428BA9: >> > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() (iFbasic.cpp:2799) >> > ==31497== by 0x4B8BB9: >> > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_ >> RSSY_vd(_LEVEL_FUNC_PACK*) >> > (iFcartsn3d.cpp:15253) >> > ==31497== by 0x40C485: main (iFluid.cpp:349) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x85B00E3: vfprintf (in /lib64/libc-2.12.so) >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > ==31497== by 0x486108: >> > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothPro >> perty_velocity_MAC_coupled_vd() >> > (iFcartsn3d.cpp:6101) >> > ==31497== by 0x4CBF0D: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18574) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x85B0101: vfprintf (in /lib64/libc-2.12.so) >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > ==31497== by 0x486108: >> > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothPro >> perty_velocity_MAC_coupled_vd() >> > (iFcartsn3d.cpp:6101) >> > ==31497== by 0x4CBF0D: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18574) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x4ED737: >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > (iFcartsn3d.cpp:24519) >> > ==31497== by 0x4CBFBC: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18594) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x4ED754: >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > (iFcartsn3d.cpp:24519) >> > ==31497== by 0x4CBFBC: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18594) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x85B4CF0: __printf_fp (in /lib64/libc-2.12.so) >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > ==31497== by 0x4ED9F7: >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > (iFcartsn3d.cpp:24549) >> > ==31497== by 0x4CBFBC: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18594) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x85B4F91: __printf_fp (in /lib64/libc-2.12.so) >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > ==31497== by 0x4ED9F7: >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > (iFcartsn3d.cpp:24549) >> > ==31497== by 0x4CBFBC: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18594) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x85AE396: __mpn_extract_double (in /lib64/libc-2.12.so >> ) >> > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > ==31497== by 0x4ED9F7: >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > (iFcartsn3d.cpp:24549) >> > ==31497== by 0x4CBFBC: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18594) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x85AE39B: __mpn_extract_double (in /lib64/libc-2.12.so >> ) >> > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > ==31497== by 0x4ED9F7: >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > (iFcartsn3d.cpp:24549) >> > ==31497== by 0x4CBFBC: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18594) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x85B59EC: __printf_fp (in /lib64/libc-2.12.so) >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > ==31497== by 0x4ED9F7: >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > (iFcartsn3d.cpp:24549) >> > ==31497== by 0x4CBFBC: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18594) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x85B5A9D: __printf_fp (in /lib64/libc-2.12.so) >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > ==31497== by 0x4ED9F7: >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > (iFcartsn3d.cpp:24549) >> > ==31497== by 0x4CBFBC: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18594) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x85B5C13: __printf_fp (in /lib64/libc-2.12.so) >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > ==31497== by 0x4ED9F7: >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > (iFcartsn3d.cpp:24549) >> > ==31497== by 0x4CBFBC: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18594) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x85B5CA4: __printf_fp (in /lib64/libc-2.12.so) >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > ==31497== by 0x4ED9F7: >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > (iFcartsn3d.cpp:24549) >> > ==31497== by 0x4CBFBC: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18594) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x85B625E: __printf_fp (in /lib64/libc-2.12.so) >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > ==31497== by 0x4ED9F7: >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > (iFcartsn3d.cpp:24549) >> > ==31497== by 0x4CBFBC: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18594) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Conditional jump or move depends on uninitialised value(s) >> > ==31497== at 0x85B6242: __printf_fp (in /lib64/libc-2.12.so) >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > ==31497== by 0x4ED9F7: >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > (iFcartsn3d.cpp:24549) >> > ==31497== by 0x4CBFBC: >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > (iFcartsn3d.cpp:18594) >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== >> > ==31497== Syscall param write(buf) points to uninitialised byte(s) >> > ==31497== at 0x86465ED: ??? (in /lib64/libc-2.12.so) >> > ==31497== by 0x85DCAD2: _IO_file_write@@GLIBC_2.2.5 (in /lib64/ >> > libc-2.12.so) >> > ==31497== by 0x85DE084: _IO_do_write@@GLIBC_2.2.5 (in /lib64/ >> libc-2.12.so >> > ) >> > ==31497== by 0x85DD287: _IO_file_sync@@GLIBC_2.2.5 (in /lib64/ >> > libc-2.12.so) >> > ==31497== by 0x85D1999: fflush (in /lib64/libc-2.12.so) >> > ==31497== by 0x6FACFA: preserve_front_advance_front3d (fadv.c:2255) >> > ==31497== by 0x6FA14E: mixed_advance_front3d (fadv.c:1895) >> > ==31497== by 0x6F9B2D: advance_front3d_tracking_control >> (fadv.c:1703) >> > ==31497== by 0x54E119: FrontAdvance (fmap.c:142) >> > ==31497== by 0x54DF27: FT_Propagate (fmap.c:80) >> > ==31497== by 0x40CA37: ifluid_driver(_Front*, >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:512) >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > ==31497== Address 0x412308a is not stack'd, malloc'd or (recently) >> free'd >> > ==31497== >> > >> > On Fri, Sep 29, 2017 at 8:45 PM, Satish Balay >> wrote: >> > >> > > Can you run this code with valgrind and send the log? >> > > >> > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > > >> > > Satish >> > > >> > > On Sat, 30 Sep 2017, Hao Zhang wrote: >> > > >> > > > I've modified my source code accordingly using PETSc 3.8.0. It >> seems no >> > > > more useful info got printed out. >> > > > >> > > > On Fri, Sep 29, 2017 at 8:15 PM, Hao Zhang >> wrote: >> > > > >> > > > > I'm using -fp_trap as I run the simulation. the output only gives >> > > > > >> > > > > floating point exception >> > > > > >> > > > > It doesn't matter coarse grid or medium grid or more refined >> grid. the >> > > > > program crashed right away. >> > > > > >> > > > > On Fri, Sep 29, 2017 at 12:19 AM, Barry Smith > > >> > > wrote: >> > > > > >> > > > >> >> > > > >> Please upgrade to the latest 3.8 version of PETSc. and then >> run with >> > > > >> -fp_trap >> > > > >> >> > > > >> This message is usually an indication that a NaN or Inf got >> into >> > > the >> > > > >> numerical computation. Using the latest PETSc will make it much >> > > easier to >> > > > >> track down. >> > > > >> >> > > > >> Barry >> > > > >> >> > > > >> > On Sep 28, 2017, at 8:57 PM, Hao Zhang >> wrote: >> > > > >> > >> > > > >> > I'm experiencing error when doing mesh refinement(mesh x2 for >> X, Y >> > > and >> > > > >> Z direction) for 3D poisson problem. >> > > > >> > PETSc produces no errors when no mesh refinement. I've pasted >> some >> > > > >> debugging info here. Please advise. >> > > > >> > >> > > > >> > >> > > > >> > [0]PETSC ERROR: --------------------- Error Message >> > > > >> -------------------------------------------------------------- >> > > > >> > [0]PETSC ERROR: Invalid argument >> > > > >> > [0]PETSC ERROR: Scalar value must be same on all processes, >> > > argument # 3 >> > > > >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ >> > > documentation/faq.html >> > > > >> for trouble shooting. >> > > > >> > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 >> > > > >> > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid on a >> > > > >> arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 23:47:39 >> 2017 >> > > > >> > [0]PETSC ERROR: Configure options --with-mpi-dir=/cm/shared/ >> > > apps/openmpi/gcc/64/1.8.5/ >> > > > >> --download-fblaslapack=/home/haozhang/tmp/fblaslapack-3.4.2. >> tar.gz >> > > > >> --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz >> > > > >> --with-valgrind-include=/home/haozhang/include/valgrind/ >> > > > >> > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in >> > > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/vec/vec/interface/r >> vector.c >> > > > >> > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in >> > > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/impls/bcgs/ >> bcgs.c >> > > > >> > [0]PETSC ERROR: #3 KSPSolve() line 460 in >> > > /home/haozhang/tmp/petsc-3.5.4 >> > > > >> _HYPRE/src/ksp/ksp/interface/itfunc.c >> > > > >> > [vn18:99838] *** Process received signal *** >> > > > >> > [vn18:99838] Signal: Aborted (6) >> > > > >> > [vn18:99838] Signal code: (-6) >> > > > >> > [vn18:99848] *** Process received signal *** >> > > > >> > [vn18:99849] *** Process received signal *** >> > > > >> > [vn18:99849] Signal: Aborted (6) >> > > > >> > [vn18:99849] Signal code: (-6) >> > > > >> > [vn18:99848] Signal: Aborted (6) >> > > > >> > [vn18:99848] Signal code: (-6) >> > > > >> > [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790 >> )[0x2aaaadbb4790] >> > > > >> > [vn18:99838] [ 1] [vn18:99848] [ 0] >> /lib64/libpthread.so.0(+0xf790 >> > > > >> )[0x2aaaadbb4790] >> > > > >> > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35) >> [0x2aaaaddf5625] >> > > > >> > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[ >> 0x2aaaaddf6e05] >> > > > >> > [vn18:99838] [ 3] [vn18:99849] [ 0] >> /lib64/libpthread.so.0(+0xf790 >> > > > >> )[0x2aaaadbb4790] >> > > > >> > [vn18:99849] [ 1] [vn15:122106] *** Process received signal *** >> > > > >> > [vn15:122106] Signal: Aborted (6) >> > > > >> > [vn15:122106] Signal code: (-6) >> > > > >> > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] >> > > > >> > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[ >> 0x2aaaaddf6e05] >> > > > >> > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4 >> > > > >> _HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscTraceBac >> > > > >> kErrorHandler+0x503)[0x2aaaaae1d7e1] >> > > > >> > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35) >> [0x2aaaaddf5625] >> > > > >> > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[ >> 0x2aaaaddf6e05] >> > > > >> > [vn18:99849] [ 3] [vn15:122107] *** Process received signal *** >> > > > >> > [vn15:122107] Signal: Aborted (6) >> > > > >> > [vn15:122107] Signal code: (-6) >> > > > >> > [vn16:86295] *** Process received signal *** >> > > > >> > [vn16:86295] Signal: Aborted (6) >> > > > >> > [vn16:86295] Signal code: (-6) >> > > > >> > [vn18:99824] *** Process received signal *** >> > > > >> > [vn18:99824] Signal: Aborted (6) >> > > > >> > [vn18:99824] Signal code: (-6) >> > > > >> > [vn18:99844] *** Process received signal *** >> > > > >> > [vn18:99844] Signal: Aborted (6) >> > > > >> > [vn18:99844] Signal code: (-6) >> > > > >> > /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/lib >> > > > >> /libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > -- >> > > > >> > Hao Zhang >> > > > >> > Dept. of Applid Mathematics and Statistics, >> > > > >> > Stony Brook University, >> > > > >> > Stony Brook, New York, 11790 >> > > > >> >> > > > >> >> > > > > >> > > > > >> > > > > -- >> > > > > Hao Zhang >> > > > > Dept. of Applid Mathematics and Statistics, >> > > > > Stony Brook University, >> > > > > Stony Brook, New York, 11790 >> > > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > >> > >> > >> > >> >> > > > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Sep 29 22:14:32 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 29 Sep 2017 22:14:32 -0500 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: On Sat, 30 Sep 2017, Hao Zhang wrote: > it seems that running petsc with download-mpich is a dead end for me. I've > re-compiled PETSc with mpich options but I can't reinstall TORQUE for all > users just for my own test. TORQUE needs root access. > Do you have more suggestion on the issue? As mentioned - the previous log pointed to errors in your code. You should look at them - and try to fix your code. > just a quick refresh. my PETSc > complained floating point exception. for coarse and medium mesh grid > without fp_trap option, my program runs. with fp_trap, program crashed > right away. After memory corruptions are eliminated - if the errors persist - you might have to use the debugger to determine why NaNs are getting generated. Satish > > Thanks again! > > On Fri, Sep 29, 2017 at 9:33 PM, Hao Zhang wrote: > > > Thanks. I will recompile and submit a new debug log > > > > On Fri, Sep 29, 2017 at 9:31 PM, Satish Balay wrote: > > > >> Most of this is from openmpi. Its best to use --download-mpich > >> for valgrind clean mpi as mentioned below. > >> > >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > >> > >> [attempting to remove openmpi messages] - all messages below appear to > >> be from your appliation code. And I don't see any messages from PETSc > >> code in this log. > >> > >> Once there is memory corruption we get random behavior from > >> application. So its best to fix all these issues and make this > >> application valgrind clean. > >> > >> Satish > >> > >> On Sat, 30 Sep 2017, Hao Zhang wrote: > >> > >> > >> > >> > ==31497== Invalid write of size 8 > >> > ==31497== at 0x643685: level_wave_func_Meniscus (imkcurve.c:1826) > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > ==31497== Address 0x7fe859c38 is on thread 1's stack > >> > ==31497== > >> > ==31497== Invalid write of size 8 > >> > ==31497== at 0x64368C: level_wave_func_Meniscus (imkcurve.c:1826) > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack > >> > ==31497== > >> > ==31497== Invalid read of size 8 > >> > ==31497== at 0x643693: level_wave_func_Meniscus (imkcurve.c:1831) > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > ==31497== Address 0x7fe859c38 is on thread 1's stack > >> > ==31497== > >> > ==31497== Invalid write of size 8 > >> > ==31497== at 0x64372D: level_wave_func_Meniscus (imkcurve.c:1848) > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > ==31497== Address 0x7fe859c18 is on thread 1's stack > >> > ==31497== > >> > ==31497== Invalid read of size 8 > >> > ==31497== at 0x7ED3127: tan (in /lib64/libm-2.12.so) > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > ==31497== Address 0x7fe859c18 is on thread 1's stack > >> > ==31497== > >> > ==31497== Invalid read of size 8 > >> > ==31497== at 0x643755: level_wave_func_Meniscus (imkcurve.c:1853) > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack > >> > ==31497== > >> > ==31497== Invalid read of size 8 > >> > ==31497== at 0x643765: level_wave_func_Meniscus (imkcurve.c:1854) > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack > >> > ==31497== > >> > ==31497== Invalid read of size 8 > >> > ==31497== at 0x643A7F: level_wave_func_Meniscus (imkcurve.c:1944) > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack > >> > ==31497== > >> > ==31497== Invalid write of size 8 > >> > ==31497== at 0x643A92: level_wave_func_Meniscus (imkcurve.c:1944) > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > ==31497== Address 0x7feffecf0 is on thread 1's stack > >> > ==31497== > >> > ==31497== Invalid read of size 8 > >> > ==31497== at 0x643AEB: level_wave_func_Meniscus (imkcurve.c:1955) > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > ==31497== Address 0x7feffecf0 is on thread 1's stack > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x428B9F: > >> > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() (iFbasic.cpp:2799) > >> > ==31497== by 0x4B8BB9: > >> > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_ > >> RSSY_vd(_LEVEL_FUNC_PACK*) > >> > (iFcartsn3d.cpp:15253) > >> > ==31497== by 0x40C485: main (iFluid.cpp:349) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x428BA9: > >> > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() (iFbasic.cpp:2799) > >> > ==31497== by 0x4B8BB9: > >> > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_ > >> RSSY_vd(_LEVEL_FUNC_PACK*) > >> > (iFcartsn3d.cpp:15253) > >> > ==31497== by 0x40C485: main (iFluid.cpp:349) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x85B00E3: vfprintf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x486108: > >> > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothPro > >> perty_velocity_MAC_coupled_vd() > >> > (iFcartsn3d.cpp:6101) > >> > ==31497== by 0x4CBF0D: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18574) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x85B0101: vfprintf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x486108: > >> > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothPro > >> perty_velocity_MAC_coupled_vd() > >> > (iFcartsn3d.cpp:6101) > >> > ==31497== by 0x4CBF0D: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18574) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x4ED737: > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > (iFcartsn3d.cpp:24519) > >> > ==31497== by 0x4CBFBC: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18594) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x4ED754: > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > (iFcartsn3d.cpp:24519) > >> > ==31497== by 0x4CBFBC: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18594) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x85B4CF0: __printf_fp (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x4ED9F7: > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > (iFcartsn3d.cpp:24549) > >> > ==31497== by 0x4CBFBC: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18594) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x85B4F91: __printf_fp (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x4ED9F7: > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > (iFcartsn3d.cpp:24549) > >> > ==31497== by 0x4CBFBC: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18594) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x85AE396: __mpn_extract_double (in /lib64/libc-2.12.so > >> ) > >> > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x4ED9F7: > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > (iFcartsn3d.cpp:24549) > >> > ==31497== by 0x4CBFBC: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18594) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x85AE39B: __mpn_extract_double (in /lib64/libc-2.12.so > >> ) > >> > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x4ED9F7: > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > (iFcartsn3d.cpp:24549) > >> > ==31497== by 0x4CBFBC: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18594) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x85B59EC: __printf_fp (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x4ED9F7: > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > (iFcartsn3d.cpp:24549) > >> > ==31497== by 0x4CBFBC: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18594) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x85B5A9D: __printf_fp (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x4ED9F7: > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > (iFcartsn3d.cpp:24549) > >> > ==31497== by 0x4CBFBC: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18594) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x85B5C13: __printf_fp (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x4ED9F7: > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > (iFcartsn3d.cpp:24549) > >> > ==31497== by 0x4CBFBC: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18594) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x85B5CA4: __printf_fp (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x4ED9F7: > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > (iFcartsn3d.cpp:24549) > >> > ==31497== by 0x4CBFBC: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18594) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x85B625E: __printf_fp (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x4ED9F7: > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > (iFcartsn3d.cpp:24549) > >> > ==31497== by 0x4CBFBC: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18594) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > >> > ==31497== at 0x85B6242: __printf_fp (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > ==31497== by 0x4ED9F7: > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > (iFcartsn3d.cpp:24549) > >> > ==31497== by 0x4CBFBC: > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > (iFcartsn3d.cpp:18594) > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== > >> > ==31497== Syscall param write(buf) points to uninitialised byte(s) > >> > ==31497== at 0x86465ED: ??? (in /lib64/libc-2.12.so) > >> > ==31497== by 0x85DCAD2: _IO_file_write@@GLIBC_2.2.5 (in /lib64/ > >> > libc-2.12.so) > >> > ==31497== by 0x85DE084: _IO_do_write@@GLIBC_2.2.5 (in /lib64/ > >> libc-2.12.so > >> > ) > >> > ==31497== by 0x85DD287: _IO_file_sync@@GLIBC_2.2.5 (in /lib64/ > >> > libc-2.12.so) > >> > ==31497== by 0x85D1999: fflush (in /lib64/libc-2.12.so) > >> > ==31497== by 0x6FACFA: preserve_front_advance_front3d (fadv.c:2255) > >> > ==31497== by 0x6FA14E: mixed_advance_front3d (fadv.c:1895) > >> > ==31497== by 0x6F9B2D: advance_front3d_tracking_control > >> (fadv.c:1703) > >> > ==31497== by 0x54E119: FrontAdvance (fmap.c:142) > >> > ==31497== by 0x54DF27: FT_Propagate (fmap.c:80) > >> > ==31497== by 0x40CA37: ifluid_driver(_Front*, > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:512) > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > ==31497== Address 0x412308a is not stack'd, malloc'd or (recently) > >> free'd > >> > ==31497== > >> > > >> > On Fri, Sep 29, 2017 at 8:45 PM, Satish Balay > >> wrote: > >> > > >> > > Can you run this code with valgrind and send the log? > >> > > > >> > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > >> > > > >> > > Satish > >> > > > >> > > On Sat, 30 Sep 2017, Hao Zhang wrote: > >> > > > >> > > > I've modified my source code accordingly using PETSc 3.8.0. It > >> seems no > >> > > > more useful info got printed out. > >> > > > > >> > > > On Fri, Sep 29, 2017 at 8:15 PM, Hao Zhang > >> wrote: > >> > > > > >> > > > > I'm using -fp_trap as I run the simulation. the output only gives > >> > > > > > >> > > > > floating point exception > >> > > > > > >> > > > > It doesn't matter coarse grid or medium grid or more refined > >> grid. the > >> > > > > program crashed right away. > >> > > > > > >> > > > > On Fri, Sep 29, 2017 at 12:19 AM, Barry Smith >> > > >> > > wrote: > >> > > > > > >> > > > >> > >> > > > >> Please upgrade to the latest 3.8 version of PETSc. and then > >> run with > >> > > > >> -fp_trap > >> > > > >> > >> > > > >> This message is usually an indication that a NaN or Inf got > >> into > >> > > the > >> > > > >> numerical computation. Using the latest PETSc will make it much > >> > > easier to > >> > > > >> track down. > >> > > > >> > >> > > > >> Barry > >> > > > >> > >> > > > >> > On Sep 28, 2017, at 8:57 PM, Hao Zhang > >> wrote: > >> > > > >> > > >> > > > >> > I'm experiencing error when doing mesh refinement(mesh x2 for > >> X, Y > >> > > and > >> > > > >> Z direction) for 3D poisson problem. > >> > > > >> > PETSc produces no errors when no mesh refinement. I've pasted > >> some > >> > > > >> debugging info here. Please advise. > >> > > > >> > > >> > > > >> > > >> > > > >> > [0]PETSC ERROR: --------------------- Error Message > >> > > > >> -------------------------------------------------------------- > >> > > > >> > [0]PETSC ERROR: Invalid argument > >> > > > >> > [0]PETSC ERROR: Scalar value must be same on all processes, > >> > > argument # 3 > >> > > > >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ > >> > > documentation/faq.html > >> > > > >> for trouble shooting. > >> > > > >> > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 > >> > > > >> > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid on a > >> > > > >> arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 23:47:39 > >> 2017 > >> > > > >> > [0]PETSC ERROR: Configure options --with-mpi-dir=/cm/shared/ > >> > > apps/openmpi/gcc/64/1.8.5/ > >> > > > >> --download-fblaslapack=/home/haozhang/tmp/fblaslapack-3.4.2. > >> tar.gz > >> > > > >> --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz > >> > > > >> --with-valgrind-include=/home/haozhang/include/valgrind/ > >> > > > >> > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in > >> > > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/vec/vec/interface/r > >> vector.c > >> > > > >> > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in > >> > > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/impls/bcgs/ > >> bcgs.c > >> > > > >> > [0]PETSC ERROR: #3 KSPSolve() line 460 in > >> > > /home/haozhang/tmp/petsc-3.5.4 > >> > > > >> _HYPRE/src/ksp/ksp/interface/itfunc.c > >> > > > >> > [vn18:99838] *** Process received signal *** > >> > > > >> > [vn18:99838] Signal: Aborted (6) > >> > > > >> > [vn18:99838] Signal code: (-6) > >> > > > >> > [vn18:99848] *** Process received signal *** > >> > > > >> > [vn18:99849] *** Process received signal *** > >> > > > >> > [vn18:99849] Signal: Aborted (6) > >> > > > >> > [vn18:99849] Signal code: (-6) > >> > > > >> > [vn18:99848] Signal: Aborted (6) > >> > > > >> > [vn18:99848] Signal code: (-6) > >> > > > >> > [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790 > >> )[0x2aaaadbb4790] > >> > > > >> > [vn18:99838] [ 1] [vn18:99848] [ 0] > >> /lib64/libpthread.so.0(+0xf790 > >> > > > >> )[0x2aaaadbb4790] > >> > > > >> > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35) > >> [0x2aaaaddf5625] > >> > > > >> > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[ > >> 0x2aaaaddf6e05] > >> > > > >> > [vn18:99838] [ 3] [vn18:99849] [ 0] > >> /lib64/libpthread.so.0(+0xf790 > >> > > > >> )[0x2aaaadbb4790] > >> > > > >> > [vn18:99849] [ 1] [vn15:122106] *** Process received signal *** > >> > > > >> > [vn15:122106] Signal: Aborted (6) > >> > > > >> > [vn15:122106] Signal code: (-6) > >> > > > >> > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > >> > > > >> > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[ > >> 0x2aaaaddf6e05] > >> > > > >> > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4 > >> > > > >> _HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscTraceBac > >> > > > >> kErrorHandler+0x503)[0x2aaaaae1d7e1] > >> > > > >> > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35) > >> [0x2aaaaddf5625] > >> > > > >> > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[ > >> 0x2aaaaddf6e05] > >> > > > >> > [vn18:99849] [ 3] [vn15:122107] *** Process received signal *** > >> > > > >> > [vn15:122107] Signal: Aborted (6) > >> > > > >> > [vn15:122107] Signal code: (-6) > >> > > > >> > [vn16:86295] *** Process received signal *** > >> > > > >> > [vn16:86295] Signal: Aborted (6) > >> > > > >> > [vn16:86295] Signal code: (-6) > >> > > > >> > [vn18:99824] *** Process received signal *** > >> > > > >> > [vn18:99824] Signal: Aborted (6) > >> > > > >> > [vn18:99824] Signal code: (-6) > >> > > > >> > [vn18:99844] *** Process received signal *** > >> > > > >> > [vn18:99844] Signal: Aborted (6) > >> > > > >> > [vn18:99844] Signal code: (-6) > >> > > > >> > /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/lib > >> > > > >> /libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] > >> > > > >> > > >> > > > >> > > >> > > > >> > > >> > > > >> > -- > >> > > > >> > Hao Zhang > >> > > > >> > Dept. of Applid Mathematics and Statistics, > >> > > > >> > Stony Brook University, > >> > > > >> > Stony Brook, New York, 11790 > >> > > > >> > >> > > > >> > >> > > > > > >> > > > > > >> > > > > -- > >> > > > > Hao Zhang > >> > > > > Dept. of Applid Mathematics and Statistics, > >> > > > > Stony Brook University, > >> > > > > Stony Brook, New York, 11790 > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > >> > > >> > > >> > > >> > >> > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > > > > > From hbcbh1999 at gmail.com Fri Sep 29 22:16:19 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Fri, 29 Sep 2017 23:16:19 -0400 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: Thanks! On Fri, Sep 29, 2017 at 11:14 PM, Satish Balay wrote: > On Sat, 30 Sep 2017, Hao Zhang wrote: > > > it seems that running petsc with download-mpich is a dead end for me. > I've > > re-compiled PETSc with mpich options but I can't reinstall TORQUE for all > > users just for my own test. TORQUE needs root access. > > Do you have more suggestion on the issue? > > As mentioned - the previous log pointed to errors in your code. You should > look > at them - and try to fix your code. > > > just a quick refresh. my PETSc > > complained floating point exception. for coarse and medium mesh grid > > without fp_trap option, my program runs. with fp_trap, program crashed > > right away. > > After memory corruptions are eliminated - if the errors persist - you > might have to use the debugger to determine why NaNs are getting > generated. > > Satish > > > > > Thanks again! > > > > On Fri, Sep 29, 2017 at 9:33 PM, Hao Zhang wrote: > > > > > Thanks. I will recompile and submit a new debug log > > > > > > On Fri, Sep 29, 2017 at 9:31 PM, Satish Balay > wrote: > > > > > >> Most of this is from openmpi. Its best to use --download-mpich > > >> for valgrind clean mpi as mentioned below. > > >> > > >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > >> > > >> [attempting to remove openmpi messages] - all messages below appear to > > >> be from your appliation code. And I don't see any messages from PETSc > > >> code in this log. > > >> > > >> Once there is memory corruption we get random behavior from > > >> application. So its best to fix all these issues and make this > > >> application valgrind clean. > > >> > > >> Satish > > >> > > >> On Sat, 30 Sep 2017, Hao Zhang wrote: > > >> > > >> > > >> > > >> > ==31497== Invalid write of size 8 > > >> > ==31497== at 0x643685: level_wave_func_Meniscus (imkcurve.c:1826) > > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > > >> > ==31497== Address 0x7fe859c38 is on thread 1's stack > > >> > ==31497== > > >> > ==31497== Invalid write of size 8 > > >> > ==31497== at 0x64368C: level_wave_func_Meniscus (imkcurve.c:1826) > > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack > > >> > ==31497== > > >> > ==31497== Invalid read of size 8 > > >> > ==31497== at 0x643693: level_wave_func_Meniscus (imkcurve.c:1831) > > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > > >> > ==31497== Address 0x7fe859c38 is on thread 1's stack > > >> > ==31497== > > >> > ==31497== Invalid write of size 8 > > >> > ==31497== at 0x64372D: level_wave_func_Meniscus (imkcurve.c:1848) > > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > > >> > ==31497== Address 0x7fe859c18 is on thread 1's stack > > >> > ==31497== > > >> > ==31497== Invalid read of size 8 > > >> > ==31497== at 0x7ED3127: tan (in /lib64/libm-2.12.so) > > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > > >> > ==31497== Address 0x7fe859c18 is on thread 1's stack > > >> > ==31497== > > >> > ==31497== Invalid read of size 8 > > >> > ==31497== at 0x643755: level_wave_func_Meniscus (imkcurve.c:1853) > > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack > > >> > ==31497== > > >> > ==31497== Invalid read of size 8 > > >> > ==31497== at 0x643765: level_wave_func_Meniscus (imkcurve.c:1854) > > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack > > >> > ==31497== > > >> > ==31497== Invalid read of size 8 > > >> > ==31497== at 0x643A7F: level_wave_func_Meniscus (imkcurve.c:1944) > > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack > > >> > ==31497== > > >> > ==31497== Invalid write of size 8 > > >> > ==31497== at 0x643A92: level_wave_func_Meniscus (imkcurve.c:1944) > > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > > >> > ==31497== Address 0x7feffecf0 is on thread 1's stack > > >> > ==31497== > > >> > ==31497== Invalid read of size 8 > > >> > ==31497== at 0x643AEB: level_wave_func_Meniscus (imkcurve.c:1955) > > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > > >> > ==31497== Address 0x7feffecf0 is on thread 1's stack > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x428B9F: > > >> > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() > (iFbasic.cpp:2799) > > >> > ==31497== by 0x4B8BB9: > > >> > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_ > > >> RSSY_vd(_LEVEL_FUNC_PACK*) > > >> > (iFcartsn3d.cpp:15253) > > >> > ==31497== by 0x40C485: main (iFluid.cpp:349) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x428BA9: > > >> > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() > (iFbasic.cpp:2799) > > >> > ==31497== by 0x4B8BB9: > > >> > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_ > > >> RSSY_vd(_LEVEL_FUNC_PACK*) > > >> > (iFcartsn3d.cpp:15253) > > >> > ==31497== by 0x40C485: main (iFluid.cpp:349) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x85B00E3: vfprintf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x486108: > > >> > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothPro > > >> perty_velocity_MAC_coupled_vd() > > >> > (iFcartsn3d.cpp:6101) > > >> > ==31497== by 0x4CBF0D: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18574) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x85B0101: vfprintf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x486108: > > >> > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothPro > > >> perty_velocity_MAC_coupled_vd() > > >> > (iFcartsn3d.cpp:6101) > > >> > ==31497== by 0x4CBF0D: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18574) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x4ED737: > > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > >> > (iFcartsn3d.cpp:24519) > > >> > ==31497== by 0x4CBFBC: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18594) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x4ED754: > > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > >> > (iFcartsn3d.cpp:24519) > > >> > ==31497== by 0x4CBFBC: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18594) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x85B4CF0: __printf_fp (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x4ED9F7: > > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > >> > (iFcartsn3d.cpp:24549) > > >> > ==31497== by 0x4CBFBC: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18594) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x85B4F91: __printf_fp (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x4ED9F7: > > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > >> > (iFcartsn3d.cpp:24549) > > >> > ==31497== by 0x4CBFBC: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18594) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x85AE396: __mpn_extract_double (in /lib64/ > libc-2.12.so > > >> ) > > >> > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x4ED9F7: > > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > >> > (iFcartsn3d.cpp:24549) > > >> > ==31497== by 0x4CBFBC: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18594) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x85AE39B: __mpn_extract_double (in /lib64/ > libc-2.12.so > > >> ) > > >> > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x4ED9F7: > > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > >> > (iFcartsn3d.cpp:24549) > > >> > ==31497== by 0x4CBFBC: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18594) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x85B59EC: __printf_fp (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x4ED9F7: > > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > >> > (iFcartsn3d.cpp:24549) > > >> > ==31497== by 0x4CBFBC: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18594) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x85B5A9D: __printf_fp (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x4ED9F7: > > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > >> > (iFcartsn3d.cpp:24549) > > >> > ==31497== by 0x4CBFBC: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18594) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x85B5C13: __printf_fp (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x4ED9F7: > > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > >> > (iFcartsn3d.cpp:24549) > > >> > ==31497== by 0x4CBFBC: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18594) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x85B5CA4: __printf_fp (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x4ED9F7: > > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > >> > (iFcartsn3d.cpp:24549) > > >> > ==31497== by 0x4CBFBC: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18594) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x85B625E: __printf_fp (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x4ED9F7: > > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > >> > (iFcartsn3d.cpp:24549) > > >> > ==31497== by 0x4CBFBC: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18594) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Conditional jump or move depends on uninitialised value(s) > > >> > ==31497== at 0x85B6242: __printf_fp (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x4ED9F7: > > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > > >> > (iFcartsn3d.cpp:24549) > > >> > ==31497== by 0x4CBFBC: > > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > > >> > (iFcartsn3d.cpp:18594) > > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== > > >> > ==31497== Syscall param write(buf) points to uninitialised byte(s) > > >> > ==31497== at 0x86465ED: ??? (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x85DCAD2: _IO_file_write@@GLIBC_2.2.5 (in /lib64/ > > >> > libc-2.12.so) > > >> > ==31497== by 0x85DE084: _IO_do_write@@GLIBC_2.2.5 (in /lib64/ > > >> libc-2.12.so > > >> > ) > > >> > ==31497== by 0x85DD287: _IO_file_sync@@GLIBC_2.2.5 (in /lib64/ > > >> > libc-2.12.so) > > >> > ==31497== by 0x85D1999: fflush (in /lib64/libc-2.12.so) > > >> > ==31497== by 0x6FACFA: preserve_front_advance_front3d > (fadv.c:2255) > > >> > ==31497== by 0x6FA14E: mixed_advance_front3d (fadv.c:1895) > > >> > ==31497== by 0x6F9B2D: advance_front3d_tracking_control > > >> (fadv.c:1703) > > >> > ==31497== by 0x54E119: FrontAdvance (fmap.c:142) > > >> > ==31497== by 0x54DF27: FT_Propagate (fmap.c:80) > > >> > ==31497== by 0x40CA37: ifluid_driver(_Front*, > > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:512) > > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > > >> > ==31497== Address 0x412308a is not stack'd, malloc'd or (recently) > > >> free'd > > >> > ==31497== > > >> > > > >> > On Fri, Sep 29, 2017 at 8:45 PM, Satish Balay > > >> wrote: > > >> > > > >> > > Can you run this code with valgrind and send the log? > > >> > > > > >> > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > >> > > > > >> > > Satish > > >> > > > > >> > > On Sat, 30 Sep 2017, Hao Zhang wrote: > > >> > > > > >> > > > I've modified my source code accordingly using PETSc 3.8.0. It > > >> seems no > > >> > > > more useful info got printed out. > > >> > > > > > >> > > > On Fri, Sep 29, 2017 at 8:15 PM, Hao Zhang > > > >> wrote: > > >> > > > > > >> > > > > I'm using -fp_trap as I run the simulation. the output only > gives > > >> > > > > > > >> > > > > floating point exception > > >> > > > > > > >> > > > > It doesn't matter coarse grid or medium grid or more refined > > >> grid. the > > >> > > > > program crashed right away. > > >> > > > > > > >> > > > > On Fri, Sep 29, 2017 at 12:19 AM, Barry Smith < > bsmith at mcs.anl.gov > > >> > > > >> > > wrote: > > >> > > > > > > >> > > > >> > > >> > > > >> Please upgrade to the latest 3.8 version of PETSc. and then > > >> run with > > >> > > > >> -fp_trap > > >> > > > >> > > >> > > > >> This message is usually an indication that a NaN or Inf > got > > >> into > > >> > > the > > >> > > > >> numerical computation. Using the latest PETSc will make it > much > > >> > > easier to > > >> > > > >> track down. > > >> > > > >> > > >> > > > >> Barry > > >> > > > >> > > >> > > > >> > On Sep 28, 2017, at 8:57 PM, Hao Zhang < > hbcbh1999 at gmail.com> > > >> wrote: > > >> > > > >> > > > >> > > > >> > I'm experiencing error when doing mesh refinement(mesh x2 > for > > >> X, Y > > >> > > and > > >> > > > >> Z direction) for 3D poisson problem. > > >> > > > >> > PETSc produces no errors when no mesh refinement. I've > pasted > > >> some > > >> > > > >> debugging info here. Please advise. > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > [0]PETSC ERROR: --------------------- Error Message > > >> > > > >> ------------------------------------------------------------ > -- > > >> > > > >> > [0]PETSC ERROR: Invalid argument > > >> > > > >> > [0]PETSC ERROR: Scalar value must be same on all processes, > > >> > > argument # 3 > > >> > > > >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ > > >> > > documentation/faq.html > > >> > > > >> for trouble shooting. > > >> > > > >> > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 > > >> > > > >> > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid > on a > > >> > > > >> arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 > 23:47:39 > > >> 2017 > > >> > > > >> > [0]PETSC ERROR: Configure options > --with-mpi-dir=/cm/shared/ > > >> > > apps/openmpi/gcc/64/1.8.5/ > > >> > > > >> --download-fblaslapack=/home/haozhang/tmp/fblaslapack-3.4.2. > > >> tar.gz > > >> > > > >> --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz > > >> > > > >> --with-valgrind-include=/home/haozhang/include/valgrind/ > > >> > > > >> > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in > > >> > > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/vec/vec/interface/r > > >> vector.c > > >> > > > >> > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in > > >> > > > >> /home/haozhang/tmp/petsc-3.5.4_HYPRE/src/ksp/ksp/impls/bcgs/ > > >> bcgs.c > > >> > > > >> > [0]PETSC ERROR: #3 KSPSolve() line 460 in > > >> > > /home/haozhang/tmp/petsc-3.5.4 > > >> > > > >> _HYPRE/src/ksp/ksp/interface/itfunc.c > > >> > > > >> > [vn18:99838] *** Process received signal *** > > >> > > > >> > [vn18:99838] Signal: Aborted (6) > > >> > > > >> > [vn18:99838] Signal code: (-6) > > >> > > > >> > [vn18:99848] *** Process received signal *** > > >> > > > >> > [vn18:99849] *** Process received signal *** > > >> > > > >> > [vn18:99849] Signal: Aborted (6) > > >> > > > >> > [vn18:99849] Signal code: (-6) > > >> > > > >> > [vn18:99848] Signal: Aborted (6) > > >> > > > >> > [vn18:99848] Signal code: (-6) > > >> > > > >> > [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790 > > >> )[0x2aaaadbb4790] > > >> > > > >> > [vn18:99838] [ 1] [vn18:99848] [ 0] > > >> /lib64/libpthread.so.0(+0xf790 > > >> > > > >> )[0x2aaaadbb4790] > > >> > > > >> > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35) > > >> [0x2aaaaddf5625] > > >> > > > >> > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[ > > >> 0x2aaaaddf6e05] > > >> > > > >> > [vn18:99838] [ 3] [vn18:99849] [ 0] > > >> /lib64/libpthread.so.0(+0xf790 > > >> > > > >> )[0x2aaaadbb4790] > > >> > > > >> > [vn18:99849] [ 1] [vn15:122106] *** Process received > signal *** > > >> > > > >> > [vn15:122106] Signal: Aborted (6) > > >> > > > >> > [vn15:122106] Signal code: (-6) > > >> > > > >> > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > > >> > > > >> > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[ > > >> 0x2aaaaddf6e05] > > >> > > > >> > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4 > > >> > > > >> _HYPRE/arch-linux2-c-debug/lib/libpetsc.so.3.5(PetscTraceBac > > >> > > > >> kErrorHandler+0x503)[0x2aaaaae1d7e1] > > >> > > > >> > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35) > > >> [0x2aaaaddf5625] > > >> > > > >> > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[ > > >> 0x2aaaaddf6e05] > > >> > > > >> > [vn18:99849] [ 3] [vn15:122107] *** Process received > signal *** > > >> > > > >> > [vn15:122107] Signal: Aborted (6) > > >> > > > >> > [vn15:122107] Signal code: (-6) > > >> > > > >> > [vn16:86295] *** Process received signal *** > > >> > > > >> > [vn16:86295] Signal: Aborted (6) > > >> > > > >> > [vn16:86295] Signal code: (-6) > > >> > > > >> > [vn18:99824] *** Process received signal *** > > >> > > > >> > [vn18:99824] Signal: Aborted (6) > > >> > > > >> > [vn18:99824] Signal code: (-6) > > >> > > > >> > [vn18:99844] *** Process received signal *** > > >> > > > >> > [vn18:99844] Signal: Aborted (6) > > >> > > > >> > [vn18:99844] Signal code: (-6) > > >> > > > >> > /home/haozhang/tmp/petsc-3.5.4_HYPRE/arch-linux2-c-debug/ > lib > > >> > > > >> /libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > -- > > >> > > > >> > Hao Zhang > > >> > > > >> > Dept. of Applid Mathematics and Statistics, > > >> > > > >> > Stony Brook University, > > >> > > > >> > Stony Brook, New York, 11790 > > >> > > > >> > > >> > > > >> > > >> > > > > > > >> > > > > > > >> > > > > -- > > >> > > > > Hao Zhang > > >> > > > > Dept. of Applid Mathematics and Statistics, > > >> > > > > Stony Brook University, > > >> > > > > Stony Brook, New York, 11790 > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > >> > > > >> > > > >> > > > >> > > >> > > > > > > > > > -- > > > Hao Zhang > > > Dept. of Applid Mathematics and Statistics, > > > Stony Brook University, > > > Stony Brook, New York, 11790 > > > > > > > > > > > > > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbcbh1999 at gmail.com Fri Sep 29 22:51:44 2017 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Fri, 29 Sep 2017 23:51:44 -0400 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: is there a builtin way in PETSc I can flush cache in C++, specifically matrix entries so that I can look up? to check which entry is NaN. Thanks. On Fri, Sep 29, 2017 at 11:16 PM, Hao Zhang wrote: > Thanks! > > On Fri, Sep 29, 2017 at 11:14 PM, Satish Balay wrote: > >> On Sat, 30 Sep 2017, Hao Zhang wrote: >> >> > it seems that running petsc with download-mpich is a dead end for me. >> I've >> > re-compiled PETSc with mpich options but I can't reinstall TORQUE for >> all >> > users just for my own test. TORQUE needs root access. >> > Do you have more suggestion on the issue? >> >> As mentioned - the previous log pointed to errors in your code. You >> should look >> at them - and try to fix your code. >> >> > just a quick refresh. my PETSc >> > complained floating point exception. for coarse and medium mesh grid >> > without fp_trap option, my program runs. with fp_trap, program crashed >> > right away. >> >> After memory corruptions are eliminated - if the errors persist - you >> might have to use the debugger to determine why NaNs are getting >> generated. >> >> Satish >> >> > >> > Thanks again! >> > >> > On Fri, Sep 29, 2017 at 9:33 PM, Hao Zhang wrote: >> > >> > > Thanks. I will recompile and submit a new debug log >> > > >> > > On Fri, Sep 29, 2017 at 9:31 PM, Satish Balay >> wrote: >> > > >> > >> Most of this is from openmpi. Its best to use --download-mpich >> > >> for valgrind clean mpi as mentioned below. >> > >> >> > >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > >> >> > >> [attempting to remove openmpi messages] - all messages below appear >> to >> > >> be from your appliation code. And I don't see any messages from PETSc >> > >> code in this log. >> > >> >> > >> Once there is memory corruption we get random behavior from >> > >> application. So its best to fix all these issues and make this >> > >> application valgrind clean. >> > >> >> > >> Satish >> > >> >> > >> On Sat, 30 Sep 2017, Hao Zhang wrote: >> > >> >> > >> >> > >> >> > >> > ==31497== Invalid write of size 8 >> > >> > ==31497== at 0x643685: level_wave_func_Meniscus >> (imkcurve.c:1826) >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > >> > ==31497== Address 0x7fe859c38 is on thread 1's stack >> > >> > ==31497== >> > >> > ==31497== Invalid write of size 8 >> > >> > ==31497== at 0x64368C: level_wave_func_Meniscus >> (imkcurve.c:1826) >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack >> > >> > ==31497== >> > >> > ==31497== Invalid read of size 8 >> > >> > ==31497== at 0x643693: level_wave_func_Meniscus >> (imkcurve.c:1831) >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > >> > ==31497== Address 0x7fe859c38 is on thread 1's stack >> > >> > ==31497== >> > >> > ==31497== Invalid write of size 8 >> > >> > ==31497== at 0x64372D: level_wave_func_Meniscus >> (imkcurve.c:1848) >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > >> > ==31497== Address 0x7fe859c18 is on thread 1's stack >> > >> > ==31497== >> > >> > ==31497== Invalid read of size 8 >> > >> > ==31497== at 0x7ED3127: tan (in /lib64/libm-2.12.so) >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > >> > ==31497== Address 0x7fe859c18 is on thread 1's stack >> > >> > ==31497== >> > >> > ==31497== Invalid read of size 8 >> > >> > ==31497== at 0x643755: level_wave_func_Meniscus >> (imkcurve.c:1853) >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack >> > >> > ==31497== >> > >> > ==31497== Invalid read of size 8 >> > >> > ==31497== at 0x643765: level_wave_func_Meniscus >> (imkcurve.c:1854) >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack >> > >> > ==31497== >> > >> > ==31497== Invalid read of size 8 >> > >> > ==31497== at 0x643A7F: level_wave_func_Meniscus >> (imkcurve.c:1944) >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack >> > >> > ==31497== >> > >> > ==31497== Invalid write of size 8 >> > >> > ==31497== at 0x643A92: level_wave_func_Meniscus >> (imkcurve.c:1944) >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > >> > ==31497== Address 0x7feffecf0 is on thread 1's stack >> > >> > ==31497== >> > >> > ==31497== Invalid read of size 8 >> > >> > ==31497== at 0x643AEB: level_wave_func_Meniscus >> (imkcurve.c:1955) >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) >> > >> > ==31497== Address 0x7feffecf0 is on thread 1's stack >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x428B9F: >> > >> > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() >> (iFbasic.cpp:2799) >> > >> > ==31497== by 0x4B8BB9: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_ >> > >> RSSY_vd(_LEVEL_FUNC_PACK*) >> > >> > (iFcartsn3d.cpp:15253) >> > >> > ==31497== by 0x40C485: main (iFluid.cpp:349) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x428BA9: >> > >> > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() >> (iFbasic.cpp:2799) >> > >> > ==31497== by 0x4B8BB9: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_ >> > >> RSSY_vd(_LEVEL_FUNC_PACK*) >> > >> > (iFcartsn3d.cpp:15253) >> > >> > ==31497== by 0x40C485: main (iFluid.cpp:349) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x85B00E3: vfprintf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x486108: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothPro >> > >> perty_velocity_MAC_coupled_vd() >> > >> > (iFcartsn3d.cpp:6101) >> > >> > ==31497== by 0x4CBF0D: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18574) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x85B0101: vfprintf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x486108: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothPro >> > >> perty_velocity_MAC_coupled_vd() >> > >> > (iFcartsn3d.cpp:6101) >> > >> > ==31497== by 0x4CBF0D: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18574) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x4ED737: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > >> > (iFcartsn3d.cpp:24519) >> > >> > ==31497== by 0x4CBFBC: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18594) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x4ED754: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > >> > (iFcartsn3d.cpp:24519) >> > >> > ==31497== by 0x4CBFBC: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18594) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x85B4CF0: __printf_fp (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x4ED9F7: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > >> > (iFcartsn3d.cpp:24549) >> > >> > ==31497== by 0x4CBFBC: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18594) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x85B4F91: __printf_fp (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x4ED9F7: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > >> > (iFcartsn3d.cpp:24549) >> > >> > ==31497== by 0x4CBFBC: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18594) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x85AE396: __mpn_extract_double (in /lib64/ >> libc-2.12.so >> > >> ) >> > >> > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x4ED9F7: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > >> > (iFcartsn3d.cpp:24549) >> > >> > ==31497== by 0x4CBFBC: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18594) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x85AE39B: __mpn_extract_double (in /lib64/ >> libc-2.12.so >> > >> ) >> > >> > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x4ED9F7: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > >> > (iFcartsn3d.cpp:24549) >> > >> > ==31497== by 0x4CBFBC: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18594) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x85B59EC: __printf_fp (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x4ED9F7: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > >> > (iFcartsn3d.cpp:24549) >> > >> > ==31497== by 0x4CBFBC: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18594) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x85B5A9D: __printf_fp (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x4ED9F7: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > >> > (iFcartsn3d.cpp:24549) >> > >> > ==31497== by 0x4CBFBC: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18594) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x85B5C13: __printf_fp (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x4ED9F7: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > >> > (iFcartsn3d.cpp:24549) >> > >> > ==31497== by 0x4CBFBC: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18594) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x85B5CA4: __printf_fp (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x4ED9F7: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > >> > (iFcartsn3d.cpp:24549) >> > >> > ==31497== by 0x4CBFBC: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18594) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x85B625E: __printf_fp (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x4ED9F7: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > >> > (iFcartsn3d.cpp:24549) >> > >> > ==31497== by 0x4CBFBC: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18594) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Conditional jump or move depends on uninitialised >> value(s) >> > >> > ==31497== at 0x85B6242: __printf_fp (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x4ED9F7: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() >> > >> > (iFcartsn3d.cpp:24549) >> > >> > ==31497== by 0x4CBFBC: >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) >> > >> > (iFcartsn3d.cpp:18594) >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== >> > >> > ==31497== Syscall param write(buf) points to uninitialised byte(s) >> > >> > ==31497== at 0x86465ED: ??? (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x85DCAD2: _IO_file_write@@GLIBC_2.2.5 (in /lib64/ >> > >> > libc-2.12.so) >> > >> > ==31497== by 0x85DE084: _IO_do_write@@GLIBC_2.2.5 (in /lib64/ >> > >> libc-2.12.so >> > >> > ) >> > >> > ==31497== by 0x85DD287: _IO_file_sync@@GLIBC_2.2.5 (in /lib64/ >> > >> > libc-2.12.so) >> > >> > ==31497== by 0x85D1999: fflush (in /lib64/libc-2.12.so) >> > >> > ==31497== by 0x6FACFA: preserve_front_advance_front3d >> (fadv.c:2255) >> > >> > ==31497== by 0x6FA14E: mixed_advance_front3d (fadv.c:1895) >> > >> > ==31497== by 0x6F9B2D: advance_front3d_tracking_control >> > >> (fadv.c:1703) >> > >> > ==31497== by 0x54E119: FrontAdvance (fmap.c:142) >> > >> > ==31497== by 0x54DF27: FT_Propagate (fmap.c:80) >> > >> > ==31497== by 0x40CA37: ifluid_driver(_Front*, >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:512) >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) >> > >> > ==31497== Address 0x412308a is not stack'd, malloc'd or (recently) >> > >> free'd >> > >> > ==31497== >> > >> > >> > >> > On Fri, Sep 29, 2017 at 8:45 PM, Satish Balay >> > >> wrote: >> > >> > >> > >> > > Can you run this code with valgrind and send the log? >> > >> > > >> > >> > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > >> > > >> > >> > > Satish >> > >> > > >> > >> > > On Sat, 30 Sep 2017, Hao Zhang wrote: >> > >> > > >> > >> > > > I've modified my source code accordingly using PETSc 3.8.0. It >> > >> seems no >> > >> > > > more useful info got printed out. >> > >> > > > >> > >> > > > On Fri, Sep 29, 2017 at 8:15 PM, Hao Zhang < >> hbcbh1999 at gmail.com> >> > >> wrote: >> > >> > > > >> > >> > > > > I'm using -fp_trap as I run the simulation. the output only >> gives >> > >> > > > > >> > >> > > > > floating point exception >> > >> > > > > >> > >> > > > > It doesn't matter coarse grid or medium grid or more refined >> > >> grid. the >> > >> > > > > program crashed right away. >> > >> > > > > >> > >> > > > > On Fri, Sep 29, 2017 at 12:19 AM, Barry Smith < >> bsmith at mcs.anl.gov >> > >> > >> > >> > > wrote: >> > >> > > > > >> > >> > > > >> >> > >> > > > >> Please upgrade to the latest 3.8 version of PETSc. and >> then >> > >> run with >> > >> > > > >> -fp_trap >> > >> > > > >> >> > >> > > > >> This message is usually an indication that a NaN or Inf >> got >> > >> into >> > >> > > the >> > >> > > > >> numerical computation. Using the latest PETSc will make it >> much >> > >> > > easier to >> > >> > > > >> track down. >> > >> > > > >> >> > >> > > > >> Barry >> > >> > > > >> >> > >> > > > >> > On Sep 28, 2017, at 8:57 PM, Hao Zhang < >> hbcbh1999 at gmail.com> >> > >> wrote: >> > >> > > > >> > >> > >> > > > >> > I'm experiencing error when doing mesh refinement(mesh x2 >> for >> > >> X, Y >> > >> > > and >> > >> > > > >> Z direction) for 3D poisson problem. >> > >> > > > >> > PETSc produces no errors when no mesh refinement. I've >> pasted >> > >> some >> > >> > > > >> debugging info here. Please advise. >> > >> > > > >> > >> > >> > > > >> > >> > >> > > > >> > [0]PETSC ERROR: --------------------- Error Message >> > >> > > > >> ------------------------------ >> -------------------------------- >> > >> > > > >> > [0]PETSC ERROR: Invalid argument >> > >> > > > >> > [0]PETSC ERROR: Scalar value must be same on all >> processes, >> > >> > > argument # 3 >> > >> > > > >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ >> > >> > > documentation/faq.html >> > >> > > > >> for trouble shooting. >> > >> > > > >> > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 >> > >> > > > >> > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid >> on a >> > >> > > > >> arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 >> 23:47:39 >> > >> 2017 >> > >> > > > >> > [0]PETSC ERROR: Configure options >> --with-mpi-dir=/cm/shared/ >> > >> > > apps/openmpi/gcc/64/1.8.5/ >> > >> > > > >> --download-fblaslapack=/home/h >> aozhang/tmp/fblaslapack-3.4.2. >> > >> tar.gz >> > >> > > > >> --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz >> > >> > > > >> --with-valgrind-include=/home/haozhang/include/valgrind/ >> > >> > > > >> > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in >> > >> > > > >> /home/haozhang/tmp/petsc-3.5.4 >> _HYPRE/src/vec/vec/interface/r >> > >> vector.c >> > >> > > > >> > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in >> > >> > > > >> /home/haozhang/tmp/petsc-3.5.4 >> _HYPRE/src/ksp/ksp/impls/bcgs/ >> > >> bcgs.c >> > >> > > > >> > [0]PETSC ERROR: #3 KSPSolve() line 460 in >> > >> > > /home/haozhang/tmp/petsc-3.5.4 >> > >> > > > >> _HYPRE/src/ksp/ksp/interface/itfunc.c >> > >> > > > >> > [vn18:99838] *** Process received signal *** >> > >> > > > >> > [vn18:99838] Signal: Aborted (6) >> > >> > > > >> > [vn18:99838] Signal code: (-6) >> > >> > > > >> > [vn18:99848] *** Process received signal *** >> > >> > > > >> > [vn18:99849] *** Process received signal *** >> > >> > > > >> > [vn18:99849] Signal: Aborted (6) >> > >> > > > >> > [vn18:99849] Signal code: (-6) >> > >> > > > >> > [vn18:99848] Signal: Aborted (6) >> > >> > > > >> > [vn18:99848] Signal code: (-6) >> > >> > > > >> > [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790 >> > >> )[0x2aaaadbb4790] >> > >> > > > >> > [vn18:99838] [ 1] [vn18:99848] [ 0] >> > >> /lib64/libpthread.so.0(+0xf790 >> > >> > > > >> )[0x2aaaadbb4790] >> > >> > > > >> > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35) >> > >> [0x2aaaaddf5625] >> > >> > > > >> > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[ >> > >> 0x2aaaaddf6e05] >> > >> > > > >> > [vn18:99838] [ 3] [vn18:99849] [ 0] >> > >> /lib64/libpthread.so.0(+0xf790 >> > >> > > > >> )[0x2aaaadbb4790] >> > >> > > > >> > [vn18:99849] [ 1] [vn15:122106] *** Process received >> signal *** >> > >> > > > >> > [vn15:122106] Signal: Aborted (6) >> > >> > > > >> > [vn15:122106] Signal code: (-6) >> > >> > > > >> > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] >> > >> > > > >> > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[ >> > >> 0x2aaaaddf6e05] >> > >> > > > >> > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4 >> > >> > > > >> _HYPRE/arch-linux2-c-debug/lib >> /libpetsc.so.3.5(PetscTraceBac >> > >> > > > >> kErrorHandler+0x503)[0x2aaaaae1d7e1] >> > >> > > > >> > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35) >> > >> [0x2aaaaddf5625] >> > >> > > > >> > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[ >> > >> 0x2aaaaddf6e05] >> > >> > > > >> > [vn18:99849] [ 3] [vn15:122107] *** Process received >> signal *** >> > >> > > > >> > [vn15:122107] Signal: Aborted (6) >> > >> > > > >> > [vn15:122107] Signal code: (-6) >> > >> > > > >> > [vn16:86295] *** Process received signal *** >> > >> > > > >> > [vn16:86295] Signal: Aborted (6) >> > >> > > > >> > [vn16:86295] Signal code: (-6) >> > >> > > > >> > [vn18:99824] *** Process received signal *** >> > >> > > > >> > [vn18:99824] Signal: Aborted (6) >> > >> > > > >> > [vn18:99824] Signal code: (-6) >> > >> > > > >> > [vn18:99844] *** Process received signal *** >> > >> > > > >> > [vn18:99844] Signal: Aborted (6) >> > >> > > > >> > [vn18:99844] Signal code: (-6) >> > >> > > > >> > /home/haozhang/tmp/petsc-3.5.4 >> _HYPRE/arch-linux2-c-debug/lib >> > >> > > > >> /libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] >> > >> > > > >> > >> > >> > > > >> > >> > >> > > > >> > >> > >> > > > >> > -- >> > >> > > > >> > Hao Zhang >> > >> > > > >> > Dept. of Applid Mathematics and Statistics, >> > >> > > > >> > Stony Brook University, >> > >> > > > >> > Stony Brook, New York, 11790 >> > >> > > > >> >> > >> > > > >> >> > >> > > > > >> > >> > > > > >> > >> > > > > -- >> > >> > > > > Hao Zhang >> > >> > > > > Dept. of Applid Mathematics and Statistics, >> > >> > > > > Stony Brook University, >> > >> > > > > Stony Brook, New York, 11790 >> > >> > > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > >> > >> > > >> > >> > >> > >> > >> > >> > >> > >> >> > >> >> > > >> > > >> > > -- >> > > Hao Zhang >> > > Dept. of Applid Mathematics and Statistics, >> > > Stony Brook University, >> > > Stony Brook, New York, 11790 >> > > >> > >> > >> > >> > >> >> > > > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 > -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sat Sep 30 06:56:02 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 30 Sep 2017 06:56:02 -0500 Subject: [petsc-users] mesh refinement PETSc3.5.4 with HYPRE In-Reply-To: References: Message-ID: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatView.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecView.html Satish On Sat, 30 Sep 2017, Hao Zhang wrote: > is there a builtin way in PETSc I can flush cache in C++, specifically > matrix entries so that I can look up? to check which entry is NaN. Thanks. > > On Fri, Sep 29, 2017 at 11:16 PM, Hao Zhang wrote: > > > Thanks! > > > > On Fri, Sep 29, 2017 at 11:14 PM, Satish Balay wrote: > > > >> On Sat, 30 Sep 2017, Hao Zhang wrote: > >> > >> > it seems that running petsc with download-mpich is a dead end for me. > >> I've > >> > re-compiled PETSc with mpich options but I can't reinstall TORQUE for > >> all > >> > users just for my own test. TORQUE needs root access. > >> > Do you have more suggestion on the issue? > >> > >> As mentioned - the previous log pointed to errors in your code. You > >> should look > >> at them - and try to fix your code. > >> > >> > just a quick refresh. my PETSc > >> > complained floating point exception. for coarse and medium mesh grid > >> > without fp_trap option, my program runs. with fp_trap, program crashed > >> > right away. > >> > >> After memory corruptions are eliminated - if the errors persist - you > >> might have to use the debugger to determine why NaNs are getting > >> generated. > >> > >> Satish > >> > >> > > >> > Thanks again! > >> > > >> > On Fri, Sep 29, 2017 at 9:33 PM, Hao Zhang wrote: > >> > > >> > > Thanks. I will recompile and submit a new debug log > >> > > > >> > > On Fri, Sep 29, 2017 at 9:31 PM, Satish Balay > >> wrote: > >> > > > >> > >> Most of this is from openmpi. Its best to use --download-mpich > >> > >> for valgrind clean mpi as mentioned below. > >> > >> > >> > >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > >> > >> > >> > >> [attempting to remove openmpi messages] - all messages below appear > >> to > >> > >> be from your appliation code. And I don't see any messages from PETSc > >> > >> code in this log. > >> > >> > >> > >> Once there is memory corruption we get random behavior from > >> > >> application. So its best to fix all these issues and make this > >> > >> application valgrind clean. > >> > >> > >> > >> Satish > >> > >> > >> > >> On Sat, 30 Sep 2017, Hao Zhang wrote: > >> > >> > >> > >> > >> > >> > >> > >> > ==31497== Invalid write of size 8 > >> > >> > ==31497== at 0x643685: level_wave_func_Meniscus > >> (imkcurve.c:1826) > >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > >> > ==31497== Address 0x7fe859c38 is on thread 1's stack > >> > >> > ==31497== > >> > >> > ==31497== Invalid write of size 8 > >> > >> > ==31497== at 0x64368C: level_wave_func_Meniscus > >> (imkcurve.c:1826) > >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack > >> > >> > ==31497== > >> > >> > ==31497== Invalid read of size 8 > >> > >> > ==31497== at 0x643693: level_wave_func_Meniscus > >> (imkcurve.c:1831) > >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > >> > ==31497== Address 0x7fe859c38 is on thread 1's stack > >> > >> > ==31497== > >> > >> > ==31497== Invalid write of size 8 > >> > >> > ==31497== at 0x64372D: level_wave_func_Meniscus > >> (imkcurve.c:1848) > >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > >> > ==31497== Address 0x7fe859c18 is on thread 1's stack > >> > >> > ==31497== > >> > >> > ==31497== Invalid read of size 8 > >> > >> > ==31497== at 0x7ED3127: tan (in /lib64/libm-2.12.so) > >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > >> > ==31497== Address 0x7fe859c18 is on thread 1's stack > >> > >> > ==31497== > >> > >> > ==31497== Invalid read of size 8 > >> > >> > ==31497== at 0x643755: level_wave_func_Meniscus > >> (imkcurve.c:1853) > >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack > >> > >> > ==31497== > >> > >> > ==31497== Invalid read of size 8 > >> > >> > ==31497== at 0x643765: level_wave_func_Meniscus > >> (imkcurve.c:1854) > >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack > >> > >> > ==31497== > >> > >> > ==31497== Invalid read of size 8 > >> > >> > ==31497== at 0x643A7F: level_wave_func_Meniscus > >> (imkcurve.c:1944) > >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > >> > ==31497== Address 0x7fe859c30 is on thread 1's stack > >> > >> > ==31497== > >> > >> > ==31497== Invalid write of size 8 > >> > >> > ==31497== at 0x643A92: level_wave_func_Meniscus > >> (imkcurve.c:1944) > >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > >> > ==31497== Address 0x7feffecf0 is on thread 1's stack > >> > >> > ==31497== > >> > >> > ==31497== Invalid read of size 8 > >> > >> > ==31497== at 0x643AEB: level_wave_func_Meniscus > >> (imkcurve.c:1955) > >> > >> > ==31497== by 0x65299B: assign_two_comp_domain (imksurf.c:2046) > >> > >> > ==31497== by 0x64623B: make_level_surface (imksurf.c:245) > >> > >> > ==31497== by 0x5458AF: FT_InitIntfc3d (finit.c:2137) > >> > >> > ==31497== by 0x544CA3: FT_InitIntfc (finit.c:1853) > >> > >> > ==31497== by 0x40C072: main (iFluid.cpp:218) > >> > >> > ==31497== Address 0x7feffecf0 is on thread 1's stack > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x428B9F: > >> > >> > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() > >> (iFbasic.cpp:2799) > >> > >> > ==31497== by 0x4B8BB9: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_ > >> > >> RSSY_vd(_LEVEL_FUNC_PACK*) > >> > >> > (iFcartsn3d.cpp:15253) > >> > >> > ==31497== by 0x40C485: main (iFluid.cpp:349) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x428BA9: > >> > >> > Incompress_Solver_Smooth_Basis::setAdvectionDt_vd() > >> (iFbasic.cpp:2799) > >> > >> > ==31497== by 0x4B8BB9: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::setInitialCondition_ > >> > >> RSSY_vd(_LEVEL_FUNC_PACK*) > >> > >> > (iFcartsn3d.cpp:15253) > >> > >> > ==31497== by 0x40C485: main (iFluid.cpp:349) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x85B00E3: vfprintf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x486108: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothPro > >> > >> perty_velocity_MAC_coupled_vd() > >> > >> > (iFcartsn3d.cpp:6101) > >> > >> > ==31497== by 0x4CBF0D: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18574) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x85B0101: vfprintf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x486108: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::compDiffWithSmoothPro > >> > >> perty_velocity_MAC_coupled_vd() > >> > >> > (iFcartsn3d.cpp:6101) > >> > >> > ==31497== by 0x4CBF0D: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18574) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x4ED737: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > >> > (iFcartsn3d.cpp:24519) > >> > >> > ==31497== by 0x4CBFBC: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18594) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x4ED754: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > >> > (iFcartsn3d.cpp:24519) > >> > >> > ==31497== by 0x4CBFBC: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18594) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x85B4CF0: __printf_fp (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x4ED9F7: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > >> > (iFcartsn3d.cpp:24549) > >> > >> > ==31497== by 0x4CBFBC: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18594) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x85B4F91: __printf_fp (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x4ED9F7: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > >> > (iFcartsn3d.cpp:24549) > >> > >> > ==31497== by 0x4CBFBC: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18594) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x85AE396: __mpn_extract_double (in /lib64/ > >> libc-2.12.so > >> > >> ) > >> > >> > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x4ED9F7: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > >> > (iFcartsn3d.cpp:24549) > >> > >> > ==31497== by 0x4CBFBC: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18594) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x85AE39B: __mpn_extract_double (in /lib64/ > >> libc-2.12.so > >> > >> ) > >> > >> > ==31497== by 0x85B51BD: __printf_fp (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x4ED9F7: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > >> > (iFcartsn3d.cpp:24549) > >> > >> > ==31497== by 0x4CBFBC: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18594) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x85B59EC: __printf_fp (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x4ED9F7: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > >> > (iFcartsn3d.cpp:24549) > >> > >> > ==31497== by 0x4CBFBC: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18594) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x85B5A9D: __printf_fp (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x4ED9F7: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > >> > (iFcartsn3d.cpp:24549) > >> > >> > ==31497== by 0x4CBFBC: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18594) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x85B5C13: __printf_fp (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x4ED9F7: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > >> > (iFcartsn3d.cpp:24549) > >> > >> > ==31497== by 0x4CBFBC: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18594) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x85B5CA4: __printf_fp (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x4ED9F7: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > >> > (iFcartsn3d.cpp:24549) > >> > >> > ==31497== by 0x4CBFBC: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18594) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x85B625E: __printf_fp (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x4ED9F7: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > >> > (iFcartsn3d.cpp:24549) > >> > >> > ==31497== by 0x4CBFBC: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18594) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Conditional jump or move depends on uninitialised > >> value(s) > >> > >> > ==31497== at 0x85B6242: __printf_fp (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85B089F: vfprintf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85BA189: printf (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x4ED9F7: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::computeSubgridModel_vd() > >> > >> > (iFcartsn3d.cpp:24549) > >> > >> > ==31497== by 0x4CBFBC: > >> > >> > Incompress_Solver_Smooth_3D_Cartesian::solve_vd(double) > >> > >> > (iFcartsn3d.cpp:18594) > >> > >> > ==31497== by 0x40C72E: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:419) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== > >> > >> > ==31497== Syscall param write(buf) points to uninitialised byte(s) > >> > >> > ==31497== at 0x86465ED: ??? (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x85DCAD2: _IO_file_write@@GLIBC_2.2.5 (in /lib64/ > >> > >> > libc-2.12.so) > >> > >> > ==31497== by 0x85DE084: _IO_do_write@@GLIBC_2.2.5 (in /lib64/ > >> > >> libc-2.12.so > >> > >> > ) > >> > >> > ==31497== by 0x85DD287: _IO_file_sync@@GLIBC_2.2.5 (in /lib64/ > >> > >> > libc-2.12.so) > >> > >> > ==31497== by 0x85D1999: fflush (in /lib64/libc-2.12.so) > >> > >> > ==31497== by 0x6FACFA: preserve_front_advance_front3d > >> (fadv.c:2255) > >> > >> > ==31497== by 0x6FA14E: mixed_advance_front3d (fadv.c:1895) > >> > >> > ==31497== by 0x6F9B2D: advance_front3d_tracking_control > >> > >> (fadv.c:1703) > >> > >> > ==31497== by 0x54E119: FrontAdvance (fmap.c:142) > >> > >> > ==31497== by 0x54DF27: FT_Propagate (fmap.c:80) > >> > >> > ==31497== by 0x40CA37: ifluid_driver(_Front*, > >> > >> > Incompress_Solver_Smooth_Basis*, _F_BASIC_DATA*) (iFluid.cpp:512) > >> > >> > ==31497== by 0x40C4F3: main (iFluid.cpp:369) > >> > >> > ==31497== Address 0x412308a is not stack'd, malloc'd or (recently) > >> > >> free'd > >> > >> > ==31497== > >> > >> > > >> > >> > On Fri, Sep 29, 2017 at 8:45 PM, Satish Balay > >> > >> wrote: > >> > >> > > >> > >> > > Can you run this code with valgrind and send the log? > >> > >> > > > >> > >> > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > >> > >> > > > >> > >> > > Satish > >> > >> > > > >> > >> > > On Sat, 30 Sep 2017, Hao Zhang wrote: > >> > >> > > > >> > >> > > > I've modified my source code accordingly using PETSc 3.8.0. It > >> > >> seems no > >> > >> > > > more useful info got printed out. > >> > >> > > > > >> > >> > > > On Fri, Sep 29, 2017 at 8:15 PM, Hao Zhang < > >> hbcbh1999 at gmail.com> > >> > >> wrote: > >> > >> > > > > >> > >> > > > > I'm using -fp_trap as I run the simulation. the output only > >> gives > >> > >> > > > > > >> > >> > > > > floating point exception > >> > >> > > > > > >> > >> > > > > It doesn't matter coarse grid or medium grid or more refined > >> > >> grid. the > >> > >> > > > > program crashed right away. > >> > >> > > > > > >> > >> > > > > On Fri, Sep 29, 2017 at 12:19 AM, Barry Smith < > >> bsmith at mcs.anl.gov > >> > >> > > >> > >> > > wrote: > >> > >> > > > > > >> > >> > > > >> > >> > >> > > > >> Please upgrade to the latest 3.8 version of PETSc. and > >> then > >> > >> run with > >> > >> > > > >> -fp_trap > >> > >> > > > >> > >> > >> > > > >> This message is usually an indication that a NaN or Inf > >> got > >> > >> into > >> > >> > > the > >> > >> > > > >> numerical computation. Using the latest PETSc will make it > >> much > >> > >> > > easier to > >> > >> > > > >> track down. > >> > >> > > > >> > >> > >> > > > >> Barry > >> > >> > > > >> > >> > >> > > > >> > On Sep 28, 2017, at 8:57 PM, Hao Zhang < > >> hbcbh1999 at gmail.com> > >> > >> wrote: > >> > >> > > > >> > > >> > >> > > > >> > I'm experiencing error when doing mesh refinement(mesh x2 > >> for > >> > >> X, Y > >> > >> > > and > >> > >> > > > >> Z direction) for 3D poisson problem. > >> > >> > > > >> > PETSc produces no errors when no mesh refinement. I've > >> pasted > >> > >> some > >> > >> > > > >> debugging info here. Please advise. > >> > >> > > > >> > > >> > >> > > > >> > > >> > >> > > > >> > [0]PETSC ERROR: --------------------- Error Message > >> > >> > > > >> ------------------------------ > >> -------------------------------- > >> > >> > > > >> > [0]PETSC ERROR: Invalid argument > >> > >> > > > >> > [0]PETSC ERROR: Scalar value must be same on all > >> processes, > >> > >> > > argument # 3 > >> > >> > > > >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ > >> > >> > > documentation/faq.html > >> > >> > > > >> for trouble shooting. > >> > >> > > > >> > [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015 > >> > >> > > > >> > [0]PETSC ERROR: /home/haozhang/FT.NO.FULL/iFluid/iFluid > >> on a > >> > >> > > > >> arch-linux2-c-debug named vn18 by haozhang Thu Sep 28 > >> 23:47:39 > >> > >> 2017 > >> > >> > > > >> > [0]PETSC ERROR: Configure options > >> --with-mpi-dir=/cm/shared/ > >> > >> > > apps/openmpi/gcc/64/1.8.5/ > >> > >> > > > >> --download-fblaslapack=/home/h > >> aozhang/tmp/fblaslapack-3.4.2. > >> > >> tar.gz > >> > >> > > > >> --download-hypre=/home/haozhang/tmp/hypre-2.9.1a.tar.gz > >> > >> > > > >> --with-valgrind-include=/home/haozhang/include/valgrind/ > >> > >> > > > >> > [0]PETSC ERROR: #1 VecAXPBYPCZ() line 731 in > >> > >> > > > >> /home/haozhang/tmp/petsc-3.5.4 > >> _HYPRE/src/vec/vec/interface/r > >> > >> vector.c > >> > >> > > > >> > [0]PETSC ERROR: #2 KSPSolve_BCGS() line 87 in > >> > >> > > > >> /home/haozhang/tmp/petsc-3.5.4 > >> _HYPRE/src/ksp/ksp/impls/bcgs/ > >> > >> bcgs.c > >> > >> > > > >> > [0]PETSC ERROR: #3 KSPSolve() line 460 in > >> > >> > > /home/haozhang/tmp/petsc-3.5.4 > >> > >> > > > >> _HYPRE/src/ksp/ksp/interface/itfunc.c > >> > >> > > > >> > [vn18:99838] *** Process received signal *** > >> > >> > > > >> > [vn18:99838] Signal: Aborted (6) > >> > >> > > > >> > [vn18:99838] Signal code: (-6) > >> > >> > > > >> > [vn18:99848] *** Process received signal *** > >> > >> > > > >> > [vn18:99849] *** Process received signal *** > >> > >> > > > >> > [vn18:99849] Signal: Aborted (6) > >> > >> > > > >> > [vn18:99849] Signal code: (-6) > >> > >> > > > >> > [vn18:99848] Signal: Aborted (6) > >> > >> > > > >> > [vn18:99848] Signal code: (-6) > >> > >> > > > >> > [vn18:99838] [ 0] /lib64/libpthread.so.0(+0xf790 > >> > >> )[0x2aaaadbb4790] > >> > >> > > > >> > [vn18:99838] [ 1] [vn18:99848] [ 0] > >> > >> /lib64/libpthread.so.0(+0xf790 > >> > >> > > > >> )[0x2aaaadbb4790] > >> > >> > > > >> > [vn18:99848] [ 1] /lib64/libc.so.6(gsignal+0x35) > >> > >> [0x2aaaaddf5625] > >> > >> > > > >> > [vn18:99838] [ 2] /lib64/libc.so.6(abort+0x175)[ > >> > >> 0x2aaaaddf6e05] > >> > >> > > > >> > [vn18:99838] [ 3] [vn18:99849] [ 0] > >> > >> /lib64/libpthread.so.0(+0xf790 > >> > >> > > > >> )[0x2aaaadbb4790] > >> > >> > > > >> > [vn18:99849] [ 1] [vn15:122106] *** Process received > >> signal *** > >> > >> > > > >> > [vn15:122106] Signal: Aborted (6) > >> > >> > > > >> > [vn15:122106] Signal code: (-6) > >> > >> > > > >> > /lib64/libc.so.6(gsignal+0x35)[0x2aaaaddf5625] > >> > >> > > > >> > [vn18:99848] [ 2] /lib64/libc.so.6(abort+0x175)[ > >> > >> 0x2aaaaddf6e05] > >> > >> > > > >> > [vn18:99848] [ 3] /home/haozhang/tmp/petsc-3.5.4 > >> > >> > > > >> _HYPRE/arch-linux2-c-debug/lib > >> /libpetsc.so.3.5(PetscTraceBac > >> > >> > > > >> kErrorHandler+0x503)[0x2aaaaae1d7e1] > >> > >> > > > >> > [vn18:99838] [ 4] /lib64/libc.so.6(gsignal+0x35) > >> > >> [0x2aaaaddf5625] > >> > >> > > > >> > [vn18:99849] [ 2] /lib64/libc.so.6(abort+0x175)[ > >> > >> 0x2aaaaddf6e05] > >> > >> > > > >> > [vn18:99849] [ 3] [vn15:122107] *** Process received > >> signal *** > >> > >> > > > >> > [vn15:122107] Signal: Aborted (6) > >> > >> > > > >> > [vn15:122107] Signal code: (-6) > >> > >> > > > >> > [vn16:86295] *** Process received signal *** > >> > >> > > > >> > [vn16:86295] Signal: Aborted (6) > >> > >> > > > >> > [vn16:86295] Signal code: (-6) > >> > >> > > > >> > [vn18:99824] *** Process received signal *** > >> > >> > > > >> > [vn18:99824] Signal: Aborted (6) > >> > >> > > > >> > [vn18:99824] Signal code: (-6) > >> > >> > > > >> > [vn18:99844] *** Process received signal *** > >> > >> > > > >> > [vn18:99844] Signal: Aborted (6) > >> > >> > > > >> > [vn18:99844] Signal code: (-6) > >> > >> > > > >> > /home/haozhang/tmp/petsc-3.5.4 > >> _HYPRE/arch-linux2-c-debug/lib > >> > >> > > > >> /libpetsc.so.3.5(PetscError+0x338)[0x2aaaaae183b3] > >> > >> > > > >> > > >> > >> > > > >> > > >> > >> > > > >> > > >> > >> > > > >> > -- > >> > >> > > > >> > Hao Zhang > >> > >> > > > >> > Dept. of Applid Mathematics and Statistics, > >> > >> > > > >> > Stony Brook University, > >> > >> > > > >> > Stony Brook, New York, 11790 > >> > >> > > > >> > >> > >> > > > >> > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > -- > >> > >> > > > > Hao Zhang > >> > >> > > > > Dept. of Applid Mathematics and Statistics, > >> > >> > > > > Stony Brook University, > >> > >> > > > > Stony Brook, New York, 11790 > >> > >> > > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > >> > >> > > > >> > >> > > >> > >> > > >> > >> > > >> > >> > >> > >> > >> > > > >> > > > >> > > -- > >> > > Hao Zhang > >> > > Dept. of Applid Mathematics and Statistics, > >> > > Stony Brook University, > >> > > Stony Brook, New York, 11790 > >> > > > >> > > >> > > >> > > >> > > >> > >> > > > > > > -- > > Hao Zhang > > Dept. of Applid Mathematics and Statistics, > > Stony Brook University, > > Stony Brook, New York, 11790 > > > > > >