<div dir="ltr">Also, I tested this on my Mac (this output) and on Cori at NERSC and the behavior looked identical.</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 18, 2017 at 8:44 PM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I get this strange error when I use GAMG, in parallel. I don't see it in serial and I don't see it with the default solver. <div><br></div><div>The problem seems to be that I use -vec_view in my code and the guts of Vec seem to be picking this up and causing havoc. In the past I have added a prefix to my -[x2_]vec_view and that seemed to work, but I would like to try to fix this and not shove it under the carpet again, if possible.<div><br></div><div>It looks like a call to VecAssemblyEnd in the guts of GAMG is calling PetscObjectViewFromOptions. I don't understand why this only happens in parallel.</div><div><br></div><div>Any ideas?</div><div>Thanks,<br><div><br></div><div><div>20:27 master= ~/Codes/picell/src$ make run NP=4</div><div>/Users/markadams/Codes/petsc/<wbr>arch-macosx-gnu-g/bin/mpiexec -n 4 ./picell.arch-macosx-gnu-g -petscspace_poly_tensor -petscspace_order 1 -mstep 2 -debug 3 -snes_monitor -ksp_monitor -snes_converged_reason -dm_refine 4 -dt .1 -use_bsp 10 -num_particles_proc -1 -num_phi_cells 8 -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append -dm_plex_periodic_cut</div><div>[0] **** Warning ****, no global point location. multiple processors (4) not supported</div><div>[0] npe=4; 2 x 2 x 1 flux tube grid; mpi_send size (chunksize) has 64 particles. ions only, BSP communication, phi section = 6.28319</div><div> createMesh ntheta_total=4.</div><div>[0] 961 equations on 4 processors, 256 local cells, (element 0 used for flux tube list)</div><div>DM Object: Parallel Mesh 4 MPI processes</div><div> type: plex</div><div>Parallel Mesh in 2 dimensions:</div><div> 0-cells: 289 289 289 289</div><div> 1-cells: 544 544 544 544</div><div> 2-cells: 256 256 256 256</div><div>Labels:</div><div> boundary: 1 strata with value/size (1 (65))</div><div> Face Sets: 2 strata with value/size (1 (31), 4 (31))</div><div> marker: 1 strata with value/size (1 (65))</div><div> depth: 3 strata with value/size (0 (289), 1 (544), 2 (256))</div><div> [0]shiftParticles: BSP have 1 for proc 0</div><div> [0]shiftParticles: BSP have 1 particles for proc 0 (1 chunks)</div><div> [0]shiftParticles: BSP sent 1</div><div> [2]shiftParticles: BSP sent 0</div><div> [3]shiftParticles: BSP sent 0</div><div> [1]shiftParticles: BSP sent 0</div><div>0) 1 local particles, 1/1 global, 0. % total particles moved in 0 messages total (to 1 processors local), 4. load imbalance factor</div><div> 0 SNES Function norm 6.176014522980e-01 </div><div> 0 KSP Residual norm 2.033453504856e+00 </div><div> 1 KSP Residual norm 1.247246018913e-01 </div><div> 2 KSP Residual norm 7.169223999741e-03 </div><div> 3 KSP Residual norm 3.494127469709e-04 </div><div> 4 KSP Residual norm 1.539576656127e-05 </div><div> 1 SNES Function norm 7.013048909984e-06 </div><div> 0 KSP Residual norm 1.539576656641e-05 </div><div> 1 KSP Residual norm 1.233278209501e-06 </div><div> 2 KSP Residual norm 6.474128793205e-08 </div><div> 3 KSP Residual norm 3.824829808334e-09 </div><div> 4 KSP Residual norm 1.813659764651e-10 </div><div> 5 KSP Residual norm 3.684110934970e-12 </div><div> 2 SNES Function norm 2.386983413511e-12 </div><div>Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 2</div><div> [0]shiftParticles: BSP have 0 for proc 0</div><div> [0]shiftParticles: BSP have 1 for proc 2</div><div> [0]shiftParticles: BSP have 1 particles for proc 2 (1 chunks)</div><div> [1]shiftParticles: BSP sent 0</div><div> [2]shiftParticles: BSP sent 0</div><div> [3]shiftParticles: BSP sent 0</div><div> [0]shiftParticles: BSP sent 1</div><div>1) 1 local particles, 1/1 global, 100. % total particles moved in 0 messages total (to 2 processors local), 4. load imbalance factor</div><div> 0 SNES Function norm 6.828943707479e-01 </div><div>HDF5-DIAG: Error detected in HDF5 (1.8.18) HDF5-DIAG: Error detected in HDF5 (1.8.18) MPI-process 3:</div><div> #000: H5F.c line 604 in H5Fopen(): unable to open file</div><div> major: File accessibilty</div><div> minor: Unable to open file</div><div> #001: H5Fint.c line 1087 in H5F_open(): unable to read superblock</div><div> major: File accessibilty</div><div> minor: Read failed</div><div> #002: H5Fsuper.c line 294 in H5F_super_read(): unable to load superblock</div><div> major: Object cache</div><div> minor: Unable to protect metadata</div><div> #003: H5AC.c line 1262 in H5AC_protect(): H5C_protect() failed.</div><div> major: Object cache</div><div> minor: Unable to protect metadata</div><div> #004: H5C.c line 3574 in H5C_protect(): can't load entry</div><div> major: Object cache</div><div> minor: Unable to load metadata into cache</div><div> #005: H5C.c line 7954 in H5C_load_entry(): unable to load entry</div><div> major: Object cache</div><div>HDF5-DIAG: Error detected in HDF5 (1.8.18) MPI-process 1:</div><div> #000: H5F.c line 604 in H5Fopen(): unable to open file</div><div> major: File accessibilty</div><div> minor: Unable to open file</div><div> #001: H5Fint.c line 1087 in H5F_open(): unable to read superblock</div><div> major: File accessibilty</div><div> minor: Read failed</div><div> #002: H5Fsuper.c line 294 in H5F_super_read(): unable to load superblock</div><div> major: Object cache</div><div> minor: Unable to protect metadata</div><div> #003: H5AC.c line 1262 in H5AC_protect(): H5C_protect() failed.</div><div> major: Object cache</div><div> minor: Unable to protect metadata</div><div> #004: H5C.c line 3574 in H5C_protect(): can't load entry</div><div> major: Object cache</div><div> minor: Unable to load metadata into cache</div><div> #005: H5C.c line 7954 in H5C_load_entry(): unable to load entry</div><div> major: Object cache</div><div> minor: Unable to load metadata into cache</div><div> #006: H5Fsuper_cache.c line 476 in H5F_sblock_load(): truncated file: eof = 365072, sblock->base_addr = 0, stored_eoa = 367568</div><div> major: File accessibilty</div><div> minor: File has been truncated</div><div> minor: Unable to load metadata into cache</div><div> #006: H5Fsuper_cache.c line 476 in H5F_sblock_load(): truncated file: eof = 365072, sblock->base_addr = 0, stored_eoa = 367568</div><div> major: File accessibilty</div><div> minor: File has been truncated</div></div><div><br></div><div>[snip]</div><div><br></div><div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: Error in external library</div><div>[0]PETSC ERROR: Error in HDF5 call H5Fopen() Status -1</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.7.6-5541-gd07ad3ccba GIT Date: 2017-09-17 13:49:32 -0400</div><div>[0]PETSC ERROR: ./picell.arch-macosx-gnu-g on a arch-macosx-gnu-g named MarksMac-5.local by markadams Mon Sep 18 20:27:26 2017</div><div>[0]PETSC ERROR: Configure options --with-cc=clang --with-cc++=clang++ COPTFLAGS="-g -mavx2" CXXOPTFLAGS="-g -mavx2" FOPTFLAGS="-g -mavx2" --download-mpich=1 --download-hypre=1 --download-metis=1 --download-parmetis=1 --download-c2html=1 --download-ctetgen --download-p4est=1 --download-superlu_dist --download-superlu --download-triangle=1 --download-hdf5=1 --download-zlib --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g --download-chaco --with-viewfromoptions=1</div><div>[0]PETSC ERROR: #1 PetscViewerFileSetName_HDF5() line 242 in /Users/markadams/Codes/petsc/<wbr>src/sys/classes/viewer/impls/<wbr>hdf5/hdf5v.c</div><div>[0]PETSC ERROR: #2 PetscViewerFileSetName() line 650 in /Users/markadams/Codes/petsc/<wbr>src/sys/classes/viewer/impls/<wbr>ascii/filev.c</div><div>[0]PETSC ERROR: #3 PetscOptionsGetViewer() line 339 in /Users/markadams/Codes/petsc/<wbr>src/sys/classes/viewer/<wbr>interface/viewreg.c</div><div>[0]PETSC ERROR: #4 PetscObjectViewFromOptions() line 2773 in /Users/markadams/Codes/petsc/<wbr>src/sys/objects/options.c</div><div>[0]PETSC ERROR: #5 VecAssemblyEnd() line 182 in /Users/markadams/Codes/petsc/<wbr>src/vec/vec/interface/vector.c</div><div>[0]PETSC ERROR: #6 PCGAMGGetDataWithGhosts() line 376 in /Users/markadams/Codes/petsc/<wbr>src/ksp/pc/impls/gamg/util.c</div><div>[0]PETSC ERROR: #7 PCGAMGProlongator_AGG() line 1071 in /Users/markadams/Codes/petsc/<wbr>src/ksp/pc/impls/gamg/agg.c</div><div>[0]PETSC ERROR: #8 PCSetUp_GAMG() line 519 in /Users/markadams/Codes/petsc/<wbr>src/ksp/pc/impls/gamg/gamg.c</div><div>[0]PETSC ERROR: #9 PCSetUp() line 924 in /Users/markadams/Codes/petsc/<wbr>src/ksp/pc/interface/precon.c</div><div>[0]PETSC ERROR: #10 KSPSetUp() line 378 in /Users/markadams/Codes/petsc/<wbr>src/ksp/ksp/interface/itfunc.c</div><div>[0]PETSC ERROR: #11 KSPSolve() line 609 in /Users/markadams/Codes/petsc/<wbr>src/ksp/ksp/interface/itfunc.c</div><div>[0]PETSC ERROR: #12 SNESSolve_NEWTONLS() line 224 in /Users/markadams/Codes/petsc/<wbr>src/snes/impls/ls/ls.c</div><div>[0]PETSC ERROR: #13 SNESSolve() line 4106 in /Users/markadams/Codes/petsc/<wbr>src/snes/interface/snes.c</div><div>[0]PETSC ERROR: #14 DMPICellSolve() line 18 in /Users/markadams/Codes/petsc/<wbr>src/snes/utils/dmpicellsnes.c</div><div>[0]PETSC ERROR: #15 go() line 1120 in /Users/markadams/Codes/picell/<wbr>src/main.c</div><div>[0]PETSC ERROR: #16 main() line 1304 in /Users/markadams/Codes/picell/<wbr>src/main.c</div><div>[0]PETSC ERROR: PETSc Option Table entries:</div><div>[0]PETSC ERROR: -debug 3</div><div>[0]PETSC ERROR: -dm_plex_periodic_cut</div><div>[0]PETSC ERROR: -dm_refine 4</div><div>[0]PETSC ERROR: -dm_view hdf5:sol.h5</div><div>[0]PETSC ERROR: -dt .1</div><div>[0]PETSC ERROR: -ksp_monitor</div><div>[0]PETSC ERROR: -ksp_type cg</div><div>[0]PETSC ERROR: -mstep 2</div><div>[0]PETSC ERROR: -num_particles_proc -1</div><div>[0]PETSC ERROR: -num_phi_cells 8</div><div>[0]PETSC ERROR: -options_left</div><div><font color="#ff0000">[0]PETSC ERROR: -pc_type gamg</font></div><div>[0]PETSC ERROR: -petscspace_order 1</div><div>[0]PETSC ERROR: -petscspace_poly_tensor</div><div>[0]PETSC ERROR: -snes_converged_reason</div><div>[0]PETSC ERROR: -snes_monitor</div><div>[0]PETSC ERROR: -use_bsp 10</div><div><font color="#ff0000">[0]PETSC ERROR: -vec_view hdf5:sol.h5::append</font></div><div>[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov-------<wbr>---</div><div>application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0</div></div><div><br></div></div></div></div>
</blockquote></div><br></div>