From fande.kong at colorado.edu Mon Jun 3 01:24:16 2013 From: fande.kong at colorado.edu (Fande Kong) Date: Mon, 3 Jun 2013 14:24:16 +0800 Subject: [petsc-users] memory crash with optimized petsc Message-ID: Hi all, I can correctly run my application based on the petsc ( --with-debugging = yes). But the same code encounters memory crash when it runs with optimized version ( --with-debugging = no) . The detailed information for errors and results is attached. The logs for configure and make are also attached. Now, there are several questions I want to know. (1) What are differences between debug version and optimized version in terms of memory management and communication? (2) I print some memory usages. Why does the optimized version cost twice more memory than the debug version? (3) How to debug the optimized code to fix memory issue? Regards, -- Fande Kong Department of Computer Science University of Colorado at Boulder -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: make and configure.zip Type: application/zip Size: 678518 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: result_debug_out.zip Type: application/zip Size: 23372 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: result_optimized_error.zip Type: application/zip Size: 13025 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: result_optimized_out.zip Type: application/zip Size: 16479 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Mon Jun 3 08:04:22 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 03 Jun 2013 08:04:22 -0500 Subject: [petsc-users] memory crash with optimized petsc In-Reply-To: References: Message-ID: <877gibs5ux.fsf@mcs.anl.gov> Fande Kong writes: > Hi all, > > I can correctly run my application based on the petsc ( --with-debugging = > yes). But the same code encounters memory crash when it runs with optimized > version ( --with-debugging = no) . The detailed information for errors and > results is attached. The logs for configure and make are also attached. > > > Now, there are several questions I want to know. > > (1) What are differences between debug version and optimized version in > terms of memory management and communication? > > (2) I print some memory usages. Why does the optimized version cost twice > more memory than the debug version? It shouldn't. What interface are you using to obtain memory usage? I suggest fixing the errors before worrying about this. You should figure out how to reproduce the issue on a smaller problem size and with fewer processes. That reduction alone may tell you where the issue occurs. > (3) How to debug the optimized code to fix memory issue? Is the debug version currently clean when run with -malloc_dump and clean when run under valgrind? If so, I would configure --with-debugging=0 COPTFLAGS='-O0 -g' this will take the "optimized" code path, but will have full information when run in the debugger. You'll need a smaller problem size and fewer processes (preferably one or two) to do this effectively. From fande.kong at colorado.edu Mon Jun 3 08:51:07 2013 From: fande.kong at colorado.edu (Fande Kong) Date: Mon, 3 Jun 2013 21:51:07 +0800 Subject: [petsc-users] memory crash with optimized petsc In-Reply-To: <877gibs5ux.fsf@mcs.anl.gov> References: <877gibs5ux.fsf@mcs.anl.gov> Message-ID: Thanks Jed? I run code with options: -malloc_debug and -malloc_dump and get the following results: [0]Total space allocated 217984 bytes [ 0]16 bytes PetscStrallocpy() line 186 in /home/fdkong/math/petsc-3.3-p7/src/sys/utils/str.c [0] PetscObjectChangeTypeName() line 129 in /home/fdkong/math/petsc-3.3-p7/src/sys/objects/pname.c [0] ISSetType() line 77 in /home/fdkong/math/petsc-3.3-p7/src/vec/is/interface/isreg.c [0] ISCreateGeneral() line 437 in /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c [0] SpmcsDMMeshGetStratumIS() line 1375 in spmcsdmmesh.cpp [0] ComputeMatrix() line 279 in linearElasticity3d.cpp [0] KSPSetUp() line 182 in /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c [0] PCSetUp() line 810 in /home/fdkong/math/petsc-3.3-p7/src/ksp/pc/interface/precon.c [0] KSPSetUp() line 182 in /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c [0] KSPSetFromOptions() line 284 in /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itcl.c [ 0]32 bytes ISCreate_General() line 516 in /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c [0] ISSetType() line 77 in /home/fdkong/math/petsc-3.3-p7/src/vec/is/interface/isreg.c [0] ISCreateGeneral() line 437 in /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c [0] SpmcsDMMeshGetStratumIS() line 1375 in spmcsdmmesh.cpp [0] ComputeMatrix() line 279 in linearElasticity3d.cpp [0] KSPSetUp() line 182 in /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c [0] PCSetUp() line 810 in /home/fdkong/math/petsc-3.3-p7/src/ksp/pc/interface/precon.c [0] KSPSetUp() line 182 in /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c [0] KSPSetFromOptions() line 284 in /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itcl.c [ 0]128 bytes ISCreate() line 43 in /home/fdkong/math/petsc-3.3-p7/src/vec/is/interface/isreg.c [0] ISCreateGeneral() line 437 in /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c [0] SpmcsDMMeshGetStratumIS() line 1375 in spmcsdmmesh.cpp [0] ComputeMatrix() line 279 in linearElasticity3d.cpp [0] KSPSetUp() line 182 in /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c [0] PCSetUp() line 810 in /home/fdkong/math/petsc-3.3-p7/src/ksp/pc/interface/precon.c [0] KSPSetUp() line 182 in /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c [0] KSPSetFromOptions() line 284 in /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itcl.c [ 0]64 bytes ISCreate() line 43 in /home/fdkong/math/petsc-3.3-p7/src/vec/is/interface/isreg.c [0] ISCreateGeneral() line 437 in /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c [0] SpmcsDMMeshGetStratumIS() line 1375 in spmcsdmmesh.cpp [0] ComputeMatrix() line 279 in linearElasticity3d.cpp [0] KSPSetUp() line 182 in /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c [0] PCSetUp() line 810 in /home/fdkong/math/petsc-3.3-p7/src/ksp/pc/interface/precon.c [0] KSPSetUp() line 182 in /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c Whether it means that I malloced some objects and then I did not free them? Or something else? Or it is normal? On Mon, Jun 3, 2013 at 9:04 PM, Jed Brown wrote: > Fande Kong writes: > > > Hi all, > > > > I can correctly run my application based on the petsc ( --with-debugging > = > > yes). But the same code encounters memory crash when it runs with > optimized > > version ( --with-debugging = no) . The detailed information for errors > and > > results is attached. The logs for configure and make are also attached. > > > > > > Now, there are several questions I want to know > > > > (1) What are differences between debug version and optimized version in > > terms of memory management and communication? > > > > (2) I print some memory usages. Why does the optimized version cost twice > > more memory than the debug version? > > It shouldn't. What interface are you using to obtain memory usage? I > suggest fixing the errors before worrying about this. > > You should figure out how to reproduce the issue on a smaller problem > size and with fewer processes. That reduction alone may tell you where > the issue occurs. > > > (3) How to debug the optimized code to fix memory issue? > > Is the debug version currently clean when run with -malloc_dump and > clean when run under valgrind? If so, I would configure > > --with-debugging=0 COPTFLAGS='-O0 -g' > > this will take the "optimized" code path, but will have full information > when run in the debugger. You'll need a smaller problem size and fewer > processes (preferably one or two) to do this effectively. > > -- Fande Kong Department of Computer Science University of Colorado at Boulder -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Jun 3 08:56:27 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 03 Jun 2013 08:56:27 -0500 Subject: [petsc-users] memory crash with optimized petsc In-Reply-To: References: <877gibs5ux.fsf@mcs.anl.gov> Message-ID: <87wqqbqovo.fsf@mcs.anl.gov> Fande Kong writes: > Thanks Jed? > > I run code with options: -malloc_debug and -malloc_dump and get the > following results: > > [0]Total space allocated 217984 bytes > [ 0]16 bytes PetscStrallocpy() line 186 in > /home/fdkong/math/petsc-3.3-p7/src/sys/utils/str.c > [0] PetscObjectChangeTypeName() line 129 in > /home/fdkong/math/petsc-3.3-p7/src/sys/objects/pname.c > [0] ISSetType() line 77 in > /home/fdkong/math/petsc-3.3-p7/src/vec/is/interface/isreg.c > [0] ISCreateGeneral() line 437 in > /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c > [0] SpmcsDMMeshGetStratumIS() line 1375 in spmcsdmmesh.cpp > [0] ComputeMatrix() line 279 in linearElasticity3d.cpp > [0] KSPSetUp() line 182 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > [0] PCSetUp() line 810 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/pc/interface/precon.c > [0] KSPSetUp() line 182 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > [0] KSPSetFromOptions() line 284 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itcl.c > [ 0]32 bytes ISCreate_General() line 516 in > /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c > [0] ISSetType() line 77 in > /home/fdkong/math/petsc-3.3-p7/src/vec/is/interface/isreg.c > [0] ISCreateGeneral() line 437 in > /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c > [0] SpmcsDMMeshGetStratumIS() line 1375 in spmcsdmmesh.cpp > [0] ComputeMatrix() line 279 in linearElasticity3d.cpp > [0] KSPSetUp() line 182 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > [0] PCSetUp() line 810 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/pc/interface/precon.c > [0] KSPSetUp() line 182 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > [0] KSPSetFromOptions() line 284 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itcl.c > [ 0]128 bytes ISCreate() line 43 in > /home/fdkong/math/petsc-3.3-p7/src/vec/is/interface/isreg.c > [0] ISCreateGeneral() line 437 in > /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c > [0] SpmcsDMMeshGetStratumIS() line 1375 in spmcsdmmesh.cpp > [0] ComputeMatrix() line 279 in linearElasticity3d.cpp > [0] KSPSetUp() line 182 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > [0] PCSetUp() line 810 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/pc/interface/precon.c > [0] KSPSetUp() line 182 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > [0] KSPSetFromOptions() line 284 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itcl.c > [ 0]64 bytes ISCreate() line 43 in > /home/fdkong/math/petsc-3.3-p7/src/vec/is/interface/isreg.c > [0] ISCreateGeneral() line 437 in > /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c > [0] SpmcsDMMeshGetStratumIS() line 1375 in spmcsdmmesh.cpp > [0] ComputeMatrix() line 279 in linearElasticity3d.cpp > [0] KSPSetUp() line 182 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > [0] PCSetUp() line 810 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/pc/interface/precon.c > [0] KSPSetUp() line 182 in > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > > Whether it means that I malloced some objects and then I did not free them? > Or something else? Or it is normal? Is this the complete output or was there more? This indicates that an index set (IS) was not freed. Note that index sets are reference-counted and normally referenced from a few places. From fande.kong at colorado.edu Mon Jun 3 09:06:43 2013 From: fande.kong at colorado.edu (Fande Kong) Date: Mon, 3 Jun 2013 22:06:43 +0800 Subject: [petsc-users] memory crash with optimized petsc In-Reply-To: <87wqqbqovo.fsf@mcs.anl.gov> References: <877gibs5ux.fsf@mcs.anl.gov> <87wqqbqovo.fsf@mcs.anl.gov> Message-ID: This was not complete. The complete result is attached. On Mon, Jun 3, 2013 at 9:56 PM, Jed Brown wrote: > Fande Kong writes: > > > Thanks Jed? > > > > I run code with options: -malloc_debug and -malloc_dump and get the > > following results: > > > > [0]Total space allocated 217984 bytes > > [ 0]16 bytes PetscStrallocpy() line 186 in > > /home/fdkong/math/petsc-3.3-p7/src/sys/utils/str.c > > [0] PetscObjectChangeTypeName() line 129 in > > /home/fdkong/math/petsc-3.3-p7/src/sys/objects/pname.c > > [0] ISSetType() line 77 in > > /home/fdkong/math/petsc-3.3-p7/src/vec/is/interface/isreg.c > > [0] ISCreateGeneral() line 437 in > > /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c > > [0] SpmcsDMMeshGetStratumIS() line 1375 in spmcsdmmesh.cpp > > [0] ComputeMatrix() line 279 in linearElasticity3d.cpp > > [0] KSPSetUp() line 182 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > > [0] PCSetUp() line 810 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/pc/interface/precon.c > > [0] KSPSetUp() line 182 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > > [0] KSPSetFromOptions() line 284 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itcl.c > > [ 0]32 bytes ISCreate_General() line 516 in > > /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c > > [0] ISSetType() line 77 in > > /home/fdkong/math/petsc-3.3-p7/src/vec/is/interface/isreg.c > > [0] ISCreateGeneral() line 437 in > > /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c > > [0] SpmcsDMMeshGetStratumIS() line 1375 in spmcsdmmesh.cpp > > [0] ComputeMatrix() line 279 in linearElasticity3d.cpp > > [0] KSPSetUp() line 182 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > > [0] PCSetUp() line 810 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/pc/interface/precon.c > > [0] KSPSetUp() line 182 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > > [0] KSPSetFromOptions() line 284 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itcl.c > > [ 0]128 bytes ISCreate() line 43 in > > /home/fdkong/math/petsc-3.3-p7/src/vec/is/interface/isreg.c > > [0] ISCreateGeneral() line 437 in > > /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c > > [0] SpmcsDMMeshGetStratumIS() line 1375 in spmcsdmmesh.cpp > > [0] ComputeMatrix() line 279 in linearElasticity3d.cpp > > [0] KSPSetUp() line 182 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > > [0] PCSetUp() line 810 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/pc/interface/precon.c > > [0] KSPSetUp() line 182 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > > [0] KSPSetFromOptions() line 284 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itcl.c > > [ 0]64 bytes ISCreate() line 43 in > > /home/fdkong/math/petsc-3.3-p7/src/vec/is/interface/isreg.c > > [0] ISCreateGeneral() line 437 in > > /home/fdkong/math/petsc-3.3-p7/src/vec/is/impls/general/general.c > > [0] SpmcsDMMeshGetStratumIS() line 1375 in spmcsdmmesh.cpp > > [0] ComputeMatrix() line 279 in linearElasticity3d.cpp > > [0] KSPSetUp() line 182 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > > [0] PCSetUp() line 810 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/pc/interface/precon.c > > [0] KSPSetUp() line 182 in > > /home/fdkong/math/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c > > > > Whether it means that I malloced some objects and then I did not free > them? > > Or something else? Or it is normal? > > Is this the complete output or was there more? This indicates that an > index set (IS) was not freed. Note that index sets are > reference-counted and normally referenced from a few places. > > -- Fande Kong Department of Computer Science University of Colorado at Boulder -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: result Type: application/octet-stream Size: 73329 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Mon Jun 3 10:00:39 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 03 Jun 2013 10:00:39 -0500 Subject: [petsc-users] memory crash with optimized petsc In-Reply-To: References: <877gibs5ux.fsf@mcs.anl.gov> <87wqqbqovo.fsf@mcs.anl.gov> Message-ID: <87li6rqlwo.fsf@mcs.anl.gov> Fande Kong writes: > This was not complete. The complete result is attached. With petsc-3.4, you can use PetscFunctionBeginUser and -objects_dump to eliminate internal allocations from the output. I'm guessing you missed a KSPDestroy. From ztdepyahoo at 163.com Mon Jun 3 22:51:47 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Tue, 4 Jun 2013 11:51:47 +0800 (CST) Subject: [petsc-users] Does MatCreateMPIAjd support hybrid mesh Message-ID: <5ed4488.ef9d.13f0d4fe4d9.Coremail.ztdepyahoo@163.com> Does MatCreateMPIAjd support hybrid mesh, i.e., the mesh is consisted by triangular cells and quadrilateral cells. thank you very much. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Jun 3 22:56:40 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 03 Jun 2013 22:56:40 -0500 Subject: [petsc-users] Does MatCreateMPIAjd support hybrid mesh In-Reply-To: <5ed4488.ef9d.13f0d4fe4d9.Coremail.ztdepyahoo@163.com> References: <5ed4488.ef9d.13f0d4fe4d9.Coremail.ztdepyahoo@163.com> Message-ID: <87ip1uo7ev.fsf@mcs.anl.gov> ??? writes: > Does MatCreateMPIAjd support hybrid mesh, i.e., the mesh is consisted by triangular cells and quadrilateral cells. Yes, it's purely algebraic. From ztdepyahoo at 163.com Mon Jun 3 23:25:21 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Tue, 4 Jun 2013 12:25:21 +0800 (CST) Subject: [petsc-users] Does MatCreateMPIAjd support hybrid mesh In-Reply-To: <87ip1uo7ev.fsf@mcs.anl.gov> References: <5ed4488.ef9d.13f0d4fe4d9.Coremail.ztdepyahoo@163.com> <87ip1uo7ev.fsf@mcs.anl.gov> Message-ID: <1f47d0d1.1de94.13f0d6ea316.Coremail.ztdepyahoo@163.com> Does petsc has some examples about this ? 2013-06-04 11:56:40?"Jed Brown" ??? >??? writes: > >> Does MatCreateMPIAjd support hybrid mesh, i.e., the mesh is consisted by triangular cells and quadrilateral cells. > >Yes, it's purely algebraic. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fengwang85 at gmail.com Mon Jun 3 23:51:40 2013 From: fengwang85 at gmail.com (Wang Feng) Date: Tue, 4 Jun 2013 12:51:40 +0800 Subject: [petsc-users] petsc crash with intel compiler Message-ID: Hi everyone I am trying to using petsc-3.1 with intel compiler version 13.0.1. Sometimes it crash with error information. [5]PETSC ERROR: --------------------- Error Message ------------------------------------ Error in external library! Error in HYPRE solver, error code 1! Configure options --prefix=/global/u2/f/fengwang/opt/petsc-3.1-intel_hyp --with-debugging=no --with-scalar-type=real --with-hypre=1 --known-mpi-shared=1 --CC=cc --CXX=CC --FC=ftn --with-x11=0 --download-superlu=download/superlu_4.3.tar.gz --with-hypre-dir=/global/u2/f/fengwang/opt/hypre-2.7.0 --with-blas-lapack-lib=-mkl While in pgi compiler, it works well. Do you have any suggestions about it? Thanks so much. -Feng -- ?????????????? Feng Wang School of Physics and Optoelctronic Technology, Dalian University of Technology Princeton Plasma Physics Lab. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 4 01:02:46 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 4 Jun 2013 02:02:46 -0400 Subject: [petsc-users] Does MatCreateMPIAjd support hybrid mesh In-Reply-To: <1f47d0d1.1de94.13f0d6ea316.Coremail.ztdepyahoo@163.com> References: <5ed4488.ef9d.13f0d4fe4d9.Coremail.ztdepyahoo@163.com> <87ip1uo7ev.fsf@mcs.anl.gov> <1f47d0d1.1de94.13f0d6ea316.Coremail.ztdepyahoo@163.com> Message-ID: On Tue, Jun 4, 2013 at 12:25 AM, ??? wrote: > Does petsc has some examples about this > Any time you use MatPartitioning with some partitioners, e.g. Chaco, we convert to Adj, so that would be an example. Matt > ? 2013-06-04 11:56:40?"Jed Brown" ??? > >??? writes: > > > >> Does MatCreateMPIAjd support hybrid mesh, i.e., the mesh is consisted by triangular cells and quadrilateral cells. > > > >Yes, it's purely algebraic. > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 4 01:05:30 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 4 Jun 2013 02:05:30 -0400 Subject: [petsc-users] petsc crash with intel compiler In-Reply-To: References: Message-ID: On Tue, Jun 4, 2013 at 12:51 AM, Wang Feng wrote: > Hi everyone > I am trying to using petsc-3.1 with intel compiler version 13.0.1. > Sometimes it crash with error information. > 1) This does not even tell use what part of Hypre you are using. Never truncate the error message. > [5]PETSC ERROR: --------------------- Error Message > ------------------------------------ > Error in external library! > Error in HYPRE solver, error code 1! > 2) This is an error inside Hypre. I suggest contacting them. > Configure options --prefix=/global/u2/f/fengwang/opt/petsc-3.1-intel_hyp > --with-debugging=no --with-scalar-type=real --with-hypre=1 > --known-mpi-shared=1 --CC=cc --CXX=CC --FC=ftn --with-x11=0 > --download-superlu=download/superlu_4.3.tar.gz > --with-hypre-dir=/global/u2/f/fengwang/opt/hypre-2.7.0 > --with-blas-lapack-lib=-mkl > > While in pgi compiler, it works well. > Do you have any suggestions about it? > 3) Even if we could track this down, it would be using the latest PETSc release 3.4 Thanks, Matt > Thanks so much. > > -Feng > > > > > > > > -- > ?????????????? > Feng Wang > School of Physics and Optoelctronic Technology, > Dalian University of Technology > Princeton Plasma Physics Lab. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From choi240 at purdue.edu Tue Jun 4 01:56:32 2013 From: choi240 at purdue.edu (Joon hee Choi) Date: Tue, 4 Jun 2013 02:56:32 -0400 (EDT) Subject: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) In-Reply-To: <314095689.8702.1370324267995.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <2006642416.8728.1370328992216.JavaMail.root@mailhub028.itcs.purdue.edu> Hello, I am trying to multiply the transpose of a dense matrix(A) and a large sparse matrix(X1). That is, A^T x X1. X1 is a 26Mil x 1200Tril, 144Mil non-zeros sparse matrix and A is a 26Mil x 10 dense matrix. I know that sparse x dense is faster than dense x sparse when using MatMatMult. Thus, I tried to implement the following code: ierr = MatTranspose(X1, MAT_INITIAL_MATRIX, &tempX1); CHKERRQ(ierr); ierr = MatMatMult(tempX1, A, MAT_INITIAL_MATRIX, 1.0, &MT); ierr = MatDestroy(&tempX1); CHKERRQ(ierr); ierr = MatTranspose(MT, MAT_INITIAL_MATRIX, &M); CHKERRQ(ierr); ierr = MatDestroy(&MT); CHKERRQ(ierr); However, I got the "out-of-memory" error when implementing MatTranspose(). I think this is because the number of columns of X1 is much larger than that of rows of X1. If there is a fast way to calculate M = A^T x X1, I would appreciate your feedback on this. Please help! Thank you, Joon From fande.kong at colorado.edu Tue Jun 4 05:19:08 2013 From: fande.kong at colorado.edu (Fande Kong) Date: Tue, 4 Jun 2013 04:19:08 -0600 Subject: [petsc-users] memory crash with optimized petsc In-Reply-To: <87li6rqlwo.fsf@mcs.anl.gov> References: <877gibs5ux.fsf@mcs.anl.gov> <87wqqbqovo.fsf@mcs.anl.gov> <87li6rqlwo.fsf@mcs.anl.gov> Message-ID: When I used valgrind to check memory, I got the following output: ==3055== Conditional jump or move depends on uninitialised value(s) ==3055== at 0x6CF359E: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) ==3055== by 0x6CF2B25: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) ==3055== by 0x6C31607: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) ==3055== by 0x400F305: call_init.part.0 (dl-init.c:85) ==3055== by 0x400F3DE: _dl_init (dl-init.c:52) ==3055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so) ==3055== by 0x47: ??? ==3055== by 0x7FEFFFE56: ??? ==3055== by 0x7FEFFFE69: ??? ==3055== by 0x7FEFFFE71: ??? ==3055== by 0x7FEFFFE75: ??? ==3055== by 0x7FEFFFE88: ??? ==3055== by 0x7FEFFFE8B: ??? ==3055== by 0x7FEFFFE95: ??? ==3055== by 0x7FEFFFE9C: ??? ==3055== by 0x7FEFFFEA5: ??? ==3055== by 0x7FEFFFEA9: ??? ==3055== by 0x7FEFFFEB8: ??? ==3055== by 0x7FEFFFEBA: ??? ==3055== by 0x7FEFFFECC: ??? ==3055== ==3055== Conditional jump or move depends on uninitialised value(s) ==3055== at 0x6CF35A9: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) ==3055== by 0x6CF2B25: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) ==3055== by 0x6C31607: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) ==3055== by 0x400F305: call_init.part.0 (dl-init.c:85) ==3055== by 0x400F3DE: _dl_init (dl-init.c:52) ==3055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so) ==3055== by 0x47: ??? ==3055== by 0x7FEFFFE56: ??? ==3055== by 0x7FEFFFE69: ??? ==3055== by 0x7FEFFFE71: ??? ==3055== by 0x7FEFFFE75: ??? ==3055== by 0x7FEFFFE88: ??? ==3055== by 0x7FEFFFE8B: ??? ==3055== by 0x7FEFFFE95: ??? ==3055== by 0x7FEFFFE9C: ??? ==3055== by 0x7FEFFFEA5: ??? ==3055== by 0x7FEFFFEA9: ??? ==3055== by 0x7FEFFFEB8: ??? ==3055== by 0x7FEFFFEBA: ??? ==3055== by 0x7FEFFFECC: ??? ==3055== How to understand the output? On Mon, Jun 3, 2013 at 9:00 AM, Jed Brown wrote: > Fande Kong writes: > > > This was not complete. The complete result is attached. > > With petsc-3.4, you can use PetscFunctionBeginUser and -objects_dump to > eliminate internal allocations from the output. I'm guessing you missed > a KSPDestroy. > > -- Fande Kong Department of Computer Science University of Colorado at Boulder -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at colorado.edu Tue Jun 4 05:22:25 2013 From: fande.kong at colorado.edu (Fande Kong) Date: Tue, 4 Jun 2013 04:22:25 -0600 Subject: [petsc-users] memory crash with optimized petsc In-Reply-To: <87li6rqlwo.fsf@mcs.anl.gov> References: <877gibs5ux.fsf@mcs.anl.gov> <87wqqbqovo.fsf@mcs.anl.gov> <87li6rqlwo.fsf@mcs.anl.gov> Message-ID: When I used valgrind to check memory, I got the following output: ==3055== Conditional jump or move depends on uninitialised value(s) ==3055== at 0x6CF359E: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) ==3055== by 0x6CF2B25: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) ==3055== by 0x6C31607: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) ==3055== by 0x400F305: call_init.part.0 (dl-init.c:85) ==3055== by 0x400F3DE: _dl_init (dl-init.c:52) ==3055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so) ==3055== by 0x47: ??? ==3055== by 0x7FEFFFE56: ??? ==3055== by 0x7FEFFFE69: ??? ==3055== by 0x7FEFFFE71: ??? ==3055== by 0x7FEFFFE75: ??? ==3055== by 0x7FEFFFE88: ??? ==3055== by 0x7FEFFFE8B: ??? ==3055== by 0x7FEFFFE95: ??? ==3055== by 0x7FEFFFE9C: ??? ==3055== by 0x7FEFFFEA5: ??? ==3055== by 0x7FEFFFEA9: ??? ==3055== by 0x7FEFFFEB8: ??? ==3055== by 0x7FEFFFEBA: ??? ==3055== by 0x7FEFFFECC: ??? ==3055== ==3055== Conditional jump or move depends on uninitialised value(s) ==3055== at 0x6CF35A9: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) ==3055== by 0x6CF2B25: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) ==3055== by 0x6C31607: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) ==3055== by 0x400F305: call_init.part.0 (dl-init.c:85) ==3055== by 0x400F3DE: _dl_init (dl-init.c:52) ==3055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so) ==3055== by 0x47: ??? ==3055== by 0x7FEFFFE56: ??? ==3055== by 0x7FEFFFE69: ??? ==3055== by 0x7FEFFFE71: ??? ==3055== by 0x7FEFFFE75: ??? ==3055== by 0x7FEFFFE88: ??? ==3055== by 0x7FEFFFE8B: ??? ==3055== by 0x7FEFFFE95: ??? ==3055== by 0x7FEFFFE9C: ??? ==3055== by 0x7FEFFFEA5: ??? ==3055== by 0x7FEFFFEA9: ??? ==3055== by 0x7FEFFFEB8: ??? ==3055== by 0x7FEFFFEBA: ??? ==3055== by 0x7FEFFFECC: ??? ==3055== How to understand the output? On Mon, Jun 3, 2013 at 9:00 AM, Jed Brown wrote: > Fande Kong writes: > > > This was not complete. The complete result is attached. > > With petsc-3.4, you can use PetscFunctionBeginUser and -objects_dump to > eliminate internal allocations from the output. I'm guessing you missed > a KSPDestroy. > > -- Fande Kong Department of Computer Science University of Colorado at Boulder -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 4 05:51:12 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 4 Jun 2013 06:51:12 -0400 Subject: [petsc-users] memory crash with optimized petsc In-Reply-To: References: <877gibs5ux.fsf@mcs.anl.gov> <87wqqbqovo.fsf@mcs.anl.gov> <87li6rqlwo.fsf@mcs.anl.gov> Message-ID: On Tue, Jun 4, 2013 at 6:22 AM, Fande Kong wrote: > When I used valgrind to check memory, I got the following output: > 1) valgrind cannot give you a stack from executables without symbols 2) You should not care about things that go on in _dl_init() Matt > ==3055== Conditional jump or move depends on uninitialised value(s) > ==3055== at 0x6CF359E: ??? (in > /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) > ==3055== by 0x6CF2B25: ??? (in > /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) > ==3055== by 0x6C31607: ??? (in > /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) > ==3055== by 0x400F305: call_init.part.0 (dl-init.c:85) > ==3055== by 0x400F3DE: _dl_init (dl-init.c:52) > ==3055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so) > ==3055== by 0x47: ??? > ==3055== by 0x7FEFFFE56: ??? > ==3055== by 0x7FEFFFE69: ??? > ==3055== by 0x7FEFFFE71: ??? > ==3055== by 0x7FEFFFE75: ??? > ==3055== by 0x7FEFFFE88: ??? > ==3055== by 0x7FEFFFE8B: ??? > ==3055== by 0x7FEFFFE95: ??? > ==3055== by 0x7FEFFFE9C: ??? > ==3055== by 0x7FEFFFEA5: ??? > ==3055== by 0x7FEFFFEA9: ??? > ==3055== by 0x7FEFFFEB8: ??? > ==3055== by 0x7FEFFFEBA: ??? > ==3055== by 0x7FEFFFECC: ??? > ==3055== > ==3055== Conditional jump or move depends on uninitialised value(s) > ==3055== at 0x6CF35A9: ??? (in > /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) > ==3055== by 0x6CF2B25: ??? (in > /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) > ==3055== by 0x6C31607: ??? (in > /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) > ==3055== by 0x400F305: call_init.part.0 (dl-init.c:85) > ==3055== by 0x400F3DE: _dl_init (dl-init.c:52) > ==3055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so) > ==3055== by 0x47: ??? > ==3055== by 0x7FEFFFE56: ??? > ==3055== by 0x7FEFFFE69: ??? > ==3055== by 0x7FEFFFE71: ??? > ==3055== by 0x7FEFFFE75: ??? > ==3055== by 0x7FEFFFE88: ??? > ==3055== by 0x7FEFFFE8B: ??? > ==3055== by 0x7FEFFFE95: ??? > ==3055== by 0x7FEFFFE9C: ??? > ==3055== by 0x7FEFFFEA5: ??? > ==3055== by 0x7FEFFFEA9: ??? > ==3055== by 0x7FEFFFEB8: ??? > ==3055== by 0x7FEFFFEBA: ??? > ==3055== by 0x7FEFFFECC: ??? > ==3055== > > How to understand the output? > > > On Mon, Jun 3, 2013 at 9:00 AM, Jed Brown wrote: > >> Fande Kong writes: >> >> > This was not complete. The complete result is attached. >> >> With petsc-3.4, you can use PetscFunctionBeginUser and -objects_dump to >> eliminate internal allocations from the output. I'm guessing you missed >> a KSPDestroy. >> >> > > > -- > Fande Kong > Department of Computer Science > University of Colorado at Boulder > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 4 06:55:31 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 04 Jun 2013 06:55:31 -0500 Subject: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) In-Reply-To: <2006642416.8728.1370328992216.JavaMail.root@mailhub028.itcs.purdue.edu> References: <2006642416.8728.1370328992216.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <874ndenl8s.fsf@mcs.anl.gov> Joon hee Choi writes: > Hello, > > I am trying to multiply the transpose of a dense matrix(A) and a large sparse matrix(X1). That is, A^T x X1. > > X1 is a 26Mil x 1200Tril, 144Mil non-zeros sparse matrix and A is a 26Mil x 10 dense matrix. > > I know that sparse x dense is faster than dense x sparse when using MatMatMult. Thus, I tried to implement the following code: > > ierr = MatTranspose(X1, MAT_INITIAL_MATRIX, &tempX1); CHKERRQ(ierr); > ierr = MatMatMult(tempX1, A, MAT_INITIAL_MATRIX, 1.0, &MT); > ierr = MatDestroy(&tempX1); CHKERRQ(ierr); > ierr = MatTranspose(MT, MAT_INITIAL_MATRIX, &M); CHKERRQ(ierr); > ierr = MatDestroy(&MT); CHKERRQ(ierr); > > However, I got the "out-of-memory" error when implementing > MatTranspose(). Which MatTranspose? > I think this is because the number of columns of X1 is much larger > than that of rows of X1. If there is a fast way to calculate M = A^T > x X1, Hong, do you have time to implement MatTransposeMatMult_MPIAIJ_MPIDense? Can you create X1 as X1^T instead? If you want to keep storing X1 is you do now, you can either store it as ten vectors and use MatMultTranspose or you can pack it into one vector and use MatCreateMAIJ(X1,10,&X1m); MatMultTranspose(X1m,Apacked,Bpacked); This is actually a better ordering for memory bandwidth. The MAIJ matrix does not need extra storage. From choi240 at purdue.edu Tue Jun 4 13:15:04 2013 From: choi240 at purdue.edu (Joon hee Choi) Date: Tue, 4 Jun 2013 14:15:04 -0400 (EDT) Subject: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) In-Reply-To: <874ndenl8s.fsf@mcs.anl.gov> Message-ID: <1222994905.9404.1370369704468.JavaMail.root@mailhub028.itcs.purdue.edu> Thank you for your fast reply. I got the out-of-memory error from MatTranspose(X1, MAT_INITIAL_MATRIX, &tempX1). Also, it may take so much time to create X1 as X1^T because nnz for X1 cannot have 1200Tril elements because of memory. Is there a fast way to create X1^T? Thank you, Joon ----- Original Message ----- From: "Jed Brown" To: "Joon hee Choi" , petsc-users at mcs.anl.gov Sent: Tuesday, June 4, 2013 7:55:31 AM Subject: Re: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) Joon hee Choi writes: > Hello, > > I am trying to multiply the transpose of a dense matrix(A) and a large sparse matrix(X1). That is, A^T x X1. > > X1 is a 26Mil x 1200Tril, 144Mil non-zeros sparse matrix and A is a 26Mil x 10 dense matrix. > > I know that sparse x dense is faster than dense x sparse when using MatMatMult. Thus, I tried to implement the following code: > > ierr = MatTranspose(X1, MAT_INITIAL_MATRIX, &tempX1); CHKERRQ(ierr); > ierr = MatMatMult(tempX1, A, MAT_INITIAL_MATRIX, 1.0, &MT); > ierr = MatDestroy(&tempX1); CHKERRQ(ierr); > ierr = MatTranspose(MT, MAT_INITIAL_MATRIX, &M); CHKERRQ(ierr); > ierr = MatDestroy(&MT); CHKERRQ(ierr); > > However, I got the "out-of-memory" error when implementing > MatTranspose(). Which MatTranspose? > I think this is because the number of columns of X1 is much larger > than that of rows of X1. If there is a fast way to calculate M = A^T > x X1, Hong, do you have time to implement MatTransposeMatMult_MPIAIJ_MPIDense? Can you create X1 as X1^T instead? If you want to keep storing X1 is you do now, you can either store it as ten vectors and use MatMultTranspose or you can pack it into one vector and use MatCreateMAIJ(X1,10,&X1m); MatMultTranspose(X1m,Apacked,Bpacked); This is actually a better ordering for memory bandwidth. The MAIJ matrix does not need extra storage. From jedbrown at mcs.anl.gov Tue Jun 4 13:35:15 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 04 Jun 2013 13:35:15 -0500 Subject: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) In-Reply-To: <1222994905.9404.1370369704468.JavaMail.root@mailhub028.itcs.purdue.edu> References: <1222994905.9404.1370369704468.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <87r4ghlo64.fsf@mcs.anl.gov> Joon hee Choi writes: > Thank you for your fast reply. > I got the out-of-memory error from MatTranspose(X1, MAT_INITIAL_MATRIX, &tempX1). > Also, it may take so much time to create X1 as X1^T because nnz for X1 cannot have 1200Tril elements because of memory. Yes, that is also the dimension of the result matrix, which would be dense with the normal rules about products of sparse and dense matrices. Instead, you should compress the column space, perhaps by just removing all the columns that have no nonzeros. > Is there a fast way to create X1^T? > > Thank you, > Joon > > > ----- Original Message ----- > From: "Jed Brown" > To: "Joon hee Choi" , petsc-users at mcs.anl.gov > Sent: Tuesday, June 4, 2013 7:55:31 AM > Subject: Re: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) > > Joon hee Choi writes: > >> Hello, >> >> I am trying to multiply the transpose of a dense matrix(A) and a large sparse matrix(X1). That is, A^T x X1. >> >> X1 is a 26Mil x 1200Tril, 144Mil non-zeros sparse matrix and A is a 26Mil x 10 dense matrix. >> >> I know that sparse x dense is faster than dense x sparse when using MatMatMult. Thus, I tried to implement the following code: >> >> ierr = MatTranspose(X1, MAT_INITIAL_MATRIX, &tempX1); CHKERRQ(ierr); >> ierr = MatMatMult(tempX1, A, MAT_INITIAL_MATRIX, 1.0, &MT); >> ierr = MatDestroy(&tempX1); CHKERRQ(ierr); >> ierr = MatTranspose(MT, MAT_INITIAL_MATRIX, &M); CHKERRQ(ierr); >> ierr = MatDestroy(&MT); CHKERRQ(ierr); >> >> However, I got the "out-of-memory" error when implementing >> MatTranspose(). > > Which MatTranspose? > >> I think this is because the number of columns of X1 is much larger >> than that of rows of X1. If there is a fast way to calculate M = A^T >> x X1, > > Hong, do you have time to implement MatTransposeMatMult_MPIAIJ_MPIDense? > > Can you create X1 as X1^T instead? > > If you want to keep storing X1 is you do now, you can either store it as > ten vectors and use MatMultTranspose or you can pack it into one vector and use > > MatCreateMAIJ(X1,10,&X1m); > MatMultTranspose(X1m,Apacked,Bpacked); > > This is actually a better ordering for memory bandwidth. The MAIJ > matrix does not need extra storage. From hzhang at mcs.anl.gov Tue Jun 4 14:04:24 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 4 Jun 2013 14:04:24 -0500 Subject: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) In-Reply-To: <874ndenl8s.fsf@mcs.anl.gov> References: <2006642416.8728.1370328992216.JavaMail.root@mailhub028.itcs.purdue.edu> <874ndenl8s.fsf@mcs.anl.gov> Message-ID: Jed, > > > > I am trying to multiply the transpose of a dense matrix(A) and a large > sparse matrix(X1). That is, A^T x X1. > > > > Hong, do you have time to implement MatTransposeMatMult_MPIAIJ_MPIDense? > He needs MatTransposeMatMult_MPIDense_MPIAIJ This requires huge amount of data communications among MPI processes. Hong > > Can you create X1 as X1^T instead? > > If you want to keep storing X1 is you do now, you can either store it as > ten vectors and use MatMultTranspose or you can pack it into one vector > and use > > MatCreateMAIJ(X1,10,&X1m); > MatMultTranspose(X1m,Apacked,Bpacked); > > This is actually a better ordering for memory bandwidth. The MAIJ > matrix does not need extra storage. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shuangshuang.Jin at pnnl.gov Tue Jun 4 14:13:00 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Tue, 4 Jun 2013 12:13:00 -0700 Subject: [petsc-users] a question about DAE Function Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> Hello, I have a question regarding how to set up the IFunction for TSSetIFunction. Please see the detailed illustration below: Inside my IFunction, I have one of the f equations defined as: if (i >= xstart && i < xstart+xlen) { f[i+2] = sin(x[i])*x[i+2] + cos(x[i])*x[i+3]; for (k = 0; k < n; k++) { f[i+2] += -(cos(x[k])*a[k])*b) - (sin(x[k])*c[k])*d); (1) } } Where x is an distributed array, a and c are constant arrays, b and d are constants. n is the length of the global length of x. The problem is, since x is distributed across all the processes, each process only owns a piece of items of x, when I do (1) as written above, no matter which processor f[i+2] is on, it has to grab all the x array data from other processors, which means I have to do a scattering of x array in advance to make the x[k] value available. This is quite an inefficient way given the scattering of vector would happen in each iteration when the IFuntion is called. I am wondering if this is the only way I have to deal with this problem, or am I trapped in a wrong direction? Your comments are highly appreciated! Thanks, Shuangshuang -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 4 14:32:12 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 4 Jun 2013 15:32:12 -0400 Subject: [petsc-users] a question about DAE Function In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> Message-ID: On Tue, Jun 4, 2013 at 3:13 PM, Jin, Shuangshuang wrote: > Hello, I have a question regarding how to set up the IFunction for > TSSetIFunction. Please see the detailed illustration below: > > Inside my IFunction, I have one of the f equations defined as: > > if (i >= xstart && i < xstart+xlen) { > f[i+2] = sin(x[i])*x[i+2] + cos(x[i])*x[i+3]; > 1) You should not be setting f[i+2]. You always set the residual local to your process. > for (k = 0; k < n; k++) { > f[i+2] += -(cos(x[k])*a[k])*b) - > (sin(x[k])*c[k])*d); (1) > } > } > 2) This in effect says that your Jacobian is completely dense, or that your problem is fully coupled. I see two choices: a) Use scalable dense linear algebra, like Elemental, and formulate your operations as linear algebra b) If this is a boundary integral equation, consider using the FMM technology Either way, this is not a PDE, and so using that PETSc parallel paradigm is wrong. Matt > Where x is an distributed array, a and c are constant arrays, b and d are > constants. n is the length of the global length of x. > > The problem is, since x is distributed across all the processes, each > process only owns a piece of items of x, when I do (1) as written above, no > matter which processor f[i+2] is on, it has to grab all the x array data > from other processors, which means I have to do a scattering of x array in > advance to make the x[k] value available. This is quite an inefficient way > given the scattering of vector would happen in each iteration when the > IFuntion is called. > > I am wondering if this is the only way I have to deal with this problem, > or am I trapped in a wrong direction? > > Your comments are highly appreciated! > > Thanks, > Shuangshuang > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 4 14:34:54 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 4 Jun 2013 14:34:54 -0500 Subject: [petsc-users] a question about DAE Function In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> Message-ID: <0285831B-1DF6-4647-9816-DB49E990D0D5@mcs.anl.gov> On Jun 4, 2013, at 2:13 PM, "Jin, Shuangshuang" wrote: > Hello, I have a question regarding how to set up the IFunction for TSSetIFunction. Please see the detailed illustration below: > > Inside my IFunction, I have one of the f equations defined as: > > if (i >= xstart && i < xstart+xlen) { > f[i+2] = sin(x[i])*x[i+2] + cos(x[i])*x[i+3]; > for (k = 0; k < n; k++) { > f[i+2] += -(cos(x[k])*a[k])*b) - (sin(x[k])*c[k])*d); (1) > } > } > > Where x is an distributed array, a and c are constant arrays, b and d are constants. n is the length of the global length of x. > > The problem is, since x is distributed across all the processes, each process only owns a piece of items of x, when I do (1) as written above, no matter which processor f[i+2] is on, it has to grab all the x array data from other processors, which means I have to do a scattering of x array in advance to make the x[k] value available. This is quite an inefficient way given the scattering of vector would happen in each iteration when the IFuntion is called. If f[i] is dependent on all the x[i] then you have no choice but to have all the x[] on each process. Yes that requires a good chunk of communication but that is life. Note that the Jacobian will be dense and the linear solves with the dense Jacobian will dominate the time anyways. PETSc has an interface with Elemental to do the dense matrix solves. Barry > > I am wondering if this is the only way I have to deal with this problem, or am I trapped in a wrong direction? > > Your comments are highly appreciated! > > Thanks, > Shuangshuang > > From jedbrown at mcs.anl.gov Tue Jun 4 15:14:28 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 04 Jun 2013 15:14:28 -0500 Subject: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) In-Reply-To: References: <2006642416.8728.1370328992216.JavaMail.root@mailhub028.itcs.purdue.edu> <874ndenl8s.fsf@mcs.anl.gov> Message-ID: <87obblljkr.fsf@mcs.anl.gov> Hong Zhang writes: > He needs MatTransposeMatMult_MPIDense_MPIAIJ > This requires huge amount of data communications among MPI processes. In the original form, the result would have ten rows and 1400 Trillion columns, which isn't something reasonably storable in a MPIAIJ matrix. So the column space needs to be compressed, but then it still makes more sense to store the result matrix in transposed form (as ten columns) since those can be reasonably distributed. My MAIJ suggestion would work that way, as would MatTransposeMatMult_MPIAIJ_MPIDense. From Shuangshuang.Jin at pnnl.gov Tue Jun 4 15:24:18 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Tue, 4 Jun 2013 13:24:18 -0700 Subject: [petsc-users] a question about DAE Function In-Reply-To: References: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9FA@EMAIL04.pnl.gov> From: Matthew Knepley [mailto:knepley at gmail.com] Sent: Tuesday, June 04, 2013 12:32 PM To: Jin, Shuangshuang Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] a question about DAE Function On Tue, Jun 4, 2013 at 3:13 PM, Jin, Shuangshuang > wrote: Hello, I have a question regarding how to set up the IFunction for TSSetIFunction. Please see the detailed illustration below: Inside my IFunction, I have one of the f equations defined as: if (i >= xstart && i < xstart+xlen) { f[i+2] = sin(x[i])*x[i+2] + cos(x[i])*x[i+3]; 1) You should not be setting f[i+2]. You always set the residual local to your process. [Jin, Shuangshuang] Sorry, I typed it wrong, the condition is "if (i+2 >= xstart && i+2 < xstart+xlen) {" instead. for (k = 0; k < n; k++) { f[i+2] += -(cos(x[k])*a[k])*b) - (sin(x[k])*c[k])*d); (1) } } 2) This in effect says that your Jacobian is completely dense, or that your problem is fully coupled. I see two choices: [Jin, Shuangshuang] I have four sets of f functions: f[i], f[i+1], f[i+2], and f[i+3]. (i=1..n), the Jacobian matrix for the whole DAE problem is sparse, but the problem is fully coupled. a) Use scalable dense linear algebra, like Elemental, and formulate your operations as linear algebra [Jin, Shuangshuang] Do you have a sample source code on the web for this? b) If this is a boundary integral equation, consider using the FMM technology [Jin, Shuangshuang] It's not. Either way, this is not a PDE, and so using that PETSc parallel paradigm is wrong. [Jin, Shuangshuang] This is a DAE, I set up the section 6.1.2 Solving Differential Algebraic Equations of the Manual, form IFUNCITON, IJacobain, and set up initial values for x. If I scatter x every time inside IFUNCTION, the problem is solved in parallel when I use multiple threads. I just felt that my way of scattering x is inefficient. I don't understand why using the PETSc parallel paradigm to solve the problem is wrong? Thanks, Shuangshuang Matt Where x is an distributed array, a and c are constant arrays, b and d are constants. n is the length of the global length of x. The problem is, since x is distributed across all the processes, each process only owns a piece of items of x, when I do (1) as written above, no matter which processor f[i+2] is on, it has to grab all the x array data from other processors, which means I have to do a scattering of x array in advance to make the x[k] value available. This is quite an inefficient way given the scattering of vector would happen in each iteration when the IFuntion is called. I am wondering if this is the only way I have to deal with this problem, or am I trapped in a wrong direction? Your comments are highly appreciated! Thanks, Shuangshuang -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 4 15:42:19 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 04 Jun 2013 15:42:19 -0500 Subject: [petsc-users] a question about DAE Function In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9FA@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9FA@EMAIL04.pnl.gov> Message-ID: <87d2s1liac.fsf@mcs.anl.gov> Your message quoting is very peculiar. Please use normal quoting in the future. "Jin, Shuangshuang" writes: > From: Matthew Knepley [mailto:knepley at gmail.com] > Sent: Tuesday, June 04, 2013 12:32 PM > To: Jin, Shuangshuang > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a question about DAE Function > > On Tue, Jun 4, 2013 at 3:13 PM, Jin, Shuangshuang > wrote: > Hello, I have a question regarding how to set up the IFunction for TSSetIFunction. Please see the detailed illustration below: > > Inside my IFunction, I have one of the f equations defined as: > > if (i >= xstart && i < xstart+xlen) { > f[i+2] = sin(x[i])*x[i+2] + cos(x[i])*x[i+3]; > > > 1) You should not be setting f[i+2]. You always set the residual local to your process. > [Jin, Shuangshuang] Sorry, I typed it wrong, the condition is "if (i+2 >= xstart && i+2 < xstart+xlen) {" instead. > > for (k = 0; k < n; k++) { > f[i+2] += -(cos(x[k])*a[k])*b) - (sin(x[k])*c[k])*d); (1) > } > } Showing something of the outer loop would have made the usage more clear. > > 2) This in effect says that your Jacobian is completely dense, or that your problem is fully coupled. I see two > choices: > [Jin, Shuangshuang] I have four sets of f functions: f[i], f[i+1], > f[i+2], and f[i+3]. (i=1..n), the Jacobian matrix for the whole > DAE problem is sparse, Your code above couples f[i+2] to x[k] for k=1..n, which is dense. > but the problem is fully coupled. "fully coupled" means "dense", unless you intended something different. What exactly is sparse? > a) Use scalable dense linear algebra, like Elemental, and formulate your operations as linear algebra > [Jin, Shuangshuang] Do you have a sample source code on the web for this? There are a couple tutorials using MatElemental, but we don't seem to agree about whether the problem formulation is correct, so don't do that yet. > b) If this is a boundary integral equation, consider using the FMM technology > [Jin, Shuangshuang] It's not. > > Either way, this is not a PDE, and so using that PETSc parallel paradigm is wrong. > [Jin, Shuangshuang] This is a DAE, I set up the section 6.1.2 Solving Differential Algebraic Equations of the Manual, form IFUNCITON, IJacobain, and set up initial values for x. If I scatter x every time inside IFUNCTION, the problem is solved in parallel when I use multiple threads. I just felt that my way of scattering x is inefficient. I don't understand why using the PETSc parallel paradigm to solve the problem is wrong? If your function is dense, then you need to gather all of x to compute a residual. That is what your code above implies, but you also said it was sparse, so which is it? From hzhang at mcs.anl.gov Tue Jun 4 16:49:38 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 4 Jun 2013 16:49:38 -0500 Subject: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) In-Reply-To: <87obblljkr.fsf@mcs.anl.gov> References: <2006642416.8728.1370328992216.JavaMail.root@mailhub028.itcs.purdue.edu> <874ndenl8s.fsf@mcs.anl.gov> <87obblljkr.fsf@mcs.anl.gov> Message-ID: Jed, Let us discuss it tomorrow in the lab. Hong On Tue, Jun 4, 2013 at 3:14 PM, Jed Brown wrote: > Hong Zhang writes: > > > He needs MatTransposeMatMult_MPIDense_MPIAIJ > > This requires huge amount of data communications among MPI processes. > > In the original form, the result would have ten rows and 1400 Trillion > columns, which isn't something reasonably storable in a MPIAIJ matrix. > So the column space needs to be compressed, but then it still makes more > sense to store the result matrix in transposed form (as ten columns) > since those can be reasonably distributed. My MAIJ suggestion would > work that way, as would MatTransposeMatMult_MPIAIJ_MPIDense. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shuangshuang.Jin at pnnl.gov Tue Jun 4 17:43:17 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Tue, 4 Jun 2013 15:43:17 -0700 Subject: [petsc-users] a question about DAE Function In-Reply-To: <87d2s1liac.fsf@mcs.anl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9FA@EMAIL04.pnl.gov> <87d2s1liac.fsf@mcs.anl.gov> Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FD87AEA8E@EMAIL04.pnl.gov> Sorry about the confusion. Please allow me to illustrate the problem in a better way below: I have a set of DAE equations, which can be defined as 4*n f[] equations: scatterMyVec(X, &x); CHKERRQ(ierr); // Scatter x to each processor // (0) ierr = DMDAVecGetArray(da, Xdot, &xdot); CHKERRQ(ierr); ierr = DMDAVecGetArray(da, F, &f); CHKERRQ(ierr); // Get local grid boundaries ierr = DMDAGetCorners(da, &xstart, NULL, NULL, &xlen, NULL, NULL); CHKERRQ(ierr); // Set up DAE equations // Compute function over the locally owned part of the grid for (i = 0; i < n; i++) { if (2*i >= xstart && 2*i < xstart+xlen) { f[2*i] = xdot[2*i] - x[2*i+1]; // (1) } if (2*i+1 >= xstart && 2*i+1 < xstart+xlen) { f[2*i+1] = xdot[2*i+1] - x[2*n+2*i+1] - x[2*i+1]); // (2) } } for (i = 0; i < n; i++) { if (2*n+2*i >= xstart && 2*n+2*i < xstart+xlen) { f[2*n+2*i] = sin(x[2*i])*x[2*n+2*i] + cos(x[2*i])*x[2*n+2*i+1]; for (k = 0; k < n; k++) { f[2*n+2*i] += -(cos(x[2*k])*a-sin(x[2*k])*b); // (3) } } if (2*n+2*i+1 >= xstart && 2*n+2*i+1 < xstart+xlen) { f[2*n+2*i+1] = -cos(x[2*i])*x[2*n+2*i] + sin(x[2*i])*x[2*n+2*i+1]; for (k = 0; k < n; k++) { f[2*n+2*i+1] += -(cos(x[2*k])*c+(sin(x[2*k])*d); // (4) } } } As you may see, all the f functions are dependent on some x values. However, the x is a distributed array of size 4*n. In order to use the access x[] from different processor, I have to scatter x before I use them to define f[], the scatterMyVec function is shown below. // Scatter the context of v to each processor to get a global array (vArray) values static PetscErrorCode scatterMyVec(Vec vin, PetscScalar **voutArray) { PetscErrorCode ierr; VecScatter ctx; Vec vout; ierr = VecScatterCreateToAll(vin, &ctx, &vout); CHKERRQ(ierr); ierr = VecScatterBegin(ctx, vin, vout, INSERT_VALUES, SCATTER_FORWARD); CHKERRQ(ierr); ierr = VecScatterEnd(ctx, vin, vout, INSERT_VALUES, SCATTER_FORWARD); CHKERRQ(ierr); ierr = VecGetArray(vout, voutArray); CHKERRQ(ierr); ierr = VecScatterDestroy(&ctx); CHKERRQ(ierr); ierr = VecRestoreArray(vout, PETSC_NULL); PetscFunctionReturn(0); } As I mentioned in the previous email, I think this is quite inefficient because the scattering of x during each iteration "requires a good chunk of communication". And as Barry pointed out, " the Jacobian will be dense and the linear solves with the dense Jacobian will dominate the time anyways. " So my question now is: 1. Am I right to use the PETSc TS parallel paradigm in this way to solve my DAE equations? My concern is although I can see the solution be computed across multiple processors, the overhead of scattering x from time to time and the linear solving with dense Jacobian will eat up the performance gains of parallel computation. 2. If the answer of the above question is "NO", what's the correct parallel tool I should look into in PETSc to solve my DAE equations? MatElemental as most of you suggested? Thanks a lot for your help! Regards, Shuangshuang -----Original Message----- From: Jed Brown [mailto:five9a2 at gmail.com] On Behalf Of Jed Brown Sent: Tuesday, June 04, 2013 1:42 PM To: Jin, Shuangshuang; Matthew Knepley Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] a question about DAE Function Your message quoting is very peculiar. Please use normal quoting in the future. "Jin, Shuangshuang" writes: > From: Matthew Knepley [mailto:knepley at gmail.com] > Sent: Tuesday, June 04, 2013 12:32 PM > To: Jin, Shuangshuang > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] a question about DAE Function > > On Tue, Jun 4, 2013 at 3:13 PM, Jin, Shuangshuang > wrote: > Hello, I have a question regarding how to set up the IFunction for TSSetIFunction. Please see the detailed illustration below: > > Inside my IFunction, I have one of the f equations defined as: > > if (i >= xstart && i < xstart+xlen) { > f[i+2] = sin(x[i])*x[i+2] + cos(x[i])*x[i+3]; > > > 1) You should not be setting f[i+2]. You always set the residual local to your process. > [Jin, Shuangshuang] Sorry, I typed it wrong, the condition is "if (i+2 >= xstart && i+2 < xstart+xlen) {" instead. > > for (k = 0; k < n; k++) { > f[i+2] += -(cos(x[k])*a[k])*b) - (sin(x[k])*c[k])*d); (1) > } > } Showing something of the outer loop would have made the usage more clear. > > 2) This in effect says that your Jacobian is completely dense, or that your problem is fully coupled. I see two > choices: > [Jin, Shuangshuang] I have four sets of f functions: f[i], f[i+1], > f[i+2], and f[i+3]. (i=1..n), the Jacobian matrix for the whole > DAE problem is sparse, Your code above couples f[i+2] to x[k] for k=1..n, which is dense. > but the problem is fully coupled. "fully coupled" means "dense", unless you intended something different. What exactly is sparse? > a) Use scalable dense linear algebra, like Elemental, and formulate > your operations as linear algebra [Jin, Shuangshuang] Do you have a sample source code on the web for this? There are a couple tutorials using MatElemental, but we don't seem to agree about whether the problem formulation is correct, so don't do that yet. > b) If this is a boundary integral equation, consider using the FMM > technology [Jin, Shuangshuang] It's not. > > Either way, this is not a PDE, and so using that PETSc parallel paradigm is wrong. > [Jin, Shuangshuang] This is a DAE, I set up the section 6.1.2 Solving Differential Algebraic Equations of the Manual, form IFUNCITON, IJacobain, and set up initial values for x. If I scatter x every time inside IFUNCTION, the problem is solved in parallel when I use multiple threads. I just felt that my way of scattering x is inefficient. I don't understand why using the PETSc parallel paradigm to solve the problem is wrong? If your function is dense, then you need to gather all of x to compute a residual. That is what your code above implies, but you also said it was sparse, so which is it? From choi240 at purdue.edu Tue Jun 4 18:15:38 2013 From: choi240 at purdue.edu (Joon hee Choi) Date: Tue, 4 Jun 2013 19:15:38 -0400 (EDT) Subject: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) In-Reply-To: <87r4ghlo64.fsf@mcs.anl.gov> Message-ID: <1497296706.9782.1370387738058.JavaMail.root@mailhub028.itcs.purdue.edu> I am trying to compress the matrix. I think it is the best way, because it may be not storable by MPI aij. If you get better solution in your lab meeting, then please let me know. :) Anyhow, thank you very much. Joon ----- Original Message ----- From: "Jed Brown" To: "Joon hee Choi" Cc: petsc-users at mcs.anl.gov Sent: Tuesday, June 4, 2013 2:35:15 PM Subject: Re: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) Joon hee Choi writes: > Thank you for your fast reply. > I got the out-of-memory error from MatTranspose(X1, MAT_INITIAL_MATRIX, &tempX1). > Also, it may take so much time to create X1 as X1^T because nnz for X1 cannot have 1200Tril elements because of memory. Yes, that is also the dimension of the result matrix, which would be dense with the normal rules about products of sparse and dense matrices. Instead, you should compress the column space, perhaps by just removing all the columns that have no nonzeros. > Is there a fast way to create X1^T? > > Thank you, > Joon > > > ----- Original Message ----- > From: "Jed Brown" > To: "Joon hee Choi" , petsc-users at mcs.anl.gov > Sent: Tuesday, June 4, 2013 7:55:31 AM > Subject: Re: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) > > Joon hee Choi writes: > >> Hello, >> >> I am trying to multiply the transpose of a dense matrix(A) and a large sparse matrix(X1). That is, A^T x X1. >> >> X1 is a 26Mil x 1200Tril, 144Mil non-zeros sparse matrix and A is a 26Mil x 10 dense matrix. >> >> I know that sparse x dense is faster than dense x sparse when using MatMatMult. Thus, I tried to implement the following code: >> >> ierr = MatTranspose(X1, MAT_INITIAL_MATRIX, &tempX1); CHKERRQ(ierr); >> ierr = MatMatMult(tempX1, A, MAT_INITIAL_MATRIX, 1.0, &MT); >> ierr = MatDestroy(&tempX1); CHKERRQ(ierr); >> ierr = MatTranspose(MT, MAT_INITIAL_MATRIX, &M); CHKERRQ(ierr); >> ierr = MatDestroy(&MT); CHKERRQ(ierr); >> >> However, I got the "out-of-memory" error when implementing >> MatTranspose(). > > Which MatTranspose? > >> I think this is because the number of columns of X1 is much larger >> than that of rows of X1. If there is a fast way to calculate M = A^T >> x X1, > > Hong, do you have time to implement MatTransposeMatMult_MPIAIJ_MPIDense? > > Can you create X1 as X1^T instead? > > If you want to keep storing X1 is you do now, you can either store it as > ten vectors and use MatMultTranspose or you can pack it into one vector and use > > MatCreateMAIJ(X1,10,&X1m); > MatMultTranspose(X1m,Apacked,Bpacked); > > This is actually a better ordering for memory bandwidth. The MAIJ > matrix does not need extra storage. From jedbrown at mcs.anl.gov Tue Jun 4 18:27:35 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 04 Jun 2013 18:27:35 -0500 Subject: [petsc-users] a question about DAE Function In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FD87AEA8E@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9FA@EMAIL04.pnl.gov> <87d2s1liac.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEA8E@EMAIL04.pnl.gov> Message-ID: <8738sxjw2g.fsf@mcs.anl.gov> "Jin, Shuangshuang" writes: > Sorry about the confusion. Please allow me to illustrate the problem in a better way below: > > I have a set of DAE equations, which can be defined as 4*n f[] equations: > [...] > As you may see, all the f functions are dependent on some x > values. However, the x is a distributed array of size 4*n. Thanks, this problem is dense. > In order to use the access x[] from different processor, I have to > scatter x before I use them to define f[], the scatterMyVec function > is shown below. > > // Scatter the context of v to each processor to get a global array (vArray) values > static PetscErrorCode scatterMyVec(Vec vin, PetscScalar **voutArray) > { > PetscErrorCode ierr; > VecScatter ctx; > Vec vout; > > ierr = VecScatterCreateToAll(vin, &ctx, &vout); CHKERRQ(ierr); > ierr = VecScatterBegin(ctx, vin, vout, INSERT_VALUES, SCATTER_FORWARD); CHKERRQ(ierr); > ierr = VecScatterEnd(ctx, vin, vout, INSERT_VALUES, SCATTER_FORWARD); CHKERRQ(ierr); > ierr = VecGetArray(vout, voutArray); CHKERRQ(ierr); > > ierr = VecScatterDestroy(&ctx); CHKERRQ(ierr); Note that you can reuse the VecScatter. > ierr = VecRestoreArray(vout, PETSC_NULL); Please don't abuse VecRestoreArray to lie to PETSc. Yes, it will "work" with normal PETSc Vecs, but not in general (e.g. GPU-backed Vecs). > > PetscFunctionReturn(0); > } > > As I mentioned in the previous email, I think this is quite > inefficient because the scattering of x during each iteration > "requires a good chunk of communication". And as Barry pointed out, " > the Jacobian will be dense and the linear solves with the dense > Jacobian will dominate the time anyways. " > > So my question now is: 1. Am I right to use the PETSc TS parallel > paradigm in this way to solve my DAE equations? My concern is although > I can see the solution be computed across multiple processors, the > overhead of scattering x from time to time and the linear solving with > dense Jacobian will eat up the performance gains of parallel > computation. The data movement is O(N) while the computation is O(N^3), so parallelism will be useful if N is large enough. > 2. If the answer of the above question is "NO", what's the correct > parallel tool I should look into in PETSc to solve my DAE equations? > MatElemental as most of you suggested? TS with MatElemental is right. Here are some tests that use MatElemental. You should use MatGetOwnershipIS() to determine which matrix entries you are responsible for computing. You can use your current code for IFunction evaluation. src/ksp/ksp/examples/tests/ex40.c src/mat/examples/tests/ex104.c src/mat/examples/tests/ex145.c src/mat/examples/tests/ex38.c src/mat/examples/tests/ex39.c From sonyablade2010 at hotmail.com Wed Jun 5 00:52:58 2013 From: sonyablade2010 at hotmail.com (Sonya Blade) Date: Wed, 5 Jun 2013 06:52:58 +0100 Subject: [petsc-users] Slepc doesn't converge for some EV problems Message-ID: Dear All, Sometimes I experience that Slepc is not converging for the requested? eigenvalues and simply resturns 0. I do take into consideration of setting the requested eigenvalues from command line options rather then setting them? manually at the body of program.? I observe this when I'm trying to solve this moderately large matrices e.g. 192x192? for small ones e.g. 48x48 there is not problem other than Slepc deviates from? requested to found EVs. For example if I request 7 eigenvalues to be found Slepc finds? 9 or 10, what could be the problem of this malfunctioning? Your help will be appreciated, Regards, From jroman at dsic.upv.es Wed Jun 5 01:33:31 2013 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 5 Jun 2013 08:33:31 +0200 Subject: [petsc-users] Slepc doesn't converge for some EV problems In-Reply-To: References: Message-ID: <1655642C-7C7D-4FFF-9860-629C2F0BAA88@dsic.upv.es> El 05/06/2013, a las 07:52, Sonya Blade escribi?: > Dear All, > > Sometimes I experience that Slepc is not converging for the requested > eigenvalues and simply resturns 0. I do take into consideration of setting > the requested eigenvalues from command line options rather then setting them > manually at the body of program. > > I observe this when I'm trying to solve this moderately large matrices e.g. 192x192 > for small ones e.g. 48x48 there is not problem other than Slepc deviates from > requested to found EVs. For example if I request 7 eigenvalues to be found Slepc finds > 9 or 10, what could be the problem of this malfunctioning? > > Your help will be appreciated, > > Regards, 192 is very small. SLEPc is intended for large sizes (10,000 or more). For such small matrices, you would rather use LAPACK. It is normal that the solver returns more eigenvalues than requested. This is not a malfunctioning. If you get zero converged eigenvalues, try changing the parameters (e.g. ncv) or using a different strategy (e.g. shift-and-invert with a reasonable target). Jose From sonyablade2010 at hotmail.com Wed Jun 5 04:50:38 2013 From: sonyablade2010 at hotmail.com (Sonya Blade) Date: Wed, 5 Jun 2013 10:50:38 +0100 Subject: [petsc-users] Slepc doesn't converge for some EV problems In-Reply-To: <1655642C-7C7D-4FFF-9860-629C2F0BAA88@dsic.upv.es> References: , <1655642C-7C7D-4FFF-9860-629C2F0BAA88@dsic.upv.es> Message-ID: Thank you very much. Regards, ---------------------------------------- > Subject: Re: [petsc-users] Slepc doesn't converge for some EV problems > From: jroman at dsic.upv.es > Date: Wed, 5 Jun 2013 08:33:31 +0200 > CC: petsc-users at mcs.anl.gov > To: sonyablade2010 at hotmail.com > > > El 05/06/2013, a las 07:52, Sonya Blade escribi?: > >> Dear All, >> >> Sometimes I experience that Slepc is not converging for the requested >> eigenvalues and simply resturns 0. I do take into consideration of setting >> the requested eigenvalues from command line options rather then setting them >> manually at the body of program. >> >> I observe this when I'm trying to solve this moderately large matrices e.g. 192x192 >> for small ones e.g. 48x48 there is not problem other than Slepc deviates from >> requested to found EVs. For example if I request 7 eigenvalues to be found Slepc finds >> 9 or 10, what could be the problem of this malfunctioning? >> >> Your help will be appreciated, >> >> Regards, > > 192 is very small. SLEPc is intended for large sizes (10,000 or more). For such small matrices, you would rather use LAPACK. > > It is normal that the solver returns more eigenvalues than requested. This is not a malfunctioning. > > If you get zero converged eigenvalues, try changing the parameters (e.g. ncv) or using a different strategy (e.g. shift-and-invert with a reasonable target). > > Jose > From gaurish108 at gmail.com Wed Jun 5 11:33:28 2013 From: gaurish108 at gmail.com (Gaurish Telang) Date: Wed, 5 Jun 2013 12:33:28 -0400 Subject: [petsc-users] Basic question on making SNES continue its iterations Message-ID: I am trying to solve a non-linear system with PETSs's SNES solvers. To run my code, I am using the following options for snes set through the command-line, seen below. mpirun -np 1 ./Testvector \ -fBus $1/Bus.mat \ -fYm $1/Ym.mat \ -ft $1/t.mat \ -flinedata $1/linedata.mat \ -nox \ -snes_atol 1.e-5\ -snes_stol 1.e-6 \ -snes_rtol 1.e-12 \ -snes_max_it 100 \ -snes_max_linear_solve_fail 100 \ -snes_converged_reason \ -ksp_type preonly \ -pc_type lu \ -pc_factor_mat_solver_package superlu_dist \ -snes_monitor \ > result_superLU_dist.tmp 2>&1; After feeding my data, I get the following output 0 SNES Function norm 2.438306296478e+02 1 SNES Function norm 1.679437793666e+02 2 SNES Function norm 1.493618467597e+02 3 SNES Function norm 4.622194635741e+01 4 SNES Function norm 4.195017632984e+01 Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 4 As you can see the the SNES function norm is *decreasing*, which indicates it is probably converging slowly to the actual solution. However after just 4 iterations, it stops and indicates non-convergence, even though my -snes_max_it = 100 as shown above. Is there a way to tell PETSc to continue its SNES iterations? Maybe some there are some default parameters which have to be changed? Thank you Gaurish -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 5 12:03:19 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 5 Jun 2013 12:03:19 -0500 Subject: [petsc-users] Basic question on making SNES continue its iterations In-Reply-To: References: Message-ID: <578764D1-AA13-4094-A8FE-0F04641976AC@mcs.anl.gov> There are many things that can go wrong with Newton's method See http://www.mcs.anl.gov/petsc/documentation/faq.html#newton On Jun 5, 2013, at 11:33 AM, Gaurish Telang wrote: > I am trying to solve a non-linear system with PETSs's SNES solvers. > > To run my code, I am using the following options for snes set through the command-line, > seen below. > > mpirun -np 1 ./Testvector \ > -fBus $1/Bus.mat \ > -fYm $1/Ym.mat \ > -ft $1/t.mat \ > -flinedata $1/linedata.mat \ > -nox \ > -snes_atol 1.e-5\ > -snes_stol 1.e-6 \ > -snes_rtol 1.e-12 \ > -snes_max_it 100 \ > -snes_max_linear_solve_fail 100 \ > -snes_converged_reason \ > -ksp_type preonly \ > -pc_type lu \ > -pc_factor_mat_solver_package superlu_dist \ > -snes_monitor \ > > result_superLU_dist.tmp 2>&1; > > > After feeding my data, I get the following output > 0 SNES Function norm 2.438306296478e+02 > 1 SNES Function norm 1.679437793666e+02 > 2 SNES Function norm 1.493618467597e+02 > 3 SNES Function norm 4.622194635741e+01 > 4 SNES Function norm 4.195017632984e+01 > Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 4 > > As you can see the the SNES function norm is *decreasing*, which indicates it is probably converging slowly to the actual solution. However after just 4 iterations, it stops and indicates non-convergence, even though my -snes_max_it = 100 as shown above. > > Is there a way to tell PETSc to continue its SNES iterations? Maybe some there are some > default parameters which have to be changed? > > Thank you > > Gaurish From knepley at gmail.com Wed Jun 5 12:04:56 2013 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 5 Jun 2013 13:04:56 -0400 Subject: [petsc-users] Basic question on making SNES continue its iterations In-Reply-To: References: Message-ID: On Wed, Jun 5, 2013 at 12:33 PM, Gaurish Telang wrote: > I am trying to solve a non-linear system with PETSs's SNES solvers. > > To run my code, I am using the following options for snes set through the > command-line, > seen below. > > mpirun -np 1 ./Testvector \ > -fBus $1/Bus.mat \ > -fYm $1/Ym.mat \ > -ft $1/t.mat \ > -flinedata $1/linedata.mat \ > -nox \ > -snes_atol 1.e-5\ > -snes_stol 1.e-6 \ > -snes_rtol 1.e-12 \ > -snes_max_it 100 \ > -snes_max_linear_solve_fail 100 \ > -snes_converged_reason \ > -ksp_type preonly \ > -pc_type lu \ > -pc_factor_mat_solver_package superlu_dist \ > -snes_monitor \ > > result_superLU_dist.tmp 2>&1; > > > After feeding my data, I get the following output > 0 SNES Function norm 2.438306296478e+02 > 1 SNES Function norm 1.679437793666e+02 > 2 SNES Function norm 1.493618467597e+02 > 3 SNES Function norm 4.622194635741e+01 > 4 SNES Function norm 4.195017632984e+01 > Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 4 > > As you can see the the SNES function norm is *decreasing*, which indicates > it is probably converging slowly to the actual solution. However after just > 4 iterations, it stops and indicates non-convergence, even though my > -snes_max_it = 100 as shown above. > > Is there a way to tell PETSc to continue its SNES iterations? Maybe some > there are some > default parameters which have to be changed? > You can try -snes_max_fail or maybe -snes_linesearch_type basic. Matt > Thank you > > Gaurish > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Wed Jun 5 13:05:20 2013 From: mrosso at uci.edu (Michele Rosso) Date: Wed, 05 Jun 2013 11:05:20 -0700 Subject: [petsc-users] Solving Poisson equation with multigrid In-Reply-To: References: <519687DD.4050209@uci.edu> <87ppwpmbne.fsf@mcs.anl.gov> <5196C171.7060400@uci.edu> <5196C5BE.7060601@uci.edu> <87mwrtm9l0.fsf@mcs.anl.gov> <5196CA3D.3070001@uci.edu> <87k3mxm8is.fsf@mcs.anl.gov> <5196CF3A.3030000@uci.edu> <87ehd5m4gn.fsf@mcs.anl.gov> <5196E8DC.1010602@uci.edu> <519FC4B0.9080702@uci.edu> <8738tcp2ya.fsf@mcs.anl.gov> <519FDD10.3060900@uci.edu> <87r4gwnj1a.fsf@mcs.anl.gov> <519FE730.9000309@uci.edu> <51A67267.4010507@uci.edu> <8761y1w1si.fsf@mcs.anl.gov> <51A91A9A.4050102@uci.edu> <8761xyrex8.fsf@mcs.anl.gov> <51A927D3.10500@uci.edu> Message-ID: <51AF7DE0.2040200@uci.edu> Thank you Satish for your help. I followed you instructions, but I am getting some errors during the compilation. I attached the python script I used to configure PETSc, and the logs for both configure and make. Michele On 05/31/2013 05:08 PM, Satish Balay wrote: > The following is my configure command to build PETSc on a cray with > cray compilers: > > $ cat reconfigure-arch-test-cray.py > #!/usr/bin/python > if __name__ == '__main__': > import sys > import os > sys.path.insert(0, os.path.abspath('config')) > import configure > configure_options = [ > '--with-cc=cc', > '--with-clanguage=C++', > '--with-clib-autodetect=0', > '--with-cxx=CC', > '--with-cxxlib-autodetect=0', > '--with-fc=ftn', > '--with-fortranlib-autodetect=0', > '--with-x=0', > 'FFLAGS=-F -em', > 'LIBS=-L/opt/cray/cce/8.1.4/CC/x86-64/lib/x86-64/ -lcray-c++-rts -lcraystdc++ -lsupc++ -lgcc_eh', > 'PETSC_ARCH=arch-test-cray', > ] > configure.petsc_configure(configure_options) > <<<<<<<<< > > And you might have to look for the recommended optimization flags - and use with: > --with-debgging=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= etc. > > And on BlueWaters - you would have to remove the following lines from > PETSC_ARCH/include/petscconf.h [before runing 'make all' to build the > libraries] > > #ifndef PETSC_HAVE_GETPWUID > #define PETSC_HAVE_GETPWUID 1 > #endif > <<<<< > > Satish > > On Fri, 31 May 2013, Michele Rosso wrote: > >> Thanks, but I'd rather use Cray since only a small part of my code relies on >> PETSc and, as suggested by the BW staff, >> Cray-compiled code performs generally better on Cray system. >> >> Michele >> >> On 05/31/2013 02:57 PM, Jed Brown wrote: >>> Michele Rosso writes: >>> >>>> Hi, >>>> >>>> I confirm that >>>> >>>> -pc_type gamg -pc_mg_cycle_type v -pc_gamg_agg_nsmooths 1 >>>> >>>> produces the correct shift in PETSc 3.4. >>>> So my problem is solved. I will upgrade to 3.4 in my productive >>>> machine installation. >>>> It is a Cray machine (Blue Waters). Assuming I want to use the Cray >>>> compiler, which options should I use for ./configure >>>> for the installation. >>> Or the Intel compiler or GCC. PETSc should run at similar speed with >>> any. IIRC, Cray recommends the Intel compiler on their machines with >>> Intel CPUs. There are examples in config/examples/. >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 1756607 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: text/x-log Size: 69781 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: reconfigure-arch-test-cray.py Type: text/x-python Size: 729 bytes Desc: not available URL: From Shuangshuang.Jin at pnnl.gov Wed Jun 5 13:45:37 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Wed, 5 Jun 2013 11:45:37 -0700 Subject: [petsc-users] a question about DAE Function In-Reply-To: <8738sxjw2g.fsf@mcs.anl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9FA@EMAIL04.pnl.gov> <87d2s1liac.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEA8E@EMAIL04.pnl.gov> <8738sxjw2g.fsf@mcs.anl.gov> Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FD87AEBC0@EMAIL04.pnl.gov> Hi, thanks for your message, but I still have a few questions here. 1. How to "reuse the VecScatter"? 2. The data movement of O(N) is for scattering X vector to each processor? And the computation of O(N^3) is for solving the DAE equation in parallel? So "the parallelism will be useful if N is large enough" as you indicated. Then does that mean my current implementation of scattering X and defining f[] on different processors to solve the DAE in parallel is the right way to do it? Then what's the benefit of using MatElemental? 3. When I looked into the example of MatElemental, it seems to me that they're all matrix operations. I cannot understand its connection with my code, especially the IFUNCTION part. I only have several functions definitions f[] using the data of a distributed array X. Don't know where and how to use MatElemental here. Please help me to clarify this questions which I'm really confused with. Thanks, Shuangshuang -----Original Message----- From: Jed Brown [mailto:five9a2 at gmail.com] On Behalf Of Jed Brown Sent: Tuesday, June 04, 2013 4:28 PM To: Jin, Shuangshuang Cc: petsc-users at mcs.anl.gov Subject: RE: [petsc-users] a question about DAE Function "Jin, Shuangshuang" writes: > Sorry about the confusion. Please allow me to illustrate the problem in a better way below: > > I have a set of DAE equations, which can be defined as 4*n f[] equations: > [...] > As you may see, all the f functions are dependent on some x values. > However, the x is a distributed array of size 4*n. Thanks, this problem is dense. > In order to use the access x[] from different processor, I have to > scatter x before I use them to define f[], the scatterMyVec function > is shown below. > > // Scatter the context of v to each processor to get a global array > (vArray) values static PetscErrorCode scatterMyVec(Vec vin, > PetscScalar **voutArray) { > PetscErrorCode ierr; > VecScatter ctx; > Vec vout; > > ierr = VecScatterCreateToAll(vin, &ctx, &vout); CHKERRQ(ierr); > ierr = VecScatterBegin(ctx, vin, vout, INSERT_VALUES, SCATTER_FORWARD); CHKERRQ(ierr); > ierr = VecScatterEnd(ctx, vin, vout, INSERT_VALUES, SCATTER_FORWARD); CHKERRQ(ierr); > ierr = VecGetArray(vout, voutArray); CHKERRQ(ierr); > > ierr = VecScatterDestroy(&ctx); CHKERRQ(ierr); Note that you can reuse the VecScatter. > ierr = VecRestoreArray(vout, PETSC_NULL); Please don't abuse VecRestoreArray to lie to PETSc. Yes, it will "work" with normal PETSc Vecs, but not in general (e.g. GPU-backed Vecs). > > PetscFunctionReturn(0); > } > > As I mentioned in the previous email, I think this is quite > inefficient because the scattering of x during each iteration > "requires a good chunk of communication". And as Barry pointed out, " > the Jacobian will be dense and the linear solves with the dense > Jacobian will dominate the time anyways. " > > So my question now is: 1. Am I right to use the PETSc TS parallel > paradigm in this way to solve my DAE equations? My concern is although > I can see the solution be computed across multiple processors, the > overhead of scattering x from time to time and the linear solving with > dense Jacobian will eat up the performance gains of parallel > computation. The data movement is O(N) while the computation is O(N^3), so parallelism will be useful if N is large enough. > 2. If the answer of the above question is "NO", what's the correct > parallel tool I should look into in PETSc to solve my DAE equations? > MatElemental as most of you suggested? TS with MatElemental is right. Here are some tests that use MatElemental. You should use MatGetOwnershipIS() to determine which matrix entries you are responsible for computing. You can use your current code for IFunction evaluation. src/ksp/ksp/examples/tests/ex40.c src/mat/examples/tests/ex104.c src/mat/examples/tests/ex145.c src/mat/examples/tests/ex38.c src/mat/examples/tests/ex39.c From jedbrown at mcs.anl.gov Wed Jun 5 14:01:31 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 05 Jun 2013 14:01:31 -0500 Subject: [petsc-users] a question about DAE Function In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FD87AEBC0@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9FA@EMAIL04.pnl.gov> <87d2s1liac.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEA8E@EMAIL04.pnl.gov> <8738sxjw2g.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEBC0@EMAIL04.pnl.gov> Message-ID: <8761xsgz5g.fsf@mcs.anl.gov> "Jin, Shuangshuang" writes: > Hi, thanks for your message, but I still have a few questions here. > > 1. How to "reuse the VecScatter"? Store it in your user context and reuse it the next time you need to scatter. > 2. The data movement of O(N) is for scattering X vector to each > processor? And the computation of O(N^3) is for solving the DAE > equation in parallel? So "the parallelism will be useful if N is large > enough" as you indicated. Then does that mean my current > implementation of scattering X and defining f[] on different > processors to solve the DAE in parallel is the right way to do it? > Then what's the benefit of using MatElemental? That's fine to define the equations, but you still have to solve them somehow. Most common methods do that using a matrix. > 3. When I looked into the example of MatElemental, it seems to me that > they're all matrix operations. I cannot understand its connection with > my code, especially the IFUNCTION part. I only have several functions > definitions f[] using the data of a distributed array X. Don't know > where and how to use MatElemental here. Are you assembling a Jacobian (TSSetIJacobian)? If not, then Elemental will not help, but keep in mind that this will converge very slowly unless your system is well-conditioned. From Shuangshuang.Jin at pnnl.gov Wed Jun 5 14:14:46 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Wed, 5 Jun 2013 12:14:46 -0700 Subject: [petsc-users] a question about DAE Function In-Reply-To: <8761xsgz5g.fsf@mcs.anl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9FA@EMAIL04.pnl.gov> <87d2s1liac.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEA8E@EMAIL04.pnl.gov> <8738sxjw2g.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEBC0@EMAIL04.pnl.gov> <8761xsgz5g.fsf@mcs.anl.gov> Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FD87AEBD3@EMAIL04.pnl.gov> I see. So for my IFunction which is called by TSSetIFunction, I can leave it as what it is right now (I mean scatter x and define equations), but for the IJacobian function which is called by TSSetIJacobian, I should define the Jacobian matrix as MATELEMENTAL type and set up its values accordingly? My Jacobian matrix also use the values of the distributed array X. Generally, the benefit of MATELEMENTAL is to avoid the scattering of X to each processor inside the IJacobian function in each iteration by "using MatGetOwnershipIS() to determine which matrix entries are responsible for computing" and such? Thanks, Shuangshuang -----Original Message----- From: Jed Brown [mailto:five9a2 at gmail.com] On Behalf Of Jed Brown Sent: Wednesday, June 05, 2013 12:02 PM To: Jin, Shuangshuang Cc: petsc-users at mcs.anl.gov Subject: RE: [petsc-users] a question about DAE Function "Jin, Shuangshuang" writes: > Hi, thanks for your message, but I still have a few questions here. > > 1. How to "reuse the VecScatter"? Store it in your user context and reuse it the next time you need to scatter. > 2. The data movement of O(N) is for scattering X vector to each > processor? And the computation of O(N^3) is for solving the DAE > equation in parallel? So "the parallelism will be useful if N is large > enough" as you indicated. Then does that mean my current > implementation of scattering X and defining f[] on different > processors to solve the DAE in parallel is the right way to do it? > Then what's the benefit of using MatElemental? That's fine to define the equations, but you still have to solve them somehow. Most common methods do that using a matrix. > 3. When I looked into the example of MatElemental, it seems to me that > they're all matrix operations. I cannot understand its connection with > my code, especially the IFUNCTION part. I only have several functions > definitions f[] using the data of a distributed array X. Don't know > where and how to use MatElemental here. Are you assembling a Jacobian (TSSetIJacobian)? If not, then Elemental will not help, but keep in mind that this will converge very slowly unless your system is well-conditioned. From jedbrown at mcs.anl.gov Wed Jun 5 14:39:02 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 05 Jun 2013 14:39:02 -0500 Subject: [petsc-users] a question about DAE Function In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FD87AEBD3@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9FA@EMAIL04.pnl.gov> <87d2s1liac.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEA8E@EMAIL04.pnl.gov> <8738sxjw2g.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEBC0@EMAIL04.pnl.gov> <8761xsgz5g.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEBD3@EMAIL04.pnl.gov> Message-ID: <87vc5sfiuh.fsf@mcs.anl.gov> "Jin, Shuangshuang" writes: > I see. So for my IFunction which is called by TSSetIFunction, I can > leave it as what it is right now (I mean scatter x and define > equations), but for the IJacobian function which is called by > TSSetIJacobian, I should define the Jacobian matrix as MATELEMENTAL > type and set up its values accordingly? Yes. > My Jacobian matrix also use the values of the distributed array > X. Generally, the benefit of MATELEMENTAL is to avoid the scattering > of X to each processor inside the IJacobian function in each iteration > by "using MatGetOwnershipIS() to determine which matrix entries are > responsible for computing" and such? The entries of the matrix might still have global dependence, but for dense problems, it's important to use a "2D" distribution. MatGetOwnershipIS tells you which of those entries you are responsible for computing. The entries are in a place that will work well for the solve. From balay at mcs.anl.gov Wed Jun 5 16:15:27 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 5 Jun 2013 16:15:27 -0500 (CDT) Subject: [petsc-users] Solving Poisson equation with multigrid In-Reply-To: <51AF7DE0.2040200@uci.edu> References: <519687DD.4050209@uci.edu> <87ppwpmbne.fsf@mcs.anl.gov> <5196C171.7060400@uci.edu> <5196C5BE.7060601@uci.edu> <87mwrtm9l0.fsf@mcs.anl.gov> <5196CA3D.3070001@uci.edu> <87k3mxm8is.fsf@mcs.anl.gov> <5196CF3A.3030000@uci.edu> <87ehd5m4gn.fsf@mcs.anl.gov> <5196E8DC.1010602@uci.edu> <519FC4B0.9080702@uci.edu> <8738tcp2ya.fsf@mcs.anl.gov> <519FDD10.3060900@uci.edu> <87r4gwnj1a.fsf@mcs.anl.gov> <519FE730.9000309@uci.edu> <51A67267.4010507@uci.edu> <8761y1w1si.fsf@mcs.anl.gov> <51A91A9A.4050102@uci.edu> <8761xyrex8.fsf@mcs.anl.gov> <51A927D3.10500@uci.edu> <51AF7DE0.2040200@uci.edu> Message-ID: Sorry my configure cmd file was from an older test case for c++. Do you need c++ build of PETSc? If not - suggest removing --with-clanguage=C++ option. [Majority of use cases should not need --with-clanguage=C++ - even if user code is using PETSc from C++] If c++ is needed - you can try adding: CXXCPPFLAGS='-h gnu' Satish On Wed, 5 Jun 2013, Michele Rosso wrote: > Thank you Satish for your help. > I followed you instructions, but I am getting some errors during the > compilation. > I attached the python script I used to configure PETSc, and the logs for both > configure and make. > > > Michele > > On 05/31/2013 05:08 PM, Satish Balay wrote: > > The following is my configure command to build PETSc on a cray with > > cray compilers: > > > > $ cat reconfigure-arch-test-cray.py > > #!/usr/bin/python > > if __name__ == '__main__': > > import sys > > import os > > sys.path.insert(0, os.path.abspath('config')) > > import configure > > configure_options = [ > > '--with-cc=cc', > > '--with-clanguage=C++', > > '--with-clib-autodetect=0', > > '--with-cxx=CC', > > '--with-cxxlib-autodetect=0', > > '--with-fc=ftn', > > '--with-fortranlib-autodetect=0', > > '--with-x=0', > > 'FFLAGS=-F -em', > > 'LIBS=-L/opt/cray/cce/8.1.4/CC/x86-64/lib/x86-64/ -lcray-c++-rts > > -lcraystdc++ -lsupc++ -lgcc_eh', > > 'PETSC_ARCH=arch-test-cray', > > ] > > configure.petsc_configure(configure_options) > > <<<<<<<<< > > > > And you might have to look for the recommended optimization flags - and use > > with: > > --with-debgging=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= etc. > > > > And on BlueWaters - you would have to remove the following lines from > > PETSC_ARCH/include/petscconf.h [before runing 'make all' to build the > > libraries] > > > > #ifndef PETSC_HAVE_GETPWUID > > #define PETSC_HAVE_GETPWUID 1 > > #endif > > <<<<< > > > > Satish > > > > On Fri, 31 May 2013, Michele Rosso wrote: > > > > > Thanks, but I'd rather use Cray since only a small part of my code relies > > > on > > > PETSc and, as suggested by the BW staff, > > > Cray-compiled code performs generally better on Cray system. > > > > > > Michele > > > > > > On 05/31/2013 02:57 PM, Jed Brown wrote: > > > > Michele Rosso writes: > > > > > > > > > Hi, > > > > > > > > > > I confirm that > > > > > > > > > > -pc_type gamg -pc_mg_cycle_type v -pc_gamg_agg_nsmooths 1 > > > > > > > > > > produces the correct shift in PETSc 3.4. > > > > > So my problem is solved. I will upgrade to 3.4 in my productive > > > > > machine installation. > > > > > It is a Cray machine (Blue Waters). Assuming I want to use the Cray > > > > > compiler, which options should I use for ./configure > > > > > for the installation. > > > > Or the Intel compiler or GCC. PETSc should run at similar speed with > > > > any. IIRC, Cray recommends the Intel compiler on their machines with > > > > Intel CPUs. There are examples in config/examples/. > > > > > > > > > From Shuangshuang.Jin at pnnl.gov Wed Jun 5 17:25:03 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Wed, 5 Jun 2013 15:25:03 -0700 Subject: [petsc-users] a question about DAE Function In-Reply-To: <87vc5sfiuh.fsf@mcs.anl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9FA@EMAIL04.pnl.gov> <87d2s1liac.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEA8E@EMAIL04.pnl.gov> <8738sxjw2g.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEBC0@EMAIL04.pnl.gov> <8761xsgz5g.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEBD3@EMAIL04.pnl.gov> <87vc5sfiuh.fsf@mcs.anl.gov> Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FD87AEC63@EMAIL04.pnl.gov> Is there anything particular installation required for me to use MATELEMENTAL? I got the following error when I tried to use MATELEMENTAL in the code: [d3m956 at olympus tests]$ mpiexec -n 1 ex145 Create Elemental matrix A [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! [0]PETSC ERROR: Unknown Mat type given: elemental! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development GIT revision: cee652cbe889c53a5c397ca74d3ef6fdcdf8b9eb GIT Date: 2013-04-21 09:49:07 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ex145 on a arch-complex named olympus.local by d3m956 Wed Jun 5 15:20:54 2013 [0]PETSC ERROR: Libraries linked from /pic/projects/ds/petsc-dev.4.23.13/arch-complex/lib [0]PETSC ERROR: Configure run at Tue Apr 23 12:17:01 2013 [0]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatSetType() line 63 in /pic/projects/ds/petsc-dev.4.23.13/src/mat/interface/matreg.c [0]PETSC ERROR: main() line 42 in src/mat/examples/tests/ex145.c -------------------------------------------------------------------------- Thanks, Shuangshuang -----Original Message----- From: Jed Brown [mailto:five9a2 at gmail.com] On Behalf Of Jed Brown Sent: Wednesday, June 05, 2013 12:39 PM To: Jin, Shuangshuang Cc: petsc-users at mcs.anl.gov Subject: RE: [petsc-users] a question about DAE Function "Jin, Shuangshuang" writes: > I see. So for my IFunction which is called by TSSetIFunction, I can > leave it as what it is right now (I mean scatter x and define > equations), but for the IJacobian function which is called by > TSSetIJacobian, I should define the Jacobian matrix as MATELEMENTAL > type and set up its values accordingly? Yes. > My Jacobian matrix also use the values of the distributed array X. > Generally, the benefit of MATELEMENTAL is to avoid the scattering of X > to each processor inside the IJacobian function in each iteration by > "using MatGetOwnershipIS() to determine which matrix entries are > responsible for computing" and such? The entries of the matrix might still have global dependence, but for dense problems, it's important to use a "2D" distribution. MatGetOwnershipIS tells you which of those entries you are responsible for computing. The entries are in a place that will work well for the solve. From jedbrown at mcs.anl.gov Wed Jun 5 18:16:33 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 05 Jun 2013 18:16:33 -0500 Subject: [petsc-users] a question about DAE Function In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FD87AEC63@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9BC@EMAIL04.pnl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AE9FA@EMAIL04.pnl.gov> <87d2s1liac.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEA8E@EMAIL04.pnl.gov> <8738sxjw2g.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEBC0@EMAIL04.pnl.gov> <8761xsgz5g.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEBD3@EMAIL04.pnl.gov> <87vc5sfiuh.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD87AEC63@EMAIL04.pnl.gov> Message-ID: <87hahcf8ry.fsf@mcs.anl.gov> "Jin, Shuangshuang" writes: > Is there anything particular installation required for me to use > MATELEMENTAL? I got the following error when I tried to use > MATELEMENTAL in the code: Yes, it's an external package. You can get it by configuring with --download-elemental. You may as well upgrade to petsc-3.4. Since you're already using Git, just $ git checkout maint $ git pull This gets you the release plus maintenance patches (but not new features). From mrosso at uci.edu Thu Jun 6 14:27:01 2013 From: mrosso at uci.edu (Michele Rosso) Date: Thu, 06 Jun 2013 12:27:01 -0700 Subject: [petsc-users] Solving Poisson equation with multigrid In-Reply-To: References: <519687DD.4050209@uci.edu> <87ppwpmbne.fsf@mcs.anl.gov> <5196C171.7060400@uci.edu> <5196C5BE.7060601@uci.edu> <87mwrtm9l0.fsf@mcs.anl.gov> <5196CA3D.3070001@uci.edu> <87k3mxm8is.fsf@mcs.anl.gov> <5196CF3A.3030000@uci.edu> <87ehd5m4gn.fsf@mcs.anl.gov> <5196E8DC.1010602@uci.edu> <519FC4B0.9080702@uci.edu> <8738tcp2ya.fsf@mcs.anl.gov> <519FDD10.3060900@uci.edu> <87r4gwnj1a.fsf@mcs.anl.gov> <519FE730.9000309@uci.edu> <51A67267.4010507@uci.edu> <8761y1w1si.fsf@mcs.anl.gov> <51A91A9A.4050102@uci.edu> <8761xyrex8.fsf@mcs.anl.gov> <51A927D3.10500@uci.edu> <51AF7DE0.2040200@uci.edu> Message-ID: <51B0E285.7020404@uci.edu> Hi Satish, I tried what you suggested: #!/usr/bin/python if __name__ == '__main__': import sys import os sys.path.insert(0, os.path.abspath('config')) import configure configure_options = [ 'PETSC_DIR=/u/sciteam/mrosso/LIBS/petsc-3.4.0', '--with-cc=cc', '--with-clib-autodetect=0', '--with-cxx=CC', '--with-cxxlib-autodetect=0', '--with-fc=ftn', '--with-fortranlib-autodetect=0', '--with-x=0', '--with-debgging=0', '--COPTFLAGS=-O3', '--FOPTFLAGS=-O3', 'FFLAGS=-F -em', 'LIBS=-L/opt/cray/cce/8.1.7/CC/x86-64/lib/x86-64/ -lcray-c++ -rts -lcraystdc++ -lsupc++ -lgcc_eh', 'PETSC_ARCH=arch-test-cray', ] configure.petsc_configure(configure_options) but now I receive another error: "C compiler you provided with -with-cc=cc does not work" I attached the log file. Thanks for your help. Michele On 06/05/2013 02:15 PM, Satish Balay wrote: > Sorry my configure cmd file was from an older test case for c++. > > Do you need c++ build of PETSc? If not - suggest removing > --with-clanguage=C++ option. > > [Majority of use cases should not need --with-clanguage=C++ - even if > user code is using PETSc from C++] > > If c++ is needed - you can try adding: CXXCPPFLAGS='-h gnu' > > Satish > > On Wed, 5 Jun 2013, Michele Rosso wrote: > >> Thank you Satish for your help. >> I followed you instructions, but I am getting some errors during the >> compilation. >> I attached the python script I used to configure PETSc, and the logs for both >> configure and make. >> >> >> Michele >> >> On 05/31/2013 05:08 PM, Satish Balay wrote: >>> The following is my configure command to build PETSc on a cray with >>> cray compilers: >>> >>> $ cat reconfigure-arch-test-cray.py >>> #!/usr/bin/python >>> if __name__ == '__main__': >>> import sys >>> import os >>> sys.path.insert(0, os.path.abspath('config')) >>> import configure >>> configure_options = [ >>> '--with-cc=cc', >>> '--with-clanguage=C++', >>> '--with-clib-autodetect=0', >>> '--with-cxx=CC', >>> '--with-cxxlib-autodetect=0', >>> '--with-fc=ftn', >>> '--with-fortranlib-autodetect=0', >>> '--with-x=0', >>> 'FFLAGS=-F -em', >>> 'LIBS=-L/opt/cray/cce/8.1.4/CC/x86-64/lib/x86-64/ -lcray-c++-rts >>> -lcraystdc++ -lsupc++ -lgcc_eh', >>> 'PETSC_ARCH=arch-test-cray', >>> ] >>> configure.petsc_configure(configure_options) >>> <<<<<<<<< >>> >>> And you might have to look for the recommended optimization flags - and use >>> with: >>> --with-debgging=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= etc. >>> >>> And on BlueWaters - you would have to remove the following lines from >>> PETSC_ARCH/include/petscconf.h [before runing 'make all' to build the >>> libraries] >>> >>> #ifndef PETSC_HAVE_GETPWUID >>> #define PETSC_HAVE_GETPWUID 1 >>> #endif >>> <<<<< >>> >>> Satish >>> >>> On Fri, 31 May 2013, Michele Rosso wrote: >>> >>>> Thanks, but I'd rather use Cray since only a small part of my code relies >>>> on >>>> PETSc and, as suggested by the BW staff, >>>> Cray-compiled code performs generally better on Cray system. >>>> >>>> Michele >>>> >>>> On 05/31/2013 02:57 PM, Jed Brown wrote: >>>>> Michele Rosso writes: >>>>> >>>>>> Hi, >>>>>> >>>>>> I confirm that >>>>>> >>>>>> -pc_type gamg -pc_mg_cycle_type v -pc_gamg_agg_nsmooths 1 >>>>>> >>>>>> produces the correct shift in PETSc 3.4. >>>>>> So my problem is solved. I will upgrade to 3.4 in my productive >>>>>> machine installation. >>>>>> It is a Cray machine (Blue Waters). Assuming I want to use the Cray >>>>>> compiler, which options should I use for ./configure >>>>>> for the installation. >>>>> Or the Intel compiler or GCC. PETSc should run at similar speed with >>>>> any. IIRC, Cray recommends the Intel compiler on their machines with >>>>> Intel CPUs. There are examples in config/examples/. >>>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 121483 bytes Desc: not available URL: From balay at mcs.anl.gov Thu Jun 6 14:30:52 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 6 Jun 2013 14:30:52 -0500 (CDT) Subject: [petsc-users] Solving Poisson equation with multigrid In-Reply-To: <51B0E285.7020404@uci.edu> References: <519687DD.4050209@uci.edu> <87ppwpmbne.fsf@mcs.anl.gov> <5196C171.7060400@uci.edu> <5196C5BE.7060601@uci.edu> <87mwrtm9l0.fsf@mcs.anl.gov> <5196CA3D.3070001@uci.edu> <87k3mxm8is.fsf@mcs.anl.gov> <5196CF3A.3030000@uci.edu> <87ehd5m4gn.fsf@mcs.anl.gov> <5196E8DC.1010602@uci.edu> <519FC4B0.9080702@uci.edu> <8738tcp2ya.fsf@mcs.anl.gov> <519FDD10.3060900@uci.edu> <87r4gwnj1a.fsf@mcs.anl.gov> <519FE730.9000309@uci.edu> <51A67267.4010507@uci.edu> <8761y1w1si.fsf@mcs.anl.gov> <51A91A9A.4050102@uci.edu> <8761xyrex8.fsf@mcs.anl.gov> <51A927D3.10500@uci.edu> <51AF7DE0.2040200@uci.edu> <51B0E285.7020404@uci.edu> Message-ID: On Thu, 6 Jun 2013, Michele Rosso wrote: > Hi Satish, > > I tried what you suggested: > > #!/usr/bin/python > if __name__ == '__main__': > import sys > import os > sys.path.insert(0, os.path.abspath('config')) > import configure > configure_options = [ > 'PETSC_DIR=/u/sciteam/mrosso/LIBS/petsc-3.4.0', > '--with-cc=cc', > '--with-clib-autodetect=0', > '--with-cxx=CC', > '--with-cxxlib-autodetect=0', > '--with-fc=ftn', > '--with-fortranlib-autodetect=0', > '--with-x=0', > '--with-debgging=0', > '--COPTFLAGS=-O3', > '--FOPTFLAGS=-O3', > 'FFLAGS=-F -em', > 'LIBS=-L/opt/cray/cce/8.1.7/CC/x86-64/lib/x86-64/ -lcray-c++ -rts you have an extra space here ^ It should be -lcray-c++-rt Satish > -lcraystdc++ -lsupc++ -lgcc_eh', > 'PETSC_ARCH=arch-test-cray', > ] > configure.petsc_configure(configure_options) > > but now I receive another error: > > "C compiler you provided with -with-cc=cc does not work" > > I attached the log file. Thanks for your help. > > Michele > > > On 06/05/2013 02:15 PM, Satish Balay wrote: > > Sorry my configure cmd file was from an older test case for c++. > > > > Do you need c++ build of PETSc? If not - suggest removing > > --with-clanguage=C++ option. > > > > [Majority of use cases should not need --with-clanguage=C++ - even if > > user code is using PETSc from C++] > > > > If c++ is needed - you can try adding: CXXCPPFLAGS='-h gnu' > > > > Satish > > > > On Wed, 5 Jun 2013, Michele Rosso wrote: > > > > > Thank you Satish for your help. > > > I followed you instructions, but I am getting some errors during the > > > compilation. > > > I attached the python script I used to configure PETSc, and the logs for > > > both > > > configure and make. > > > > > > > > > Michele > > > > > > On 05/31/2013 05:08 PM, Satish Balay wrote: > > > > The following is my configure command to build PETSc on a cray with > > > > cray compilers: > > > > > > > > $ cat reconfigure-arch-test-cray.py > > > > #!/usr/bin/python > > > > if __name__ == '__main__': > > > > import sys > > > > import os > > > > sys.path.insert(0, os.path.abspath('config')) > > > > import configure > > > > configure_options = [ > > > > '--with-cc=cc', > > > > '--with-clanguage=C++', > > > > '--with-clib-autodetect=0', > > > > '--with-cxx=CC', > > > > '--with-cxxlib-autodetect=0', > > > > '--with-fc=ftn', > > > > '--with-fortranlib-autodetect=0', > > > > '--with-x=0', > > > > 'FFLAGS=-F -em', > > > > 'LIBS=-L/opt/cray/cce/8.1.4/CC/x86-64/lib/x86-64/ -lcray-c++-rts > > > > -lcraystdc++ -lsupc++ -lgcc_eh', > > > > 'PETSC_ARCH=arch-test-cray', > > > > ] > > > > configure.petsc_configure(configure_options) > > > > <<<<<<<<< > > > > > > > > And you might have to look for the recommended optimization flags - and > > > > use > > > > with: > > > > --with-debgging=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= etc. > > > > > > > > And on BlueWaters - you would have to remove the following lines from > > > > PETSC_ARCH/include/petscconf.h [before runing 'make all' to build the > > > > libraries] > > > > > > > > #ifndef PETSC_HAVE_GETPWUID > > > > #define PETSC_HAVE_GETPWUID 1 > > > > #endif > > > > <<<<< > > > > > > > > Satish > > > > > > > > On Fri, 31 May 2013, Michele Rosso wrote: > > > > > > > > > Thanks, but I'd rather use Cray since only a small part of my code > > > > > relies > > > > > on > > > > > PETSc and, as suggested by the BW staff, > > > > > Cray-compiled code performs generally better on Cray system. > > > > > > > > > > Michele > > > > > > > > > > On 05/31/2013 02:57 PM, Jed Brown wrote: > > > > > > Michele Rosso writes: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I confirm that > > > > > > > > > > > > > > -pc_type gamg -pc_mg_cycle_type v > > > > > > > -pc_gamg_agg_nsmooths 1 > > > > > > > > > > > > > > produces the correct shift in PETSc 3.4. > > > > > > > So my problem is solved. I will upgrade to 3.4 in my productive > > > > > > > machine installation. > > > > > > > It is a Cray machine (Blue Waters). Assuming I want to use the > > > > > > > Cray > > > > > > > compiler, which options should I use for ./configure > > > > > > > for the installation. > > > > > > Or the Intel compiler or GCC. PETSc should run at similar speed > > > > > > with > > > > > > any. IIRC, Cray recommends the Intel compiler on their machines > > > > > > with > > > > > > Intel CPUs. There are examples in config/examples/. > > > > > > > > > > > From a.vergottis at gmail.com Thu Jun 6 15:02:31 2013 From: a.vergottis at gmail.com (Anthony Vergottis) Date: Thu, 6 Jun 2013 21:02:31 +0100 Subject: [petsc-users] Multiple KSP Solver Message-ID: Dear All, This message is related to the multiple use of a single KSP solver context. *My question is*: If I set up a KSP solver is it possible to use it as many times as I require? *What I mean*: If I create a KSP solver called *solver* and associate with it the required Matrix i.e. the A matrix of the system and the right had side vector b. I then solve the system calling KSPSolve(*solver*,b,Solution); for example. Say now I want to re-use the same context *solver* but I supply a different right had side vector b and again call the same command KSPSolve(*solver*,b,Solution);, is this sufficient to perform the new solve correctly and get a new solution vector x? Will it reuse the before specified matrix A in the system (and the same solver eg. GMRES and preconditioner)? or do I need to recreate a new solver each time? is there any specific behaviour of Petsc I need to be aware off? Solution algorithm is an iterative one and I require to solve 5 different equations at different time-step, the only thing that changes from each time step is the right had side vector of the system. So the matrix A for all the equations are the same at each time step. I hope I explained this good enough. Sorry if this is a trivial questions. Thanks in advance. Regards, Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jun 6 15:08:13 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 6 Jun 2013 15:08:13 -0500 Subject: [petsc-users] Multiple KSP Solver In-Reply-To: References: Message-ID: <3BAACD71-BDD0-4102-97CC-B1BF493375C8@mcs.anl.gov> Just call KSPSolve() with each right hand side. You don't need to do anything else. If you change the matrix then you need to call KSPSetOperators() with the updated matrix. Barry On Jun 6, 2013, at 3:02 PM, Anthony Vergottis wrote: > Dear All, > > This message is related to the multiple use of a single KSP solver context. > > My question is: If I set up a KSP solver is it possible to use it as many times as I require? > > What I mean: If I create a KSP solver called solver and associate with it the required Matrix i.e. the A matrix of the system and the right had side vector b. I then solve the system calling KSPSolve(solver,b,Solution); for example. > > Say now I want to re-use the same context solver but I supply a different right had side vector b and again call the same command > KSPSolve(solver,b,Solution);, is this sufficient to perform the new solve correctly and get a new solution vector x? Will it reuse the before specified matrix A in the system (and the same solver eg. GMRES and preconditioner)? or do I need to recreate a new solver each time? is there any specific behaviour of Petsc I need to be aware off? > > Solution algorithm is an iterative one and I require to solve 5 different equations at different time-step, the only thing that changes from each time step is the right had side vector of the system. So the matrix A for all the equations are the same at each time step. > > I hope I explained this good enough. Sorry if this is a trivial questions. > > Thanks in advance. > > Regards, > Anthony > From mpovolot at purdue.edu Thu Jun 6 16:55:31 2013 From: mpovolot at purdue.edu (Michael Povolotskyi) Date: Thu, 06 Jun 2013 17:55:31 -0400 Subject: [petsc-users] error while allocating dense distributed matrix Message-ID: <51B10553.9030700@purdue.edu> Dear Petsc developers, petsc gives error with the error code equal to 60 when I'm doing the following: ierr = MatCreateMPIDense(comm, m, num_cols ,num_rows,num_cols,data,petsc_matrix) on rank = 0 the gebugger gives: (gdb) print m $1 = 240 (gdb) print num_cols $2 = 18 (gdb) print num_rows $3 = 360 on rank = 1 the gebugger gives: ierr = MatCreateMPIDense(comm, m, num_cols ,num_rows,num_cols,data,petsc_matrix); (gdb) print m $1 = 120 (gdb) print num_cols $2 = 18 (gdb) print num_rows $3 = 360 (gdb) Could you, please, tell me what is wrong here? Thank you, Michael. From bsmith at mcs.anl.gov Thu Jun 6 18:30:36 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 6 Jun 2013 18:30:36 -0500 Subject: [petsc-users] error while allocating dense distributed matrix In-Reply-To: <51B10553.9030700@purdue.edu> References: <51B10553.9030700@purdue.edu> Message-ID: The first "num_cols" argument refers to the number of "local" columns on that process the second one the total number of columns. With size > 1 they cannot be equal. The sum of all the "local" columns across all processes must equal the total number of columns. This is not explained well in the MatCreateDense manual page, sorry. Look at the manual page http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateAIJ.html and see the definition of m and n which explains it much better. The "local" row and column sizes are define by the "left" and "right" parallel vector layouts for vectors compatible with the matrix. Barry On Jun 6, 2013, at 4:55 PM, Michael Povolotskyi wrote: > Dear Petsc developers, > petsc gives error with the error code equal to 60 when I'm doing the following: > > ierr = MatCreateMPIDense(comm, m, num_cols ,num_rows,num_cols,data,petsc_matrix) > > on rank = 0 the gebugger gives: > (gdb) print m > $1 = 240 > (gdb) print num_cols > $2 = 18 > (gdb) print num_rows > $3 = 360 > > on rank = 1 the gebugger gives: > ierr = MatCreateMPIDense(comm, m, num_cols ,num_rows,num_cols,data,petsc_matrix); > (gdb) print m > $1 = 120 > (gdb) print num_cols > $2 = 18 > (gdb) print num_rows > $3 = 360 > (gdb) > > Could you, please, tell me what is wrong here? > Thank you, > Michael. From mrosso at uci.edu Thu Jun 6 18:47:30 2013 From: mrosso at uci.edu (Michele Rosso) Date: Thu, 06 Jun 2013 16:47:30 -0700 Subject: [petsc-users] Solving Poisson equation with multigrid In-Reply-To: References: <519687DD.4050209@uci.edu> <87ppwpmbne.fsf@mcs.anl.gov> <5196C171.7060400@uci.edu> <5196C5BE.7060601@uci.edu> <87mwrtm9l0.fsf@mcs.anl.gov> <5196CA3D.3070001@uci.edu> <87k3mxm8is.fsf@mcs.anl.gov> <5196CF3A.3030000@uci.edu> <87ehd5m4gn.fsf@mcs.anl.gov> <5196E8DC.1010602@uci.edu> <519FC4B0.9080702@uci.edu> <8738tcp2ya.fsf@mcs.anl.gov> <519FDD10.3060900@uci.edu> <87r4gwnj1a.fsf@mcs.anl.gov> <519FE730.9000309@uci.edu> <51A67267.4010507@uci.edu> <8761y1w1si.fsf@mcs.anl.gov> <51A91A9A.4050102@uci.edu> <8761xyrex8.fsf@mcs.anl.gov> <51A927D3.10500@uci.edu> <51AF7DE0.2040200@uci.edu> <51B0E285.7020404@uci.edu> Message-ID: <51B11F92.3050704@uci.edu> Thanks Satish, now the compilation goes smooth, despite a warning during the configuration step: =============================================================================== Warning: PETSC_ARCH from environment does not match command-line or name of script. Warning: Using from command-line or name of script: arch-test-cray, ignoring environment: interlagos =============================================================================== I still have a problem though: when I compile my code the following warning is issued: /u/sciteam/mrosso/LIBS/petsc-3.4.0/arch-test-cray/lib/libpetsc.a(dlimpl.o): In function `PetscDLOpen': /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.0/src/sys/dll/dlimpl.c:109: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /u/sciteam/mrosso/LIBS/petsc-3.4.0/arch-test-cray/lib/libpetsc.a(send.o): In function `PetscOpenSocket': /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.0/src/sys/classes/viewer/impls/socket/send.c:97: warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking and the run stops because of the following error: libhugetlbfs: ERROR: RTLD_NEXT used in code not dynamically loaded Thanks again for your help. Michele > On Thu, 6 Jun 2013, Michele Rosso wrote: > >> Hi Satish, >> >> I tried what you suggested: >> >> #!/usr/bin/python >> if __name__ == '__main__': >> import sys >> import os >> sys.path.insert(0, os.path.abspath('config')) >> import configure >> configure_options = [ >> 'PETSC_DIR=/u/sciteam/mrosso/LIBS/petsc-3.4.0', >> '--with-cc=cc', >> '--with-clib-autodetect=0', >> '--with-cxx=CC', >> '--with-cxxlib-autodetect=0', >> '--with-fc=ftn', >> '--with-fortranlib-autodetect=0', >> '--with-x=0', >> '--with-debgging=0', >> '--COPTFLAGS=-O3', >> '--FOPTFLAGS=-O3', >> 'FFLAGS=-F -em', >> 'LIBS=-L/opt/cray/cce/8.1.7/CC/x86-64/lib/x86-64/ -lcray-c++ -rts > you have an extra space here ^ It should be -lcray-c++-rt > > Satish > > >> -lcraystdc++ -lsupc++ -lgcc_eh', >> 'PETSC_ARCH=arch-test-cray', >> ] >> configure.petsc_configure(configure_options) >> >> but now I receive another error: >> >> "C compiler you provided with -with-cc=cc does not work" >> >> I attached the log file. Thanks for your help. >> >> Michele >> >> >> On 06/05/2013 02:15 PM, Satish Balay wrote: >>> Sorry my configure cmd file was from an older test case for c++. >>> >>> Do you need c++ build of PETSc? If not - suggest removing >>> --with-clanguage=C++ option. >>> >>> [Majority of use cases should not need --with-clanguage=C++ - even if >>> user code is using PETSc from C++] >>> >>> If c++ is needed - you can try adding: CXXCPPFLAGS='-h gnu' >>> >>> Satish >>> >>> On Wed, 5 Jun 2013, Michele Rosso wrote: >>> >>>> Thank you Satish for your help. >>>> I followed you instructions, but I am getting some errors during the >>>> compilation. >>>> I attached the python script I used to configure PETSc, and the logs for >>>> both >>>> configure and make. >>>> >>>> >>>> Michele >>>> >>>> On 05/31/2013 05:08 PM, Satish Balay wrote: >>>>> The following is my configure command to build PETSc on a cray with >>>>> cray compilers: >>>>> >>>>> $ cat reconfigure-arch-test-cray.py >>>>> #!/usr/bin/python >>>>> if __name__ == '__main__': >>>>> import sys >>>>> import os >>>>> sys.path.insert(0, os.path.abspath('config')) >>>>> import configure >>>>> configure_options = [ >>>>> '--with-cc=cc', >>>>> '--with-clanguage=C++', >>>>> '--with-clib-autodetect=0', >>>>> '--with-cxx=CC', >>>>> '--with-cxxlib-autodetect=0', >>>>> '--with-fc=ftn', >>>>> '--with-fortranlib-autodetect=0', >>>>> '--with-x=0', >>>>> 'FFLAGS=-F -em', >>>>> 'LIBS=-L/opt/cray/cce/8.1.4/CC/x86-64/lib/x86-64/ -lcray-c++-rts >>>>> -lcraystdc++ -lsupc++ -lgcc_eh', >>>>> 'PETSC_ARCH=arch-test-cray', >>>>> ] >>>>> configure.petsc_configure(configure_options) >>>>> <<<<<<<<< >>>>> >>>>> And you might have to look for the recommended optimization flags - and >>>>> use >>>>> with: >>>>> --with-debgging=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= etc. >>>>> >>>>> And on BlueWaters - you would have to remove the following lines from >>>>> PETSC_ARCH/include/petscconf.h [before runing 'make all' to build the >>>>> libraries] >>>>> >>>>> #ifndef PETSC_HAVE_GETPWUID >>>>> #define PETSC_HAVE_GETPWUID 1 >>>>> #endif >>>>> <<<<< >>>>> >>>>> Satish >>>>> >>>>> On Fri, 31 May 2013, Michele Rosso wrote: >>>>> >>>>>> Thanks, but I'd rather use Cray since only a small part of my code >>>>>> relies >>>>>> on >>>>>> PETSc and, as suggested by the BW staff, >>>>>> Cray-compiled code performs generally better on Cray system. >>>>>> >>>>>> Michele >>>>>> >>>>>> On 05/31/2013 02:57 PM, Jed Brown wrote: >>>>>>> Michele Rosso writes: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I confirm that >>>>>>>> >>>>>>>> -pc_type gamg -pc_mg_cycle_type v >>>>>>>> -pc_gamg_agg_nsmooths 1 >>>>>>>> >>>>>>>> produces the correct shift in PETSc 3.4. >>>>>>>> So my problem is solved. I will upgrade to 3.4 in my productive >>>>>>>> machine installation. >>>>>>>> It is a Cray machine (Blue Waters). Assuming I want to use the >>>>>>>> Cray >>>>>>>> compiler, which options should I use for ./configure >>>>>>>> for the installation. >>>>>>> Or the Intel compiler or GCC. PETSc should run at similar speed >>>>>>> with >>>>>>> any. IIRC, Cray recommends the Intel compiler on their machines >>>>>>> with >>>>>>> Intel CPUs. There are examples in config/examples/. >>>>>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jun 6 19:20:52 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 6 Jun 2013 19:20:52 -0500 (CDT) Subject: [petsc-users] Solving Poisson equation with multigrid In-Reply-To: <51B11F92.3050704@uci.edu> References: <519687DD.4050209@uci.edu> <5196C5BE.7060601@uci.edu> <87mwrtm9l0.fsf@mcs.anl.gov> <5196CA3D.3070001@uci.edu> <87k3mxm8is.fsf@mcs.anl.gov> <5196CF3A.3030000@uci.edu> <87ehd5m4gn.fsf@mcs.anl.gov> <5196E8DC.1010602@uci.edu> <519FC4B0.9080702@uci.edu> <8738tcp2ya.fsf@mcs.anl.gov> <519FDD10.3060900@uci.edu> <87r4gwnj1a.fsf@mcs.anl.gov> <519FE730.9000309@uci.edu> <51A67267.4010507@uci.edu> <8761y1w1si.fsf@mcs.anl.gov> <51A91A9A.4050102@uci.edu> <8761xyrex8.fsf@mcs.anl.gov> <51A927D3.10500@uci.edu> <51AF7DE0.2040200@uci.edu> <51B0E285.7020404@uci.edu> <51B11F92.3050704@uci.edu> Message-ID: On Thu, 6 Jun 2013, Michele Rosso wrote: > Thanks Satish, now the compilation goes smooth, despite a warning during the > configuration step: > > =============================================================================== > Warning: PETSC_ARCH from environment does not match command-line > or name of script. > Warning: Using from command-line or name of script: > arch-test-cray, ignoring environment: interlagos > =============================================================================== > > I still have a problem though: when I compile my code the following warning > is issued: > > /u/sciteam/mrosso/LIBS/petsc-3.4.0/arch-test-cray/lib/libpetsc.a(dlimpl.o): In > function `PetscDLOpen': > /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.0/src/sys/dll/dlimpl.c:109: warning: > Using 'dlopen' in statically linked applications requires at runtime the > shared libraries from the glibc version used for linking > /u/sciteam/mrosso/LIBS/petsc-3.4.0/arch-test-cray/lib/libpetsc.a(send.o): In > function `PetscOpenSocket': > /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.0/src/sys/classes/viewer/impls/socket/send.c:97: > warning: Using 'gethostbyname' in statically linked applications requires at > runtime the shared libraries from the glibc version used for linking > > and the run stops because of the following error: > > libhugetlbfs: ERROR: RTLD_NEXT used in code not dynamically loaded Do you get this error with PETSc examples as well? http://www.nersc.gov/users/computational-systems/hopper/running-jobs/trouble-shooting/ says: >>> Make sure to load the craype-hugepages2m module. <<< Satish > > Thanks again for your help. > > Michele > > > On Thu, 6 Jun 2013, Michele Rosso wrote: > > > > > Hi Satish, > > > > > > I tried what you suggested: > > > > > > #!/usr/bin/python > > > if __name__ == '__main__': > > > import sys > > > import os > > > sys.path.insert(0, os.path.abspath('config')) > > > import configure > > > configure_options = [ > > > 'PETSC_DIR=/u/sciteam/mrosso/LIBS/petsc-3.4.0', > > > '--with-cc=cc', > > > '--with-clib-autodetect=0', > > > '--with-cxx=CC', > > > '--with-cxxlib-autodetect=0', > > > '--with-fc=ftn', > > > '--with-fortranlib-autodetect=0', > > > '--with-x=0', > > > '--with-debgging=0', > > > '--COPTFLAGS=-O3', > > > '--FOPTFLAGS=-O3', > > > 'FFLAGS=-F -em', > > > 'LIBS=-L/opt/cray/cce/8.1.7/CC/x86-64/lib/x86-64/ -lcray-c++ -rts > > you have an extra space here ^ It > > should be -lcray-c++-rt > > > > Satish > > > > > > > -lcraystdc++ -lsupc++ -lgcc_eh', > > > 'PETSC_ARCH=arch-test-cray', > > > ] > > > configure.petsc_configure(configure_options) > > > > > > but now I receive another error: > > > > > > "C compiler you provided with -with-cc=cc does not work" > > > > > > I attached the log file. Thanks for your help. > > > > > > Michele > > > > > > > > > On 06/05/2013 02:15 PM, Satish Balay wrote: > > > > Sorry my configure cmd file was from an older test case for c++. > > > > > > > > Do you need c++ build of PETSc? If not - suggest removing > > > > --with-clanguage=C++ option. > > > > > > > > [Majority of use cases should not need --with-clanguage=C++ - even if > > > > user code is using PETSc from C++] > > > > > > > > If c++ is needed - you can try adding: CXXCPPFLAGS='-h gnu' > > > > > > > > Satish > > > > > > > > On Wed, 5 Jun 2013, Michele Rosso wrote: > > > > > > > > > Thank you Satish for your help. > > > > > I followed you instructions, but I am getting some errors during the > > > > > compilation. > > > > > I attached the python script I used to configure PETSc, and the logs > > > > > for > > > > > both > > > > > configure and make. > > > > > > > > > > > > > > > Michele > > > > > > > > > > On 05/31/2013 05:08 PM, Satish Balay wrote: > > > > > > The following is my configure command to build PETSc on a cray with > > > > > > cray compilers: > > > > > > > > > > > > $ cat reconfigure-arch-test-cray.py > > > > > > #!/usr/bin/python > > > > > > if __name__ == '__main__': > > > > > > import sys > > > > > > import os > > > > > > sys.path.insert(0, os.path.abspath('config')) > > > > > > import configure > > > > > > configure_options = [ > > > > > > '--with-cc=cc', > > > > > > '--with-clanguage=C++', > > > > > > '--with-clib-autodetect=0', > > > > > > '--with-cxx=CC', > > > > > > '--with-cxxlib-autodetect=0', > > > > > > '--with-fc=ftn', > > > > > > '--with-fortranlib-autodetect=0', > > > > > > '--with-x=0', > > > > > > 'FFLAGS=-F -em', > > > > > > 'LIBS=-L/opt/cray/cce/8.1.4/CC/x86-64/lib/x86-64/ > > > > > > -lcray-c++-rts > > > > > > -lcraystdc++ -lsupc++ -lgcc_eh', > > > > > > 'PETSC_ARCH=arch-test-cray', > > > > > > ] > > > > > > configure.petsc_configure(configure_options) > > > > > > <<<<<<<<< > > > > > > > > > > > > And you might have to look for the recommended optimization flags - > > > > > > and > > > > > > use > > > > > > with: > > > > > > --with-debgging=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= etc. > > > > > > > > > > > > And on BlueWaters - you would have to remove the following lines > > > > > > from > > > > > > PETSC_ARCH/include/petscconf.h [before runing 'make all' to build > > > > > > the > > > > > > libraries] > > > > > > > > > > > > #ifndef PETSC_HAVE_GETPWUID > > > > > > #define PETSC_HAVE_GETPWUID 1 > > > > > > #endif > > > > > > <<<<< > > > > > > > > > > > > Satish > > > > > > > > > > > > On Fri, 31 May 2013, Michele Rosso wrote: > > > > > > > > > > > > > Thanks, but I'd rather use Cray since only a small part of my code > > > > > > > relies > > > > > > > on > > > > > > > PETSc and, as suggested by the BW staff, > > > > > > > Cray-compiled code performs generally better on Cray system. > > > > > > > > > > > > > > Michele > > > > > > > > > > > > > > On 05/31/2013 02:57 PM, Jed Brown wrote: > > > > > > > > Michele Rosso writes: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I confirm that > > > > > > > > > > > > > > > > > > -pc_type gamg -pc_mg_cycle_type v > > > > > > > > > -pc_gamg_agg_nsmooths 1 > > > > > > > > > > > > > > > > > > produces the correct shift in PETSc 3.4. > > > > > > > > > So my problem is solved. I will upgrade to 3.4 in my > > > > > > > > > productive > > > > > > > > > machine installation. > > > > > > > > > It is a Cray machine (Blue Waters). Assuming I want to use the > > > > > > > > > Cray > > > > > > > > > compiler, which options should I use for ./configure > > > > > > > > > for the installation. > > > > > > > > Or the Intel compiler or GCC. PETSc should run at similar speed > > > > > > > > with > > > > > > > > any. IIRC, Cray recommends the Intel compiler on their machines > > > > > > > > with > > > > > > > > Intel CPUs. There are examples in config/examples/. > > > > > > > > > > > > > From fsantost at student.ethz.ch Fri Jun 7 08:50:14 2013 From: fsantost at student.ethz.ch (Santos Teixeira Frederico) Date: Fri, 7 Jun 2013 13:50:14 +0000 Subject: [petsc-users] No more BLACS in version 3.4? Message-ID: <682CC3CD7A208742B8C2D116C67199010B1932EE@MBX13.d.ethz.ch> Gentlemen, In the version 3.4.0, when I type "./configure --help" I see no reference to BLACS. On the other hand, I found it in the release 3.3-p7. Aren't you supporting BLACS anymore? I did not find reference to this change in the "Changes" too. Regards, Frederico. From jedbrown at mcs.anl.gov Fri Jun 7 08:54:51 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 07 Jun 2013 08:54:51 -0500 Subject: [petsc-users] No more BLACS in version 3.4? In-Reply-To: <682CC3CD7A208742B8C2D116C67199010B1932EE@MBX13.d.ethz.ch> References: <682CC3CD7A208742B8C2D116C67199010B1932EE@MBX13.d.ethz.ch> Message-ID: <87txla9gb8.fsf@mcs.anl.gov> Santos Teixeira Frederico writes: > Gentlemen, > > In the version 3.4.0, when I type "./configure --help" I see no > reference to BLACS. On the other hand, I found it in the release > 3.3-p7. Aren't you supporting BLACS anymore? I did not find reference > to this change in the "Changes" too. BLACS was only ever needed for ScaLAPACK (which in turn is used by MUMPS). Since ScaLAPACK 2.0.2, BLACS is no longer needed as a separate install. No functionality has been lost. From mpovolot at purdue.edu Fri Jun 7 09:08:36 2013 From: mpovolot at purdue.edu (Michael Povolotskyi) Date: Fri, 07 Jun 2013 10:08:36 -0400 Subject: [petsc-users] error while allocating dense distributed matrix In-Reply-To: References: <51B10553.9030700@purdue.edu> Message-ID: <51B1E964.2050706@purdue.edu> Thank you, Barry. What I did I changed the call to ierr = MatCreateMPIDense(comm, m, num_cols ,PETSC_DECIDE,PETSC_DECIDE,data,petsc_matrix) and it works. Do you approve this change? Michael. On 06/06/2013 07:30 PM, Barry Smith wrote: > The first "num_cols" argument refers to the number of "local" columns on that process the second one the total number of columns. With size > 1 they cannot be equal. The sum of all the "local" columns across all processes must equal the total number of columns. > > This is not explained well in the MatCreateDense manual page, sorry. Look at the manual page http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateAIJ.html and see the definition of m and n which explains it much better. The "local" row and column sizes are define by the "left" and "right" parallel vector layouts for vectors compatible with the matrix. > > Barry > > On Jun 6, 2013, at 4:55 PM, Michael Povolotskyi wrote: > >> Dear Petsc developers, >> petsc gives error with the error code equal to 60 when I'm doing the following: >> >> ierr = MatCreateMPIDense(comm, m, num_cols ,num_rows,num_cols,data,petsc_matrix) >> >> on rank = 0 the gebugger gives: >> (gdb) print m >> $1 = 240 >> (gdb) print num_cols >> $2 = 18 >> (gdb) print num_rows >> $3 = 360 >> >> on rank = 1 the gebugger gives: >> ierr = MatCreateMPIDense(comm, m, num_cols ,num_rows,num_cols,data,petsc_matrix); >> (gdb) print m >> $1 = 120 >> (gdb) print num_cols >> $2 = 18 >> (gdb) print num_rows >> $3 = 360 >> (gdb) >> >> Could you, please, tell me what is wrong here? >> Thank you, >> Michael. From jedbrown at mcs.anl.gov Fri Jun 7 09:17:41 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 07 Jun 2013 09:17:41 -0500 Subject: [petsc-users] error while allocating dense distributed matrix In-Reply-To: <51B1E964.2050706@purdue.edu> References: <51B10553.9030700@purdue.edu> <51B1E964.2050706@purdue.edu> Message-ID: <87obbi9f96.fsf@mcs.anl.gov> Michael Povolotskyi writes: > Thank you, Barry. > What I did I changed the call to > > ierr = MatCreateMPIDense(comm, m, num_cols ,PETSC_DECIDE,PETSC_DECIDE,data,petsc_matrix) This is right if 'num_cols' is the local number of columns (i.e., number of owned entries in a vector x, when computing y = A*x). Also, the function has been called MatCreateDense() since petsc-3.3, so please upgrade (to petsc-3.4). From parsani.matteo at gmail.com Fri Jun 7 13:20:13 2013 From: parsani.matteo at gmail.com (Matteo Parsani) Date: Fri, 7 Jun 2013 14:20:13 -0400 Subject: [petsc-users] unexpected ordering when VecSetValues set multiples values In-Reply-To: <039BF5D1-9DBB-4201-916F-4F981FC25E50@mcs.anl.gov> References: <6C9A6BAE-E7B8-4498-8D77-DD23C81334EC@mcs.anl.gov> <039BF5D1-9DBB-4201-916F-4F981FC25E50@mcs.anl.gov> Message-ID: Hello Barry, I have tried to follow the debugger but I got lost. I have run my code in this way: ./NSE -start_in_debugger noxterm then after getting some output I get gdb() Then I type *break vecsetvalues_ * and a break point is generated Breakpoint 1 at 0x2ae97eda5827: file /scratch/home0/pmatteo/research/lib_src/petsc/src/vec/vec/interface/ftn-auto/rvectorf.c, line 239. I keep typing *n* and it goes on. It shows me every time vectsetvalues is called but it never goes inside vectsetvalues. I have tested the VecSetValues with just two elements and it does not work. It sets correctly the values both in serial and in parallel if I set one value at the time. Thanks ** On Tue, Apr 30, 2013 at 8:35 AM, Barry Smith wrote: > > On Apr 30, 2013, at 7:21 AM, Matteo Parsani > wrote: > > > Hello Barry, > > it does not work also if m is not so large. > > Then you need to run in the debugger. You can use the command line > options > > -start_in_debugger noxterm > > in the debugger put a break point in vecsetvalues_ (yes lower case > followed by an underscore) then when it > gets to that point print the size of the array and the values in the > integer and double array to see if they match, then step into the > VecSetValues() function that is called by vecsetvalues_() and look at the > array values ago, continue to step along and it will go into > VecSetValues_Seq() and start actually putting the values into the Vec array. > > Barry > > > > > Thanks again. > > > > > > > > > > > > > > On Fri, Apr 26, 2013 at 3:23 PM, Barry Smith wrote: > > > > Shouldn't matter that it is called from Fortran we do it all the time. > > > > Does it work if the final m is not very large? > > > > You may need to run in the debugger and follow the values through the > code to see why they don't get to where they belong. > > > > > > Barry > > > > Could also be a buggy fortran compiler. > > > > > > On Apr 26, 2013, at 2:06 PM, Matteo Parsani > wrote: > > > > > Hello Barry, > > > sorry I modified few things just before to write the mail. > > > The correct loop with the correct name of the variables is the > following > > > > > > ! Number of DOFs owned by this process > > > len_local = (elem_high-elem_low+1)*nodesperelem*nequations > > > > > > ! Allocate memory for x_vec_local and index_list > > > allocate(x_vec_local(len_local)) > > > allocate(index_list(len_local)) > > > > > > > > > m = 1 > > > ! Loop over all elements > > > do ielem = elem_low, elem_high > > > ! Loop over all nodes in the element > > > do inode = 1, nodesperelem > > > !Loop over all equations > > > do ieq = 1, nequations > > > ! Add element to x_vec_local > > > x_vec_local(m) = ug(ieq,inode,ielem) > > > ! Add element to index list > > > index_list(m) = > (elem_low-1)*nodesperelem*nequations+m-1 > > > ! Update m index > > > m = m+1 > > > end do > > > end do > > > end do > > > > > > ! HERE I HAVE PRINTED x_vec_local, ug and index_list > > > > > > ! Set values in the portion of the vector owned by the process > > > call > VecSetValues(x_vec_in,m-1,index_list,x_vec_local,INSERT_VALUES,& > > > & ierr_local) > > > > > > ! Assemble initial guess > > > call VecAssemblyBegin(x_vec_in,ierr_local) > > > call VecAssemblyEnd(x_vec_in,ierr_local) > > > > > > > > > > > > I have printed the values and the indices I want to pass to > VecSetValues() and they are correct. > > > > > > I also printed the values after VecSetValues() has been called and > they are wrong. > > > > > > The attachment shows that. > > > > > > > > > Could it be a problem of VecSetValues() + F90 when more than 1 > elements is set? > > > > > > I order to debug my the code I am running just with 1 processor. Thus > the process owns all the DOFs. > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 26, 2013 at 2:39 PM, Barry Smith > wrote: > > > > > > On Apr 26, 2013, at 10:20 AM, Matteo Parsani > wrote: > > > > > > > Hello, > > > > I have some problem when I try to set multiple values to a PETSc > vector that I will use later on with SNES. I am using Fortran 90. > > > > Here the problem and two fixes that however are not so good for > performances reasons. The code is very simple. > > > > > > > > Standard approach that does not work correctly: (I am probably doing > something wrong) > > > > > > > > m = 1 > > > > ! Loop over all elements > > > > do ielem = elem_low, elem_high > > > > ! Loop over all nodes in the element > > > > do inode = 1, nodesperelem > > > > !Loop over all equations > > > > do ieq = 1, nequations > > > > ! Add element to x_vec_local > > > > x_vec_local(m) = ug(ieq,inode,ielem) > > > > ! Add element to index list > > > > ind(m) = (elem_low-1)*nodesperelem*nequations+m-1 > > > > ! Update m index > > > > m = m+1 > > > > end do > > > > end do > > > > end do > > > > > > > > ! Set values in the portion of the vector owned by the process > > > > call > VecSetValues(x_vec_in,len_local,index_list,x_vec_local,INSERT_VALUES,& > > > > > > What is len_local and index_list? They do not appear in the loop > above. Shouldn't you be passing m-1 for the length and ind for the indices? > > > > > > I would first print out the all the values in your input to > VecSetValues() and make sure they are correct. > > > > > > Barry > > > > > > > & ierr_local) > > > > > > > > ! Assemble initial guess > > > > call VecAssemblyBegin(x_vec_in,ierr_local) > > > > call VecAssemblyEnd(x_vec_in,ierr_local) > > > > > > > > Then I print my expected values and the values contained in the > PETSc vector to a file. See attachment. I am running in serial for the > moment BUT strangely if you look at the file I have attached the first 79 > DOFs values have a wrong ordering and the remaining 80 are zero. > > > > > > > > > > > > 1st approach: set just one value at the time inside the loop. > > > > m = 1 > > > > ! Loop over all elements > > > > do ielem = elem_low, elem_high > > > > ! Loop over all nodes in the element > > > > do inode = 1, nodesperelem > > > > !Loop over all equations > > > > do ieq = 1, nequations > > > > ! Add element to x_vec_local > > > > value = ug(ieq,inode,ielem) > > > > ! Add element to index list > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 > > > > call > VecSetValues(x_vec_in,1,ind,value,INSERT_VALUES,& > > > > & ierr_local) > > > > ! Update m index > > > > m = m+1 > > > > end do > > > > end do > > > > end do > > > > > > > > > > > > This works fine. As you can see I am using the same expression used > in the previous loop to compute the index of the element that I have to add > in the x_vec_in, i.e. > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 > > > > > > > > Thus I cannot see which is the problem. > > > > > > > > 2nd approach: get the pointer to the local part of the global vector > and use it to set the values in the global vector > > > > > > > > m = 1 > > > > ! Loop over all elements > > > > do ielem = elem_low, elem_high > > > > ! Loop over all nodes in the element > > > > do inode = 1, nodesperelem > > > > !Loop over all equations > > > > do ieq = 1, nequations > > > > ! Add element to x_vec_local > > > > tmp(m) = ug(ieq,inode,ielem) > > > > ! Update m index > > > > m = m+1 > > > > end do > > > > end do > > > > end do > > > > > > > > > > > > This works fine too. > > > > > > > > > > > > Jut to be complete. I use the following two approaches to view the > vector: > > > > > > > > call VecView(x_vec_in,PETSC_VIEWER_STDOUT_WORLD,ierr_local) > > > > > > > > > > > > and > > > > > > > > call VecGetArrayF90(x_vec_in,tmp,ierr_local) > > > > > > > > > > > > m = 1 > > > > ! Loop over all elements > > > > do ielem = elem_low, elem_high > > > > ! Loop over all nodes in the element > > > > do inode = 1, nodesperelem > > > > !Loop over all equations > > > > do ieq = 1, nequations > > > > write(*,*) m,index_list(m),x_vec_local(m),tmp(m) > > > > ! Update m index > > > > m = m+1 > > > > end do > > > > end do > > > > end do > > > > > > > > > > > > Thank you. > > > > > > > > > > > > -- > > > > Matteo > > > > > > > > > > > > > > > > > > > -- > > > Matteo > > > > > > > > > > > > > -- > > Matteo > > -- Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 7 13:32:23 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 7 Jun 2013 13:32:23 -0500 Subject: [petsc-users] unexpected ordering when VecSetValues set multiples values In-Reply-To: References: <6C9A6BAE-E7B8-4498-8D77-DD23C81334EC@mcs.anl.gov> <039BF5D1-9DBB-4201-916F-4F981FC25E50@mcs.anl.gov> Message-ID: <3FA4BA36-1D14-4576-A24A-775761D718D8@mcs.anl.gov> Are, seemingly the correct values getting into the vecsetvalues_ call? Put a break point also in VecSetValues_Seq() and VecSetValues_MPI() when it gets into either of those routines you can do step in the debugger and watch it take the values passed in and attempt to put them in the appropriate places in the vector's array. Barry On Jun 7, 2013, at 1:20 PM, Matteo Parsani wrote: > Hello Barry, > I have tried to follow the debugger but I got lost. > > I have run my code in this way: > > ./NSE -start_in_debugger noxterm > > then after getting some output I get > > gdb() > > Then I type break vecsetvalues_ > > > and a break point is generated > > Breakpoint 1 at 0x2ae97eda5827: file /scratch/home0/pmatteo/research/lib_src/petsc/src/vec/vec/interface/ftn-auto/rvectorf.c, line 239. > > > I keep typing n and it goes on. It shows me every time vectsetvalues is called but it never goes inside vectsetvalues. > > I have tested the VecSetValues with just two elements and it does not work. It sets correctly the values both in serial and in parallel if I set one value at the time. > > Thanks > > > > On Tue, Apr 30, 2013 at 8:35 AM, Barry Smith wrote: > > On Apr 30, 2013, at 7:21 AM, Matteo Parsani wrote: > > > Hello Barry, > > it does not work also if m is not so large. > > Then you need to run in the debugger. You can use the command line options > > -start_in_debugger noxterm > > in the debugger put a break point in vecsetvalues_ (yes lower case followed by an underscore) then when it > gets to that point print the size of the array and the values in the integer and double array to see if they match, then step into the VecSetValues() function that is called by vecsetvalues_() and look at the array values ago, continue to step along and it will go into VecSetValues_Seq() and start actually putting the values into the Vec array. > > Barry > > > > > Thanks again. > > > > > > > > > > > > > > On Fri, Apr 26, 2013 at 3:23 PM, Barry Smith wrote: > > > > Shouldn't matter that it is called from Fortran we do it all the time. > > > > Does it work if the final m is not very large? > > > > You may need to run in the debugger and follow the values through the code to see why they don't get to where they belong. > > > > > > Barry > > > > Could also be a buggy fortran compiler. > > > > > > On Apr 26, 2013, at 2:06 PM, Matteo Parsani wrote: > > > > > Hello Barry, > > > sorry I modified few things just before to write the mail. > > > The correct loop with the correct name of the variables is the following > > > > > > ! Number of DOFs owned by this process > > > len_local = (elem_high-elem_low+1)*nodesperelem*nequations > > > > > > ! Allocate memory for x_vec_local and index_list > > > allocate(x_vec_local(len_local)) > > > allocate(index_list(len_local)) > > > > > > > > > m = 1 > > > ! Loop over all elements > > > do ielem = elem_low, elem_high > > > ! Loop over all nodes in the element > > > do inode = 1, nodesperelem > > > !Loop over all equations > > > do ieq = 1, nequations > > > ! Add element to x_vec_local > > > x_vec_local(m) = ug(ieq,inode,ielem) > > > ! Add element to index list > > > index_list(m) = (elem_low-1)*nodesperelem*nequations+m-1 > > > ! Update m index > > > m = m+1 > > > end do > > > end do > > > end do > > > > > > ! HERE I HAVE PRINTED x_vec_local, ug and index_list > > > > > > ! Set values in the portion of the vector owned by the process > > > call VecSetValues(x_vec_in,m-1,index_list,x_vec_local,INSERT_VALUES,& > > > & ierr_local) > > > > > > ! Assemble initial guess > > > call VecAssemblyBegin(x_vec_in,ierr_local) > > > call VecAssemblyEnd(x_vec_in,ierr_local) > > > > > > > > > > > > I have printed the values and the indices I want to pass to VecSetValues() and they are correct. > > > > > > I also printed the values after VecSetValues() has been called and they are wrong. > > > > > > The attachment shows that. > > > > > > > > > Could it be a problem of VecSetValues() + F90 when more than 1 elements is set? > > > > > > I order to debug my the code I am running just with 1 processor. Thus the process owns all the DOFs. > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 26, 2013 at 2:39 PM, Barry Smith wrote: > > > > > > On Apr 26, 2013, at 10:20 AM, Matteo Parsani wrote: > > > > > > > Hello, > > > > I have some problem when I try to set multiple values to a PETSc vector that I will use later on with SNES. I am using Fortran 90. > > > > Here the problem and two fixes that however are not so good for performances reasons. The code is very simple. > > > > > > > > Standard approach that does not work correctly: (I am probably doing something wrong) > > > > > > > > m = 1 > > > > ! Loop over all elements > > > > do ielem = elem_low, elem_high > > > > ! Loop over all nodes in the element > > > > do inode = 1, nodesperelem > > > > !Loop over all equations > > > > do ieq = 1, nequations > > > > ! Add element to x_vec_local > > > > x_vec_local(m) = ug(ieq,inode,ielem) > > > > ! Add element to index list > > > > ind(m) = (elem_low-1)*nodesperelem*nequations+m-1 > > > > ! Update m index > > > > m = m+1 > > > > end do > > > > end do > > > > end do > > > > > > > > ! Set values in the portion of the vector owned by the process > > > > call VecSetValues(x_vec_in,len_local,index_list,x_vec_local,INSERT_VALUES,& > > > > > > What is len_local and index_list? They do not appear in the loop above. Shouldn't you be passing m-1 for the length and ind for the indices? > > > > > > I would first print out the all the values in your input to VecSetValues() and make sure they are correct. > > > > > > Barry > > > > > > > & ierr_local) > > > > > > > > ! Assemble initial guess > > > > call VecAssemblyBegin(x_vec_in,ierr_local) > > > > call VecAssemblyEnd(x_vec_in,ierr_local) > > > > > > > > Then I print my expected values and the values contained in the PETSc vector to a file. See attachment. I am running in serial for the moment BUT strangely if you look at the file I have attached the first 79 DOFs values have a wrong ordering and the remaining 80 are zero. > > > > > > > > > > > > 1st approach: set just one value at the time inside the loop. > > > > m = 1 > > > > ! Loop over all elements > > > > do ielem = elem_low, elem_high > > > > ! Loop over all nodes in the element > > > > do inode = 1, nodesperelem > > > > !Loop over all equations > > > > do ieq = 1, nequations > > > > ! Add element to x_vec_local > > > > value = ug(ieq,inode,ielem) > > > > ! Add element to index list > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 > > > > call VecSetValues(x_vec_in,1,ind,value,INSERT_VALUES,& > > > > & ierr_local) > > > > ! Update m index > > > > m = m+1 > > > > end do > > > > end do > > > > end do > > > > > > > > > > > > This works fine. As you can see I am using the same expression used in the previous loop to compute the index of the element that I have to add in the x_vec_in, i.e. > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 > > > > > > > > Thus I cannot see which is the problem. > > > > > > > > 2nd approach: get the pointer to the local part of the global vector and use it to set the values in the global vector > > > > > > > > m = 1 > > > > ! Loop over all elements > > > > do ielem = elem_low, elem_high > > > > ! Loop over all nodes in the element > > > > do inode = 1, nodesperelem > > > > !Loop over all equations > > > > do ieq = 1, nequations > > > > ! Add element to x_vec_local > > > > tmp(m) = ug(ieq,inode,ielem) > > > > ! Update m index > > > > m = m+1 > > > > end do > > > > end do > > > > end do > > > > > > > > > > > > This works fine too. > > > > > > > > > > > > Jut to be complete. I use the following two approaches to view the vector: > > > > > > > > call VecView(x_vec_in,PETSC_VIEWER_STDOUT_WORLD,ierr_local) > > > > > > > > > > > > and > > > > > > > > call VecGetArrayF90(x_vec_in,tmp,ierr_local) > > > > > > > > > > > > m = 1 > > > > ! Loop over all elements > > > > do ielem = elem_low, elem_high > > > > ! Loop over all nodes in the element > > > > do inode = 1, nodesperelem > > > > !Loop over all equations > > > > do ieq = 1, nequations > > > > write(*,*) m,index_list(m),x_vec_local(m),tmp(m) > > > > ! Update m index > > > > m = m+1 > > > > end do > > > > end do > > > > end do > > > > > > > > > > > > Thank you. > > > > > > > > > > > > -- > > > > Matteo > > > > > > > > > > > > > > > > > > > -- > > > Matteo > > > > > > > > > > > > > -- > > Matteo > > > > > -- > Matteo From parsani.matteo at gmail.com Fri Jun 7 14:03:37 2013 From: parsani.matteo at gmail.com (Matteo Parsani) Date: Fri, 7 Jun 2013 15:03:37 -0400 Subject: [petsc-users] unexpected ordering when VecSetValues set multiples values In-Reply-To: <3FA4BA36-1D14-4576-A24A-775761D718D8@mcs.anl.gov> References: <6C9A6BAE-E7B8-4498-8D77-DD23C81334EC@mcs.anl.gov> <039BF5D1-9DBB-4201-916F-4F981FC25E50@mcs.anl.gov> <3FA4BA36-1D14-4576-A24A-775761D718D8@mcs.anl.gov> Message-ID: Yes, they are correct. When asked, the debugger prints the right values for bot the vector of the index and the vector of the values. Sorry, how can I set a break point in VecSetValues_Seq() and VecSetValues_MPI()? If I type break VecSetValues_Seq() or break VecSetValues_MPI() or /home0/pmatteo/research/lib_src/petsc/src/vec/vec/impls/seq/bvec2.c:1050 I get Function "vecsetvalues_seq" not defined. or Function "vecsetvalues_mpi" not defined. Thanks On Fri, Jun 7, 2013 at 2:32 PM, Barry Smith wrote: > > Are, seemingly the correct values getting into the vecsetvalues_ call? > > Put a break point also in VecSetValues_Seq() and VecSetValues_MPI() > when it gets into either of those routines you can do step in the debugger > and watch it take the values passed in and attempt to put them in the > appropriate places in the vector's array. > > Barry > > On Jun 7, 2013, at 1:20 PM, Matteo Parsani > wrote: > > > Hello Barry, > > I have tried to follow the debugger but I got lost. > > > > I have run my code in this way: > > > > ./NSE -start_in_debugger noxterm > > > > then after getting some output I get > > > > gdb() > > > > Then I type break vecsetvalues_ > > > > > > and a break point is generated > > > > Breakpoint 1 at 0x2ae97eda5827: file > /scratch/home0/pmatteo/research/lib_src/petsc/src/vec/vec/interface/ftn-auto/rvectorf.c, > line 239. > > > > > > I keep typing n and it goes on. It shows me every time vectsetvalues is > called but it never goes inside vectsetvalues. > > > > I have tested the VecSetValues with just two elements and it does not > work. It sets correctly the values both in serial and in parallel if I set > one value at the time. > > > > Thanks > > > > > > > > On Tue, Apr 30, 2013 at 8:35 AM, Barry Smith wrote: > > > > On Apr 30, 2013, at 7:21 AM, Matteo Parsani > wrote: > > > > > Hello Barry, > > > it does not work also if m is not so large. > > > > Then you need to run in the debugger. You can use the command line > options > > > > -start_in_debugger noxterm > > > > in the debugger put a break point in vecsetvalues_ (yes lower case > followed by an underscore) then when it > > gets to that point print the size of the array and the values in the > integer and double array to see if they match, then step into the > VecSetValues() function that is called by vecsetvalues_() and look at the > array values ago, continue to step along and it will go into > VecSetValues_Seq() and start actually putting the values into the Vec array. > > > > Barry > > > > > > > > Thanks again. > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 26, 2013 at 3:23 PM, Barry Smith > wrote: > > > > > > Shouldn't matter that it is called from Fortran we do it all the > time. > > > > > > Does it work if the final m is not very large? > > > > > > You may need to run in the debugger and follow the values through > the code to see why they don't get to where they belong. > > > > > > > > > Barry > > > > > > Could also be a buggy fortran compiler. > > > > > > > > > On Apr 26, 2013, at 2:06 PM, Matteo Parsani > wrote: > > > > > > > Hello Barry, > > > > sorry I modified few things just before to write the mail. > > > > The correct loop with the correct name of the variables is the > following > > > > > > > > ! Number of DOFs owned by this process > > > > len_local = (elem_high-elem_low+1)*nodesperelem*nequations > > > > > > > > ! Allocate memory for x_vec_local and index_list > > > > allocate(x_vec_local(len_local)) > > > > allocate(index_list(len_local)) > > > > > > > > > > > > m = 1 > > > > ! Loop over all elements > > > > do ielem = elem_low, elem_high > > > > ! Loop over all nodes in the element > > > > do inode = 1, nodesperelem > > > > !Loop over all equations > > > > do ieq = 1, nequations > > > > ! Add element to x_vec_local > > > > x_vec_local(m) = ug(ieq,inode,ielem) > > > > ! Add element to index list > > > > index_list(m) = > (elem_low-1)*nodesperelem*nequations+m-1 > > > > ! Update m index > > > > m = m+1 > > > > end do > > > > end do > > > > end do > > > > > > > > ! HERE I HAVE PRINTED x_vec_local, ug and index_list > > > > > > > > ! Set values in the portion of the vector owned by the > process > > > > call > VecSetValues(x_vec_in,m-1,index_list,x_vec_local,INSERT_VALUES,& > > > > & ierr_local) > > > > > > > > ! Assemble initial guess > > > > call VecAssemblyBegin(x_vec_in,ierr_local) > > > > call VecAssemblyEnd(x_vec_in,ierr_local) > > > > > > > > > > > > > > > > I have printed the values and the indices I want to pass to > VecSetValues() and they are correct. > > > > > > > > I also printed the values after VecSetValues() has been called and > they are wrong. > > > > > > > > The attachment shows that. > > > > > > > > > > > > Could it be a problem of VecSetValues() + F90 when more than 1 > elements is set? > > > > > > > > I order to debug my the code I am running just with 1 processor. > Thus the process owns all the DOFs. > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 26, 2013 at 2:39 PM, Barry Smith > wrote: > > > > > > > > On Apr 26, 2013, at 10:20 AM, Matteo Parsani < > parsani.matteo at gmail.com> wrote: > > > > > > > > > Hello, > > > > > I have some problem when I try to set multiple values to a PETSc > vector that I will use later on with SNES. I am using Fortran 90. > > > > > Here the problem and two fixes that however are not so good for > performances reasons. The code is very simple. > > > > > > > > > > Standard approach that does not work correctly: (I am probably > doing something wrong) > > > > > > > > > > m = 1 > > > > > ! Loop over all elements > > > > > do ielem = elem_low, elem_high > > > > > ! Loop over all nodes in the element > > > > > do inode = 1, nodesperelem > > > > > !Loop over all equations > > > > > do ieq = 1, nequations > > > > > ! Add element to x_vec_local > > > > > x_vec_local(m) = ug(ieq,inode,ielem) > > > > > ! Add element to index list > > > > > ind(m) = > (elem_low-1)*nodesperelem*nequations+m-1 > > > > > ! Update m index > > > > > m = m+1 > > > > > end do > > > > > end do > > > > > end do > > > > > > > > > > ! Set values in the portion of the vector owned by the > process > > > > > call > VecSetValues(x_vec_in,len_local,index_list,x_vec_local,INSERT_VALUES,& > > > > > > > > What is len_local and index_list? They do not appear in the loop > above. Shouldn't you be passing m-1 for the length and ind for the indices? > > > > > > > > I would first print out the all the values in your input to > VecSetValues() and make sure they are correct. > > > > > > > > Barry > > > > > > > > > & ierr_local) > > > > > > > > > > ! Assemble initial guess > > > > > call VecAssemblyBegin(x_vec_in,ierr_local) > > > > > call VecAssemblyEnd(x_vec_in,ierr_local) > > > > > > > > > > Then I print my expected values and the values contained in the > PETSc vector to a file. See attachment. I am running in serial for the > moment BUT strangely if you look at the file I have attached the first 79 > DOFs values have a wrong ordering and the remaining 80 are zero. > > > > > > > > > > > > > > > 1st approach: set just one value at the time inside the loop. > > > > > m = 1 > > > > > ! Loop over all elements > > > > > do ielem = elem_low, elem_high > > > > > ! Loop over all nodes in the element > > > > > do inode = 1, nodesperelem > > > > > !Loop over all equations > > > > > do ieq = 1, nequations > > > > > ! Add element to x_vec_local > > > > > value = ug(ieq,inode,ielem) > > > > > ! Add element to index list > > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 > > > > > call > VecSetValues(x_vec_in,1,ind,value,INSERT_VALUES,& > > > > > & ierr_local) > > > > > ! Update m index > > > > > m = m+1 > > > > > end do > > > > > end do > > > > > end do > > > > > > > > > > > > > > > This works fine. As you can see I am using the same expression > used in the previous loop to compute the index of the element that I have > to add in the x_vec_in, i.e. > > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 > > > > > > > > > > Thus I cannot see which is the problem. > > > > > > > > > > 2nd approach: get the pointer to the local part of the global > vector and use it to set the values in the global vector > > > > > > > > > > m = 1 > > > > > ! Loop over all elements > > > > > do ielem = elem_low, elem_high > > > > > ! Loop over all nodes in the element > > > > > do inode = 1, nodesperelem > > > > > !Loop over all equations > > > > > do ieq = 1, nequations > > > > > ! Add element to x_vec_local > > > > > tmp(m) = ug(ieq,inode,ielem) > > > > > ! Update m index > > > > > m = m+1 > > > > > end do > > > > > end do > > > > > end do > > > > > > > > > > > > > > > This works fine too. > > > > > > > > > > > > > > > Jut to be complete. I use the following two approaches to view the > vector: > > > > > > > > > > call VecView(x_vec_in,PETSC_VIEWER_STDOUT_WORLD,ierr_local) > > > > > > > > > > > > > > > and > > > > > > > > > > call VecGetArrayF90(x_vec_in,tmp,ierr_local) > > > > > > > > > > > > > > > m = 1 > > > > > ! Loop over all elements > > > > > do ielem = elem_low, elem_high > > > > > ! Loop over all nodes in the element > > > > > do inode = 1, nodesperelem > > > > > !Loop over all equations > > > > > do ieq = 1, nequations > > > > > write(*,*) > m,index_list(m),x_vec_local(m),tmp(m) > > > > > ! Update m index > > > > > m = m+1 > > > > > end do > > > > > end do > > > > > end do > > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > -- > > > > > Matteo > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Matteo > > > > > > > > > > > > > > > > > > > -- > > > Matteo > > > > > > > > > > -- > > Matteo > > -- Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From parsani.matteo at gmail.com Fri Jun 7 14:57:16 2013 From: parsani.matteo at gmail.com (Matteo Parsani) Date: Fri, 7 Jun 2013 15:57:16 -0400 Subject: [petsc-users] unexpected ordering when VecSetValues set multiples values In-Reply-To: References: <6C9A6BAE-E7B8-4498-8D77-DD23C81334EC@mcs.anl.gov> <039BF5D1-9DBB-4201-916F-4F981FC25E50@mcs.anl.gov> <3FA4BA36-1D14-4576-A24A-775761D718D8@mcs.anl.gov> Message-ID: Hello Barry, I have tried to pass to VectSetValues two values *call VecSetValues(x_vec_in,2,index_list(1:2),x_vec_local(1:2),INSERT_VALUES,ierr_local) * By looking at the values of index_list and x_vec_local with gdb I see the correct numbers: (gdb) print x_vec_local(1) $1 = 0.9394877605834675 (gdb) print x_vec_local(2) $2 = 0.55294328167143081 (gdb) print index_list(1) $3 = 0 (gdb) print index_list(2) $4 = 1 Now, when I ask gdb to print the values of ni, ix and y in *PetscErrorCode VecSetValues_Seq(Vec xin,PetscInt ni,const PetscInt ix[],const PetscScalar y[],InsertMode m)* I get (gdb) print ni $5 = 2 (gdb) print ix[0] $6 = 0 (gdb) print ix[1] $7 = 0 print y[0] $13 = 0.9394877605834675 (gdb) print y[1] $14 = 0.55294328167143081 The values of y match with the values I am passing in, i.e. x_vec_local. However the index ix is wrong because the first two components of it are zero. Thus, when i = 1 in the VecSetValues_Seq for loop, the first element that was placed correctly is replaced by the second element. This is also waht I see if I use call VecView(x_vec_in,PETSC_VIEWER_STDOUT_WORLD,ierr_local) Now if I keep printing ix in gdb I see the following pattern: (gdb) print ix[0] $18 = 0 (gdb) print ix[1] $19 = 0 (gdb) print ix[2] $20 = 1 (gdb) print ix[3] $21 = 0 (gdb) print ix[4] $22 = 2 (gdb) print ix[5] $23 = 0 (gdb) print ix[6] $24 = 3 which I think is strange since I ix should be a vector of length 2 because I am calling VecSetValues as *call VecSetValues(x_vec_in,2,index_list(1:2),x_vec_local(1:2),INSERT_VALUES,ierr_local) * Am I doing something wrong? Thank you in advance. On Fri, Jun 7, 2013 at 3:03 PM, Matteo Parsani wrote: > Yes, they are correct. When asked, the debugger prints the right values > for bot the vector of the index and the vector of the values. > > Sorry, how can I set a break point in VecSetValues_Seq() and > VecSetValues_MPI()? > If I type > > break VecSetValues_Seq() > > or > > break VecSetValues_MPI() > > or > > /home0/pmatteo/research/lib_src/petsc/src/vec/vec/impls/seq/bvec2.c:1050 > > I get > > Function "vecsetvalues_seq" not defined. > > or > > Function "vecsetvalues_mpi" not defined. > > > Thanks > > > > On Fri, Jun 7, 2013 at 2:32 PM, Barry Smith wrote: > >> >> Are, seemingly the correct values getting into the vecsetvalues_ call? >> >> Put a break point also in VecSetValues_Seq() and VecSetValues_MPI() >> when it gets into either of those routines you can do step in the debugger >> and watch it take the values passed in and attempt to put them in the >> appropriate places in the vector's array. >> >> Barry >> >> On Jun 7, 2013, at 1:20 PM, Matteo Parsani >> wrote: >> >> > Hello Barry, >> > I have tried to follow the debugger but I got lost. >> > >> > I have run my code in this way: >> > >> > ./NSE -start_in_debugger noxterm >> > >> > then after getting some output I get >> > >> > gdb() >> > >> > Then I type break vecsetvalues_ >> > >> > >> > and a break point is generated >> > >> > Breakpoint 1 at 0x2ae97eda5827: file >> /scratch/home0/pmatteo/research/lib_src/petsc/src/vec/vec/interface/ftn-auto/rvectorf.c, >> line 239. >> > >> > >> > I keep typing n and it goes on. It shows me every time vectsetvalues is >> called but it never goes inside vectsetvalues. >> > >> > I have tested the VecSetValues with just two elements and it does not >> work. It sets correctly the values both in serial and in parallel if I set >> one value at the time. >> > >> > Thanks >> > >> > >> > >> > On Tue, Apr 30, 2013 at 8:35 AM, Barry Smith >> wrote: >> > >> > On Apr 30, 2013, at 7:21 AM, Matteo Parsani >> wrote: >> > >> > > Hello Barry, >> > > it does not work also if m is not so large. >> > >> > Then you need to run in the debugger. You can use the command line >> options >> > >> > -start_in_debugger noxterm >> > >> > in the debugger put a break point in vecsetvalues_ (yes lower >> case followed by an underscore) then when it >> > gets to that point print the size of the array and the values in the >> integer and double array to see if they match, then step into the >> VecSetValues() function that is called by vecsetvalues_() and look at the >> array values ago, continue to step along and it will go into >> VecSetValues_Seq() and start actually putting the values into the Vec array. >> > >> > Barry >> > >> > > >> > > Thanks again. >> > > >> > > >> > > >> > > >> > > >> > > >> > > On Fri, Apr 26, 2013 at 3:23 PM, Barry Smith >> wrote: >> > > >> > > Shouldn't matter that it is called from Fortran we do it all the >> time. >> > > >> > > Does it work if the final m is not very large? >> > > >> > > You may need to run in the debugger and follow the values through >> the code to see why they don't get to where they belong. >> > > >> > > >> > > Barry >> > > >> > > Could also be a buggy fortran compiler. >> > > >> > > >> > > On Apr 26, 2013, at 2:06 PM, Matteo Parsani >> wrote: >> > > >> > > > Hello Barry, >> > > > sorry I modified few things just before to write the mail. >> > > > The correct loop with the correct name of the variables is the >> following >> > > > >> > > > ! Number of DOFs owned by this process >> > > > len_local = (elem_high-elem_low+1)*nodesperelem*nequations >> > > > >> > > > ! Allocate memory for x_vec_local and index_list >> > > > allocate(x_vec_local(len_local)) >> > > > allocate(index_list(len_local)) >> > > > >> > > > >> > > > m = 1 >> > > > ! Loop over all elements >> > > > do ielem = elem_low, elem_high >> > > > ! Loop over all nodes in the element >> > > > do inode = 1, nodesperelem >> > > > !Loop over all equations >> > > > do ieq = 1, nequations >> > > > ! Add element to x_vec_local >> > > > x_vec_local(m) = ug(ieq,inode,ielem) >> > > > ! Add element to index list >> > > > index_list(m) = >> (elem_low-1)*nodesperelem*nequations+m-1 >> > > > ! Update m index >> > > > m = m+1 >> > > > end do >> > > > end do >> > > > end do >> > > > >> > > > ! HERE I HAVE PRINTED x_vec_local, ug and index_list >> > > > >> > > > ! Set values in the portion of the vector owned by the >> process >> > > > call >> VecSetValues(x_vec_in,m-1,index_list,x_vec_local,INSERT_VALUES,& >> > > > & ierr_local) >> > > > >> > > > ! Assemble initial guess >> > > > call VecAssemblyBegin(x_vec_in,ierr_local) >> > > > call VecAssemblyEnd(x_vec_in,ierr_local) >> > > > >> > > > >> > > > >> > > > I have printed the values and the indices I want to pass to >> VecSetValues() and they are correct. >> > > > >> > > > I also printed the values after VecSetValues() has been called and >> they are wrong. >> > > > >> > > > The attachment shows that. >> > > > >> > > > >> > > > Could it be a problem of VecSetValues() + F90 when more than 1 >> elements is set? >> > > > >> > > > I order to debug my the code I am running just with 1 processor. >> Thus the process owns all the DOFs. >> > > > >> > > > Thank you. >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > On Fri, Apr 26, 2013 at 2:39 PM, Barry Smith >> wrote: >> > > > >> > > > On Apr 26, 2013, at 10:20 AM, Matteo Parsani < >> parsani.matteo at gmail.com> wrote: >> > > > >> > > > > Hello, >> > > > > I have some problem when I try to set multiple values to a PETSc >> vector that I will use later on with SNES. I am using Fortran 90. >> > > > > Here the problem and two fixes that however are not so good for >> performances reasons. The code is very simple. >> > > > > >> > > > > Standard approach that does not work correctly: (I am probably >> doing something wrong) >> > > > > >> > > > > m = 1 >> > > > > ! Loop over all elements >> > > > > do ielem = elem_low, elem_high >> > > > > ! Loop over all nodes in the element >> > > > > do inode = 1, nodesperelem >> > > > > !Loop over all equations >> > > > > do ieq = 1, nequations >> > > > > ! Add element to x_vec_local >> > > > > x_vec_local(m) = ug(ieq,inode,ielem) >> > > > > ! Add element to index list >> > > > > ind(m) = >> (elem_low-1)*nodesperelem*nequations+m-1 >> > > > > ! Update m index >> > > > > m = m+1 >> > > > > end do >> > > > > end do >> > > > > end do >> > > > > >> > > > > ! Set values in the portion of the vector owned by the >> process >> > > > > call >> VecSetValues(x_vec_in,len_local,index_list,x_vec_local,INSERT_VALUES,& >> > > > >> > > > What is len_local and index_list? They do not appear in the >> loop above. Shouldn't you be passing m-1 for the length and ind for the >> indices? >> > > > >> > > > I would first print out the all the values in your input to >> VecSetValues() and make sure they are correct. >> > > > >> > > > Barry >> > > > >> > > > > & ierr_local) >> > > > > >> > > > > ! Assemble initial guess >> > > > > call VecAssemblyBegin(x_vec_in,ierr_local) >> > > > > call VecAssemblyEnd(x_vec_in,ierr_local) >> > > > > >> > > > > Then I print my expected values and the values contained in the >> PETSc vector to a file. See attachment. I am running in serial for the >> moment BUT strangely if you look at the file I have attached the first 79 >> DOFs values have a wrong ordering and the remaining 80 are zero. >> > > > > >> > > > > >> > > > > 1st approach: set just one value at the time inside the loop. >> > > > > m = 1 >> > > > > ! Loop over all elements >> > > > > do ielem = elem_low, elem_high >> > > > > ! Loop over all nodes in the element >> > > > > do inode = 1, nodesperelem >> > > > > !Loop over all equations >> > > > > do ieq = 1, nequations >> > > > > ! Add element to x_vec_local >> > > > > value = ug(ieq,inode,ielem) >> > > > > ! Add element to index list >> > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 >> > > > > call >> VecSetValues(x_vec_in,1,ind,value,INSERT_VALUES,& >> > > > > & ierr_local) >> > > > > ! Update m index >> > > > > m = m+1 >> > > > > end do >> > > > > end do >> > > > > end do >> > > > > >> > > > > >> > > > > This works fine. As you can see I am using the same expression >> used in the previous loop to compute the index of the element that I have >> to add in the x_vec_in, i.e. >> > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 >> > > > > >> > > > > Thus I cannot see which is the problem. >> > > > > >> > > > > 2nd approach: get the pointer to the local part of the global >> vector and use it to set the values in the global vector >> > > > > >> > > > > m = 1 >> > > > > ! Loop over all elements >> > > > > do ielem = elem_low, elem_high >> > > > > ! Loop over all nodes in the element >> > > > > do inode = 1, nodesperelem >> > > > > !Loop over all equations >> > > > > do ieq = 1, nequations >> > > > > ! Add element to x_vec_local >> > > > > tmp(m) = ug(ieq,inode,ielem) >> > > > > ! Update m index >> > > > > m = m+1 >> > > > > end do >> > > > > end do >> > > > > end do >> > > > > >> > > > > >> > > > > This works fine too. >> > > > > >> > > > > >> > > > > Jut to be complete. I use the following two approaches to view >> the vector: >> > > > > >> > > > > call VecView(x_vec_in,PETSC_VIEWER_STDOUT_WORLD,ierr_local) >> > > > > >> > > > > >> > > > > and >> > > > > >> > > > > call VecGetArrayF90(x_vec_in,tmp,ierr_local) >> > > > > >> > > > > >> > > > > m = 1 >> > > > > ! Loop over all elements >> > > > > do ielem = elem_low, elem_high >> > > > > ! Loop over all nodes in the element >> > > > > do inode = 1, nodesperelem >> > > > > !Loop over all equations >> > > > > do ieq = 1, nequations >> > > > > write(*,*) >> m,index_list(m),x_vec_local(m),tmp(m) >> > > > > ! Update m index >> > > > > m = m+1 >> > > > > end do >> > > > > end do >> > > > > end do >> > > > > >> > > > > >> > > > > Thank you. >> > > > > >> > > > > >> > > > > -- >> > > > > Matteo >> > > > > >> > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > Matteo >> > > > >> > > >> > > >> > > >> > > >> > > -- >> > > Matteo >> > >> > >> > >> > >> > -- >> > Matteo >> >> > > > -- > Matteo > -- Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 7 15:08:44 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 7 Jun 2013 15:08:44 -0500 Subject: [petsc-users] unexpected ordering when VecSetValues set multiples values In-Reply-To: References: <6C9A6BAE-E7B8-4498-8D77-DD23C81334EC@mcs.anl.gov> <039BF5D1-9DBB-4201-916F-4F981FC25E50@mcs.anl.gov> <3FA4BA36-1D14-4576-A24A-775761D718D8@mcs.anl.gov> Message-ID: How large is your integer on the Fortran side and the C side? Normally a PetscInt is a C int and a Fortran integer*4 which are each 4 bytes long. If PETSc is configured with --with-64-bit-indices then PetscInt is a C long long int and Fortran integer*8 both 8 bytes long. It looks like you are using 8 byte Fortran integers but have not compiled PETSc with --with-64-bit-indices Possibly you are using come fortran compile time flag that "promotes" float to double and integer*4 to integer*8 in Fortran. 1) we recommend never using any fortran compiler flag that promotes types automatically. 2) likely you just want to use 4 byte integers in fortran. If you want to use 8 bytes then configure PETSc with --with-64-bit-indices Barry On Jun 7, 2013, at 2:57 PM, Matteo Parsani wrote: > Hello Barry, > I have tried to pass to VectSetValues two values > > call VecSetValues(x_vec_in,2,index_list(1:2),x_vec_local(1:2),INSERT_VALUES,ierr_local) > > By looking at the values of index_list and x_vec_local with gdb I see the correct numbers: > > (gdb) print x_vec_local(1) > $1 = 0.9394877605834675 > (gdb) print x_vec_local(2) > $2 = 0.55294328167143081 > > (gdb) print index_list(1) > $3 = 0 > (gdb) print index_list(2) > $4 = 1 > > Now, when I ask gdb to print the values of ni, ix and y in > > PetscErrorCode VecSetValues_Seq(Vec xin,PetscInt ni,const PetscInt ix[],const PetscScalar y[],InsertMode m) > > > I get > > (gdb) print ni > $5 = 2 > > (gdb) print ix[0] > $6 = 0 > (gdb) print ix[1] > $7 = 0 > > > print y[0] > $13 = 0.9394877605834675 > (gdb) print y[1] > $14 = 0.55294328167143081 > > > The values of y match with the values I am passing in, i.e. x_vec_local. However the index ix is wrong because the first two components of it are zero. Thus, when i = 1 in the VecSetValues_Seq for loop, the first element that was placed correctly is replaced by the second element. This is also waht I see if I use > > call VecView(x_vec_in,PETSC_VIEWER_STDOUT_WORLD,ierr_local) > > Now if I keep printing ix in gdb I see the following pattern: > > (gdb) print ix[0] > $18 = 0 > (gdb) print ix[1] > $19 = 0 > (gdb) print ix[2] > $20 = 1 > (gdb) print ix[3] > $21 = 0 > (gdb) print ix[4] > $22 = 2 > (gdb) print ix[5] > $23 = 0 > (gdb) print ix[6] > $24 = 3 > > > which I think is strange since I ix should be a vector of length 2 because I am calling VecSetValues as > > > call VecSetValues(x_vec_in,2,index_list(1:2),x_vec_local(1:2),INSERT_VALUES,ierr_local) > > Am I doing something wrong? > > Thank you in advance. > > > > > > On Fri, Jun 7, 2013 at 3:03 PM, Matteo Parsani wrote: > Yes, they are correct. When asked, the debugger prints the right values for bot the vector of the index and the vector of the values. > > Sorry, how can I set a break point in VecSetValues_Seq() and VecSetValues_MPI()? > If I type > > break VecSetValues_Seq() > > or > > break VecSetValues_MPI() > > or > > /home0/pmatteo/research/lib_src/petsc/src/vec/vec/impls/seq/bvec2.c:1050 > > I get > > Function "vecsetvalues_seq" not defined. > > or > > Function "vecsetvalues_mpi" not defined. > > > Thanks > > > > On Fri, Jun 7, 2013 at 2:32 PM, Barry Smith wrote: > > Are, seemingly the correct values getting into the vecsetvalues_ call? > > Put a break point also in VecSetValues_Seq() and VecSetValues_MPI() when it gets into either of those routines you can do step in the debugger and watch it take the values passed in and attempt to put them in the appropriate places in the vector's array. > > Barry > > On Jun 7, 2013, at 1:20 PM, Matteo Parsani wrote: > > > Hello Barry, > > I have tried to follow the debugger but I got lost. > > > > I have run my code in this way: > > > > ./NSE -start_in_debugger noxterm > > > > then after getting some output I get > > > > gdb() > > > > Then I type break vecsetvalues_ > > > > > > and a break point is generated > > > > Breakpoint 1 at 0x2ae97eda5827: file /scratch/home0/pmatteo/research/lib_src/petsc/src/vec/vec/interface/ftn-auto/rvectorf.c, line 239. > > > > > > I keep typing n and it goes on. It shows me every time vectsetvalues is called but it never goes inside vectsetvalues. > > > > I have tested the VecSetValues with just two elements and it does not work. It sets correctly the values both in serial and in parallel if I set one value at the time. > > > > Thanks > > > > > > > > On Tue, Apr 30, 2013 at 8:35 AM, Barry Smith wrote: > > > > On Apr 30, 2013, at 7:21 AM, Matteo Parsani wrote: > > > > > Hello Barry, > > > it does not work also if m is not so large. > > > > Then you need to run in the debugger. You can use the command line options > > > > -start_in_debugger noxterm > > > > in the debugger put a break point in vecsetvalues_ (yes lower case followed by an underscore) then when it > > gets to that point print the size of the array and the values in the integer and double array to see if they match, then step into the VecSetValues() function that is called by vecsetvalues_() and look at the array values ago, continue to step along and it will go into VecSetValues_Seq() and start actually putting the values into the Vec array. > > > > Barry > > > > > > > > Thanks again. > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 26, 2013 at 3:23 PM, Barry Smith wrote: > > > > > > Shouldn't matter that it is called from Fortran we do it all the time. > > > > > > Does it work if the final m is not very large? > > > > > > You may need to run in the debugger and follow the values through the code to see why they don't get to where they belong. > > > > > > > > > Barry > > > > > > Could also be a buggy fortran compiler. > > > > > > > > > On Apr 26, 2013, at 2:06 PM, Matteo Parsani wrote: > > > > > > > Hello Barry, > > > > sorry I modified few things just before to write the mail. > > > > The correct loop with the correct name of the variables is the following > > > > > > > > ! Number of DOFs owned by this process > > > > len_local = (elem_high-elem_low+1)*nodesperelem*nequations > > > > > > > > ! Allocate memory for x_vec_local and index_list > > > > allocate(x_vec_local(len_local)) > > > > allocate(index_list(len_local)) > > > > > > > > > > > > m = 1 > > > > ! Loop over all elements > > > > do ielem = elem_low, elem_high > > > > ! Loop over all nodes in the element > > > > do inode = 1, nodesperelem > > > > !Loop over all equations > > > > do ieq = 1, nequations > > > > ! Add element to x_vec_local > > > > x_vec_local(m) = ug(ieq,inode,ielem) > > > > ! Add element to index list > > > > index_list(m) = (elem_low-1)*nodesperelem*nequations+m-1 > > > > ! Update m index > > > > m = m+1 > > > > end do > > > > end do > > > > end do > > > > > > > > ! HERE I HAVE PRINTED x_vec_local, ug and index_list > > > > > > > > ! Set values in the portion of the vector owned by the process > > > > call VecSetValues(x_vec_in,m-1,index_list,x_vec_local,INSERT_VALUES,& > > > > & ierr_local) > > > > > > > > ! Assemble initial guess > > > > call VecAssemblyBegin(x_vec_in,ierr_local) > > > > call VecAssemblyEnd(x_vec_in,ierr_local) > > > > > > > > > > > > > > > > I have printed the values and the indices I want to pass to VecSetValues() and they are correct. > > > > > > > > I also printed the values after VecSetValues() has been called and they are wrong. > > > > > > > > The attachment shows that. > > > > > > > > > > > > Could it be a problem of VecSetValues() + F90 when more than 1 elements is set? > > > > > > > > I order to debug my the code I am running just with 1 processor. Thus the process owns all the DOFs. > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 26, 2013 at 2:39 PM, Barry Smith wrote: > > > > > > > > On Apr 26, 2013, at 10:20 AM, Matteo Parsani wrote: > > > > > > > > > Hello, > > > > > I have some problem when I try to set multiple values to a PETSc vector that I will use later on with SNES. I am using Fortran 90. > > > > > Here the problem and two fixes that however are not so good for performances reasons. The code is very simple. > > > > > > > > > > Standard approach that does not work correctly: (I am probably doing something wrong) > > > > > > > > > > m = 1 > > > > > ! Loop over all elements > > > > > do ielem = elem_low, elem_high > > > > > ! Loop over all nodes in the element > > > > > do inode = 1, nodesperelem > > > > > !Loop over all equations > > > > > do ieq = 1, nequations > > > > > ! Add element to x_vec_local > > > > > x_vec_local(m) = ug(ieq,inode,ielem) > > > > > ! Add element to index list > > > > > ind(m) = (elem_low-1)*nodesperelem*nequations+m-1 > > > > > ! Update m index > > > > > m = m+1 > > > > > end do > > > > > end do > > > > > end do > > > > > > > > > > ! Set values in the portion of the vector owned by the process > > > > > call VecSetValues(x_vec_in,len_local,index_list,x_vec_local,INSERT_VALUES,& > > > > > > > > What is len_local and index_list? They do not appear in the loop above. Shouldn't you be passing m-1 for the length and ind for the indices? > > > > > > > > I would first print out the all the values in your input to VecSetValues() and make sure they are correct. > > > > > > > > Barry > > > > > > > > > & ierr_local) > > > > > > > > > > ! Assemble initial guess > > > > > call VecAssemblyBegin(x_vec_in,ierr_local) > > > > > call VecAssemblyEnd(x_vec_in,ierr_local) > > > > > > > > > > Then I print my expected values and the values contained in the PETSc vector to a file. See attachment. I am running in serial for the moment BUT strangely if you look at the file I have attached the first 79 DOFs values have a wrong ordering and the remaining 80 are zero. > > > > > > > > > > > > > > > 1st approach: set just one value at the time inside the loop. > > > > > m = 1 > > > > > ! Loop over all elements > > > > > do ielem = elem_low, elem_high > > > > > ! Loop over all nodes in the element > > > > > do inode = 1, nodesperelem > > > > > !Loop over all equations > > > > > do ieq = 1, nequations > > > > > ! Add element to x_vec_local > > > > > value = ug(ieq,inode,ielem) > > > > > ! Add element to index list > > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 > > > > > call VecSetValues(x_vec_in,1,ind,value,INSERT_VALUES,& > > > > > & ierr_local) > > > > > ! Update m index > > > > > m = m+1 > > > > > end do > > > > > end do > > > > > end do > > > > > > > > > > > > > > > This works fine. As you can see I am using the same expression used in the previous loop to compute the index of the element that I have to add in the x_vec_in, i.e. > > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 > > > > > > > > > > Thus I cannot see which is the problem. > > > > > > > > > > 2nd approach: get the pointer to the local part of the global vector and use it to set the values in the global vector > > > > > > > > > > m = 1 > > > > > ! Loop over all elements > > > > > do ielem = elem_low, elem_high > > > > > ! Loop over all nodes in the element > > > > > do inode = 1, nodesperelem > > > > > !Loop over all equations > > > > > do ieq = 1, nequations > > > > > ! Add element to x_vec_local > > > > > tmp(m) = ug(ieq,inode,ielem) > > > > > ! Update m index > > > > > m = m+1 > > > > > end do > > > > > end do > > > > > end do > > > > > > > > > > > > > > > This works fine too. > > > > > > > > > > > > > > > Jut to be complete. I use the following two approaches to view the vector: > > > > > > > > > > call VecView(x_vec_in,PETSC_VIEWER_STDOUT_WORLD,ierr_local) > > > > > > > > > > > > > > > and > > > > > > > > > > call VecGetArrayF90(x_vec_in,tmp,ierr_local) > > > > > > > > > > > > > > > m = 1 > > > > > ! Loop over all elements > > > > > do ielem = elem_low, elem_high > > > > > ! Loop over all nodes in the element > > > > > do inode = 1, nodesperelem > > > > > !Loop over all equations > > > > > do ieq = 1, nequations > > > > > write(*,*) m,index_list(m),x_vec_local(m),tmp(m) > > > > > ! Update m index > > > > > m = m+1 > > > > > end do > > > > > end do > > > > > end do > > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > -- > > > > > Matteo > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Matteo > > > > > > > > > > > > > > > > > > > -- > > > Matteo > > > > > > > > > > -- > > Matteo > > > > > -- > Matteo > > > > -- > Matteo From balay at mcs.anl.gov Fri Jun 7 15:14:22 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 7 Jun 2013 15:14:22 -0500 (CDT) Subject: [petsc-users] unexpected ordering when VecSetValues set multiples values In-Reply-To: References: <6C9A6BAE-E7B8-4498-8D77-DD23C81334EC@mcs.anl.gov> <039BF5D1-9DBB-4201-916F-4F981FC25E50@mcs.anl.gov> <3FA4BA36-1D14-4576-A24A-775761D718D8@mcs.anl.gov> Message-ID: On Fri, 7 Jun 2013, Barry Smith wrote: > > How large is your integer on the Fortran side and the C side? > > Normally a PetscInt is a C int and a Fortran integer*4 which are each 4 bytes long. > > If PETSc is configured with --with-64-bit-indices then PetscInt is a C long long int and Fortran integer*8 both 8 bytes long. > > It looks like you are using 8 byte Fortran integers but have not compiled PETSc with --with-64-bit-indices Possibly you are using come fortran compile time flag that "promotes" float to double and integer*4 to integer*8 in Fortran. > > 1) we recommend never using any fortran compiler flag that promotes types automatically. If Petsc datatypes are used in fortran code [i.e declare variable as PetscInt - and not integer for functions that expect PetscInt] - then the code [from PETSc part] would be resiliant to -i8 type options. Satish > > 2) likely you just want to use 4 byte integers in fortran. If you want to use 8 bytes then configure PETSc with --with-64-bit-indices > > Barry > > > > > On Jun 7, 2013, at 2:57 PM, Matteo Parsani wrote: > > > Hello Barry, > > I have tried to pass to VectSetValues two values > > > > call VecSetValues(x_vec_in,2,index_list(1:2),x_vec_local(1:2),INSERT_VALUES,ierr_local) > > > > By looking at the values of index_list and x_vec_local with gdb I see the correct numbers: > > > > (gdb) print x_vec_local(1) > > $1 = 0.9394877605834675 > > (gdb) print x_vec_local(2) > > $2 = 0.55294328167143081 > > > > (gdb) print index_list(1) > > $3 = 0 > > (gdb) print index_list(2) > > $4 = 1 > > > > Now, when I ask gdb to print the values of ni, ix and y in > > > > PetscErrorCode VecSetValues_Seq(Vec xin,PetscInt ni,const PetscInt ix[],const PetscScalar y[],InsertMode m) > > > > > > I get > > > > (gdb) print ni > > $5 = 2 > > > > (gdb) print ix[0] > > $6 = 0 > > (gdb) print ix[1] > > $7 = 0 > > > > > > print y[0] > > $13 = 0.9394877605834675 > > (gdb) print y[1] > > $14 = 0.55294328167143081 > > > > > > The values of y match with the values I am passing in, i.e. x_vec_local. However the index ix is wrong because the first two components of it are zero. Thus, when i = 1 in the VecSetValues_Seq for loop, the first element that was placed correctly is replaced by the second element. This is also waht I see if I use > > > > call VecView(x_vec_in,PETSC_VIEWER_STDOUT_WORLD,ierr_local) > > > > Now if I keep printing ix in gdb I see the following pattern: > > > > (gdb) print ix[0] > > $18 = 0 > > (gdb) print ix[1] > > $19 = 0 > > (gdb) print ix[2] > > $20 = 1 > > (gdb) print ix[3] > > $21 = 0 > > (gdb) print ix[4] > > $22 = 2 > > (gdb) print ix[5] > > $23 = 0 > > (gdb) print ix[6] > > $24 = 3 > > > > > > which I think is strange since I ix should be a vector of length 2 because I am calling VecSetValues as > > > > > > call VecSetValues(x_vec_in,2,index_list(1:2),x_vec_local(1:2),INSERT_VALUES,ierr_local) > > > > Am I doing something wrong? > > > > Thank you in advance. > > > > > > > > > > > > On Fri, Jun 7, 2013 at 3:03 PM, Matteo Parsani wrote: > > Yes, they are correct. When asked, the debugger prints the right values for bot the vector of the index and the vector of the values. > > > > Sorry, how can I set a break point in VecSetValues_Seq() and VecSetValues_MPI()? > > If I type > > > > break VecSetValues_Seq() > > > > or > > > > break VecSetValues_MPI() > > > > or > > > > /home0/pmatteo/research/lib_src/petsc/src/vec/vec/impls/seq/bvec2.c:1050 > > > > I get > > > > Function "vecsetvalues_seq" not defined. > > > > or > > > > Function "vecsetvalues_mpi" not defined. > > > > > > Thanks > > > > > > > > On Fri, Jun 7, 2013 at 2:32 PM, Barry Smith wrote: > > > > Are, seemingly the correct values getting into the vecsetvalues_ call? > > > > Put a break point also in VecSetValues_Seq() and VecSetValues_MPI() when it gets into either of those routines you can do step in the debugger and watch it take the values passed in and attempt to put them in the appropriate places in the vector's array. > > > > Barry > > > > On Jun 7, 2013, at 1:20 PM, Matteo Parsani wrote: > > > > > Hello Barry, > > > I have tried to follow the debugger but I got lost. > > > > > > I have run my code in this way: > > > > > > ./NSE -start_in_debugger noxterm > > > > > > then after getting some output I get > > > > > > gdb() > > > > > > Then I type break vecsetvalues_ > > > > > > > > > and a break point is generated > > > > > > Breakpoint 1 at 0x2ae97eda5827: file /scratch/home0/pmatteo/research/lib_src/petsc/src/vec/vec/interface/ftn-auto/rvectorf.c, line 239. > > > > > > > > > I keep typing n and it goes on. It shows me every time vectsetvalues is called but it never goes inside vectsetvalues. > > > > > > I have tested the VecSetValues with just two elements and it does not work. It sets correctly the values both in serial and in parallel if I set one value at the time. > > > > > > Thanks > > > > > > > > > > > > On Tue, Apr 30, 2013 at 8:35 AM, Barry Smith wrote: > > > > > > On Apr 30, 2013, at 7:21 AM, Matteo Parsani wrote: > > > > > > > Hello Barry, > > > > it does not work also if m is not so large. > > > > > > Then you need to run in the debugger. You can use the command line options > > > > > > -start_in_debugger noxterm > > > > > > in the debugger put a break point in vecsetvalues_ (yes lower case followed by an underscore) then when it > > > gets to that point print the size of the array and the values in the integer and double array to see if they match, then step into the VecSetValues() function that is called by vecsetvalues_() and look at the array values ago, continue to step along and it will go into VecSetValues_Seq() and start actually putting the values into the Vec array. > > > > > > Barry > > > > > > > > > > > Thanks again. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 26, 2013 at 3:23 PM, Barry Smith wrote: > > > > > > > > Shouldn't matter that it is called from Fortran we do it all the time. > > > > > > > > Does it work if the final m is not very large? > > > > > > > > You may need to run in the debugger and follow the values through the code to see why they don't get to where they belong. > > > > > > > > > > > > Barry > > > > > > > > Could also be a buggy fortran compiler. > > > > > > > > > > > > On Apr 26, 2013, at 2:06 PM, Matteo Parsani wrote: > > > > > > > > > Hello Barry, > > > > > sorry I modified few things just before to write the mail. > > > > > The correct loop with the correct name of the variables is the following > > > > > > > > > > ! Number of DOFs owned by this process > > > > > len_local = (elem_high-elem_low+1)*nodesperelem*nequations > > > > > > > > > > ! Allocate memory for x_vec_local and index_list > > > > > allocate(x_vec_local(len_local)) > > > > > allocate(index_list(len_local)) > > > > > > > > > > > > > > > m = 1 > > > > > ! Loop over all elements > > > > > do ielem = elem_low, elem_high > > > > > ! Loop over all nodes in the element > > > > > do inode = 1, nodesperelem > > > > > !Loop over all equations > > > > > do ieq = 1, nequations > > > > > ! Add element to x_vec_local > > > > > x_vec_local(m) = ug(ieq,inode,ielem) > > > > > ! Add element to index list > > > > > index_list(m) = (elem_low-1)*nodesperelem*nequations+m-1 > > > > > ! Update m index > > > > > m = m+1 > > > > > end do > > > > > end do > > > > > end do > > > > > > > > > > ! HERE I HAVE PRINTED x_vec_local, ug and index_list > > > > > > > > > > ! Set values in the portion of the vector owned by the process > > > > > call VecSetValues(x_vec_in,m-1,index_list,x_vec_local,INSERT_VALUES,& > > > > > & ierr_local) > > > > > > > > > > ! Assemble initial guess > > > > > call VecAssemblyBegin(x_vec_in,ierr_local) > > > > > call VecAssemblyEnd(x_vec_in,ierr_local) > > > > > > > > > > > > > > > > > > > > I have printed the values and the indices I want to pass to VecSetValues() and they are correct. > > > > > > > > > > I also printed the values after VecSetValues() has been called and they are wrong. > > > > > > > > > > The attachment shows that. > > > > > > > > > > > > > > > Could it be a problem of VecSetValues() + F90 when more than 1 elements is set? > > > > > > > > > > I order to debug my the code I am running just with 1 processor. Thus the process owns all the DOFs. > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 26, 2013 at 2:39 PM, Barry Smith wrote: > > > > > > > > > > On Apr 26, 2013, at 10:20 AM, Matteo Parsani wrote: > > > > > > > > > > > Hello, > > > > > > I have some problem when I try to set multiple values to a PETSc vector that I will use later on with SNES. I am using Fortran 90. > > > > > > Here the problem and two fixes that however are not so good for performances reasons. The code is very simple. > > > > > > > > > > > > Standard approach that does not work correctly: (I am probably doing something wrong) > > > > > > > > > > > > m = 1 > > > > > > ! Loop over all elements > > > > > > do ielem = elem_low, elem_high > > > > > > ! Loop over all nodes in the element > > > > > > do inode = 1, nodesperelem > > > > > > !Loop over all equations > > > > > > do ieq = 1, nequations > > > > > > ! Add element to x_vec_local > > > > > > x_vec_local(m) = ug(ieq,inode,ielem) > > > > > > ! Add element to index list > > > > > > ind(m) = (elem_low-1)*nodesperelem*nequations+m-1 > > > > > > ! Update m index > > > > > > m = m+1 > > > > > > end do > > > > > > end do > > > > > > end do > > > > > > > > > > > > ! Set values in the portion of the vector owned by the process > > > > > > call VecSetValues(x_vec_in,len_local,index_list,x_vec_local,INSERT_VALUES,& > > > > > > > > > > What is len_local and index_list? They do not appear in the loop above. Shouldn't you be passing m-1 for the length and ind for the indices? > > > > > > > > > > I would first print out the all the values in your input to VecSetValues() and make sure they are correct. > > > > > > > > > > Barry > > > > > > > > > > > & ierr_local) > > > > > > > > > > > > ! Assemble initial guess > > > > > > call VecAssemblyBegin(x_vec_in,ierr_local) > > > > > > call VecAssemblyEnd(x_vec_in,ierr_local) > > > > > > > > > > > > Then I print my expected values and the values contained in the PETSc vector to a file. See attachment. I am running in serial for the moment BUT strangely if you look at the file I have attached the first 79 DOFs values have a wrong ordering and the remaining 80 are zero. > > > > > > > > > > > > > > > > > > 1st approach: set just one value at the time inside the loop. > > > > > > m = 1 > > > > > > ! Loop over all elements > > > > > > do ielem = elem_low, elem_high > > > > > > ! Loop over all nodes in the element > > > > > > do inode = 1, nodesperelem > > > > > > !Loop over all equations > > > > > > do ieq = 1, nequations > > > > > > ! Add element to x_vec_local > > > > > > value = ug(ieq,inode,ielem) > > > > > > ! Add element to index list > > > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 > > > > > > call VecSetValues(x_vec_in,1,ind,value,INSERT_VALUES,& > > > > > > & ierr_local) > > > > > > ! Update m index > > > > > > m = m+1 > > > > > > end do > > > > > > end do > > > > > > end do > > > > > > > > > > > > > > > > > > This works fine. As you can see I am using the same expression used in the previous loop to compute the index of the element that I have to add in the x_vec_in, i.e. > > > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 > > > > > > > > > > > > Thus I cannot see which is the problem. > > > > > > > > > > > > 2nd approach: get the pointer to the local part of the global vector and use it to set the values in the global vector > > > > > > > > > > > > m = 1 > > > > > > ! Loop over all elements > > > > > > do ielem = elem_low, elem_high > > > > > > ! Loop over all nodes in the element > > > > > > do inode = 1, nodesperelem > > > > > > !Loop over all equations > > > > > > do ieq = 1, nequations > > > > > > ! Add element to x_vec_local > > > > > > tmp(m) = ug(ieq,inode,ielem) > > > > > > ! Update m index > > > > > > m = m+1 > > > > > > end do > > > > > > end do > > > > > > end do > > > > > > > > > > > > > > > > > > This works fine too. > > > > > > > > > > > > > > > > > > Jut to be complete. I use the following two approaches to view the vector: > > > > > > > > > > > > call VecView(x_vec_in,PETSC_VIEWER_STDOUT_WORLD,ierr_local) > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > call VecGetArrayF90(x_vec_in,tmp,ierr_local) > > > > > > > > > > > > > > > > > > m = 1 > > > > > > ! Loop over all elements > > > > > > do ielem = elem_low, elem_high > > > > > > ! Loop over all nodes in the element > > > > > > do inode = 1, nodesperelem > > > > > > !Loop over all equations > > > > > > do ieq = 1, nequations > > > > > > write(*,*) m,index_list(m),x_vec_local(m),tmp(m) > > > > > > ! Update m index > > > > > > m = m+1 > > > > > > end do > > > > > > end do > > > > > > end do > > > > > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > -- > > > > > > Matteo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Matteo > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Matteo > > > > > > > > > > > > > > > -- > > > Matteo > > > > > > > > > > -- > > Matteo > > > > > > > > -- > > Matteo > > From ckontzialis at lycos.com Mon Jun 10 10:14:32 2013 From: ckontzialis at lycos.com (Konstantinos Kontzialis) Date: Mon, 10 Jun 2013 12:14:32 -0300 Subject: [petsc-users] CGNS Message-ID: <51B5ED58.3010405@lycos.com> Dear all, How do I install PETSc with support for cgns? Kostas From parsani.matteo at gmail.com Mon Jun 10 10:53:15 2013 From: parsani.matteo at gmail.com (Matteo Parsani) Date: Mon, 10 Jun 2013 11:53:15 -0400 Subject: [petsc-users] unexpected ordering when VecSetValues set multiples values In-Reply-To: References: <6C9A6BAE-E7B8-4498-8D77-DD23C81334EC@mcs.anl.gov> <039BF5D1-9DBB-4201-916F-4F981FC25E50@mcs.anl.gov> <3FA4BA36-1D14-4576-A24A-775761D718D8@mcs.anl.gov> Message-ID: Thanks Barry and Satish. As you said I have a wrong flag in my makefile. Best Regards On Fri, Jun 7, 2013 at 4:14 PM, Satish Balay wrote: > On Fri, 7 Jun 2013, Barry Smith wrote: > > > > > How large is your integer on the Fortran side and the C side? > > > > Normally a PetscInt is a C int and a Fortran integer*4 which are > each 4 bytes long. > > > > If PETSc is configured with --with-64-bit-indices then PetscInt is a > C long long int and Fortran integer*8 both 8 bytes long. > > > > It looks like you are using 8 byte Fortran integers but have not > compiled PETSc with --with-64-bit-indices Possibly you are using come > fortran compile time flag that "promotes" float to double and integer*4 to > integer*8 in Fortran. > > > > 1) we recommend never using any fortran compiler flag that promotes > types automatically. > > If Petsc datatypes are used in fortran code [i.e declare variable as > PetscInt - and not integer for functions that expect PetscInt] - then > the code [from PETSc part] would be resiliant to -i8 type options. > > Satish > > > > > 2) likely you just want to use 4 byte integers in fortran. If you want > to use 8 bytes then configure PETSc with --with-64-bit-indices > > > > Barry > > > > > > > > > > On Jun 7, 2013, at 2:57 PM, Matteo Parsani > wrote: > > > > > Hello Barry, > > > I have tried to pass to VectSetValues two values > > > > > > call > VecSetValues(x_vec_in,2,index_list(1:2),x_vec_local(1:2),INSERT_VALUES,ierr_local) > > > > > > By looking at the values of index_list and x_vec_local with gdb I see > the correct numbers: > > > > > > (gdb) print x_vec_local(1) > > > $1 = 0.9394877605834675 > > > (gdb) print x_vec_local(2) > > > $2 = 0.55294328167143081 > > > > > > (gdb) print index_list(1) > > > $3 = 0 > > > (gdb) print index_list(2) > > > $4 = 1 > > > > > > Now, when I ask gdb to print the values of ni, ix and y in > > > > > > PetscErrorCode VecSetValues_Seq(Vec xin,PetscInt ni,const PetscInt > ix[],const PetscScalar y[],InsertMode m) > > > > > > > > > I get > > > > > > (gdb) print ni > > > $5 = 2 > > > > > > (gdb) print ix[0] > > > $6 = 0 > > > (gdb) print ix[1] > > > $7 = 0 > > > > > > > > > print y[0] > > > $13 = 0.9394877605834675 > > > (gdb) print y[1] > > > $14 = 0.55294328167143081 > > > > > > > > > The values of y match with the values I am passing in, i.e. > x_vec_local. However the index ix is wrong because the first two components > of it are zero. Thus, when i = 1 in the VecSetValues_Seq for loop, the > first element that was placed correctly is replaced by the second element. > This is also waht I see if I use > > > > > > call VecView(x_vec_in,PETSC_VIEWER_STDOUT_WORLD,ierr_local) > > > > > > Now if I keep printing ix in gdb I see the following pattern: > > > > > > (gdb) print ix[0] > > > $18 = 0 > > > (gdb) print ix[1] > > > $19 = 0 > > > (gdb) print ix[2] > > > $20 = 1 > > > (gdb) print ix[3] > > > $21 = 0 > > > (gdb) print ix[4] > > > $22 = 2 > > > (gdb) print ix[5] > > > $23 = 0 > > > (gdb) print ix[6] > > > $24 = 3 > > > > > > > > > which I think is strange since I ix should be a vector of length 2 > because I am calling VecSetValues as > > > > > > > > > call > VecSetValues(x_vec_in,2,index_list(1:2),x_vec_local(1:2),INSERT_VALUES,ierr_local) > > > > > > Am I doing something wrong? > > > > > > Thank you in advance. > > > > > > > > > > > > > > > > > > On Fri, Jun 7, 2013 at 3:03 PM, Matteo Parsani < > parsani.matteo at gmail.com> wrote: > > > Yes, they are correct. When asked, the debugger prints the right > values for bot the vector of the index and the vector of the values. > > > > > > Sorry, how can I set a break point in VecSetValues_Seq() and > VecSetValues_MPI()? > > > If I type > > > > > > break VecSetValues_Seq() > > > > > > or > > > > > > break VecSetValues_MPI() > > > > > > or > > > > > > > /home0/pmatteo/research/lib_src/petsc/src/vec/vec/impls/seq/bvec2.c:1050 > > > > > > I get > > > > > > Function "vecsetvalues_seq" not defined. > > > > > > or > > > > > > Function "vecsetvalues_mpi" not defined. > > > > > > > > > Thanks > > > > > > > > > > > > On Fri, Jun 7, 2013 at 2:32 PM, Barry Smith > wrote: > > > > > > Are, seemingly the correct values getting into the vecsetvalues_ > call? > > > > > > Put a break point also in VecSetValues_Seq() and VecSetValues_MPI() > when it gets into either of those routines you can do step in the debugger > and watch it take the values passed in and attempt to put them in the > appropriate places in the vector's array. > > > > > > Barry > > > > > > On Jun 7, 2013, at 1:20 PM, Matteo Parsani > wrote: > > > > > > > Hello Barry, > > > > I have tried to follow the debugger but I got lost. > > > > > > > > I have run my code in this way: > > > > > > > > ./NSE -start_in_debugger noxterm > > > > > > > > then after getting some output I get > > > > > > > > gdb() > > > > > > > > Then I type break vecsetvalues_ > > > > > > > > > > > > and a break point is generated > > > > > > > > Breakpoint 1 at 0x2ae97eda5827: file > /scratch/home0/pmatteo/research/lib_src/petsc/src/vec/vec/interface/ftn-auto/rvectorf.c, > line 239. > > > > > > > > > > > > I keep typing n and it goes on. It shows me every time vectsetvalues > is called but it never goes inside vectsetvalues. > > > > > > > > I have tested the VecSetValues with just two elements and it does > not work. It sets correctly the values both in serial and in parallel if I > set one value at the time. > > > > > > > > Thanks > > > > > > > > > > > > > > > > On Tue, Apr 30, 2013 at 8:35 AM, Barry Smith > wrote: > > > > > > > > On Apr 30, 2013, at 7:21 AM, Matteo Parsani < > parsani.matteo at gmail.com> wrote: > > > > > > > > > Hello Barry, > > > > > it does not work also if m is not so large. > > > > > > > > Then you need to run in the debugger. You can use the command > line options > > > > > > > > -start_in_debugger noxterm > > > > > > > > in the debugger put a break point in vecsetvalues_ (yes lower > case followed by an underscore) then when it > > > > gets to that point print the size of the array and the values in the > integer and double array to see if they match, then step into the > VecSetValues() function that is called by vecsetvalues_() and look at the > array values ago, continue to step along and it will go into > VecSetValues_Seq() and start actually putting the values into the Vec array. > > > > > > > > Barry > > > > > > > > > > > > > > Thanks again. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 26, 2013 at 3:23 PM, Barry Smith > wrote: > > > > > > > > > > Shouldn't matter that it is called from Fortran we do it all the > time. > > > > > > > > > > Does it work if the final m is not very large? > > > > > > > > > > You may need to run in the debugger and follow the values > through the code to see why they don't get to where they belong. > > > > > > > > > > > > > > > Barry > > > > > > > > > > Could also be a buggy fortran compiler. > > > > > > > > > > > > > > > On Apr 26, 2013, at 2:06 PM, Matteo Parsani < > parsani.matteo at gmail.com> wrote: > > > > > > > > > > > Hello Barry, > > > > > > sorry I modified few things just before to write the mail. > > > > > > The correct loop with the correct name of the variables is the > following > > > > > > > > > > > > ! Number of DOFs owned by this process > > > > > > len_local = > (elem_high-elem_low+1)*nodesperelem*nequations > > > > > > > > > > > > ! Allocate memory for x_vec_local and index_list > > > > > > allocate(x_vec_local(len_local)) > > > > > > allocate(index_list(len_local)) > > > > > > > > > > > > > > > > > > m = 1 > > > > > > ! Loop over all elements > > > > > > do ielem = elem_low, elem_high > > > > > > ! Loop over all nodes in the element > > > > > > do inode = 1, nodesperelem > > > > > > !Loop over all equations > > > > > > do ieq = 1, nequations > > > > > > ! Add element to x_vec_local > > > > > > x_vec_local(m) = ug(ieq,inode,ielem) > > > > > > ! Add element to index list > > > > > > index_list(m) = > (elem_low-1)*nodesperelem*nequations+m-1 > > > > > > ! Update m index > > > > > > m = m+1 > > > > > > end do > > > > > > end do > > > > > > end do > > > > > > > > > > > > ! HERE I HAVE PRINTED x_vec_local, ug and index_list > > > > > > > > > > > > ! Set values in the portion of the vector owned by the > process > > > > > > call > VecSetValues(x_vec_in,m-1,index_list,x_vec_local,INSERT_VALUES,& > > > > > > & ierr_local) > > > > > > > > > > > > ! Assemble initial guess > > > > > > call VecAssemblyBegin(x_vec_in,ierr_local) > > > > > > call VecAssemblyEnd(x_vec_in,ierr_local) > > > > > > > > > > > > > > > > > > > > > > > > I have printed the values and the indices I want to pass to > VecSetValues() and they are correct. > > > > > > > > > > > > I also printed the values after VecSetValues() has been called > and they are wrong. > > > > > > > > > > > > The attachment shows that. > > > > > > > > > > > > > > > > > > Could it be a problem of VecSetValues() + F90 when more than 1 > elements is set? > > > > > > > > > > > > I order to debug my the code I am running just with 1 processor. > Thus the process owns all the DOFs. > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 26, 2013 at 2:39 PM, Barry Smith > wrote: > > > > > > > > > > > > On Apr 26, 2013, at 10:20 AM, Matteo Parsani < > parsani.matteo at gmail.com> wrote: > > > > > > > > > > > > > Hello, > > > > > > > I have some problem when I try to set multiple values to a > PETSc vector that I will use later on with SNES. I am using Fortran 90. > > > > > > > Here the problem and two fixes that however are not so good > for performances reasons. The code is very simple. > > > > > > > > > > > > > > Standard approach that does not work correctly: (I am probably > doing something wrong) > > > > > > > > > > > > > > m = 1 > > > > > > > ! Loop over all elements > > > > > > > do ielem = elem_low, elem_high > > > > > > > ! Loop over all nodes in the element > > > > > > > do inode = 1, nodesperelem > > > > > > > !Loop over all equations > > > > > > > do ieq = 1, nequations > > > > > > > ! Add element to x_vec_local > > > > > > > x_vec_local(m) = ug(ieq,inode,ielem) > > > > > > > ! Add element to index list > > > > > > > ind(m) = > (elem_low-1)*nodesperelem*nequations+m-1 > > > > > > > ! Update m index > > > > > > > m = m+1 > > > > > > > end do > > > > > > > end do > > > > > > > end do > > > > > > > > > > > > > > ! Set values in the portion of the vector owned by the > process > > > > > > > call > VecSetValues(x_vec_in,len_local,index_list,x_vec_local,INSERT_VALUES,& > > > > > > > > > > > > What is len_local and index_list? They do not appear in the > loop above. Shouldn't you be passing m-1 for the length and ind for the > indices? > > > > > > > > > > > > I would first print out the all the values in your input to > VecSetValues() and make sure they are correct. > > > > > > > > > > > > Barry > > > > > > > > > > > > > & ierr_local) > > > > > > > > > > > > > > ! Assemble initial guess > > > > > > > call VecAssemblyBegin(x_vec_in,ierr_local) > > > > > > > call VecAssemblyEnd(x_vec_in,ierr_local) > > > > > > > > > > > > > > Then I print my expected values and the values contained in > the PETSc vector to a file. See attachment. I am running in serial for the > moment BUT strangely if you look at the file I have attached the first 79 > DOFs values have a wrong ordering and the remaining 80 are zero. > > > > > > > > > > > > > > > > > > > > > 1st approach: set just one value at the time inside the loop. > > > > > > > m = 1 > > > > > > > ! Loop over all elements > > > > > > > do ielem = elem_low, elem_high > > > > > > > ! Loop over all nodes in the element > > > > > > > do inode = 1, nodesperelem > > > > > > > !Loop over all equations > > > > > > > do ieq = 1, nequations > > > > > > > ! Add element to x_vec_local > > > > > > > value = ug(ieq,inode,ielem) > > > > > > > ! Add element to index list > > > > > > > ind = > (elem_low-1)*nodesperelem*nequations+m-1 > > > > > > > call > VecSetValues(x_vec_in,1,ind,value,INSERT_VALUES,& > > > > > > > & ierr_local) > > > > > > > ! Update m index > > > > > > > m = m+1 > > > > > > > end do > > > > > > > end do > > > > > > > end do > > > > > > > > > > > > > > > > > > > > > This works fine. As you can see I am using the same expression > used in the previous loop to compute the index of the element that I have > to add in the x_vec_in, i.e. > > > > > > > ind = (elem_low-1)*nodesperelem*nequations+m-1 > > > > > > > > > > > > > > Thus I cannot see which is the problem. > > > > > > > > > > > > > > 2nd approach: get the pointer to the local part of the global > vector and use it to set the values in the global vector > > > > > > > > > > > > > > m = 1 > > > > > > > ! Loop over all elements > > > > > > > do ielem = elem_low, elem_high > > > > > > > ! Loop over all nodes in the element > > > > > > > do inode = 1, nodesperelem > > > > > > > !Loop over all equations > > > > > > > do ieq = 1, nequations > > > > > > > ! Add element to x_vec_local > > > > > > > tmp(m) = ug(ieq,inode,ielem) > > > > > > > ! Update m index > > > > > > > m = m+1 > > > > > > > end do > > > > > > > end do > > > > > > > end do > > > > > > > > > > > > > > > > > > > > > This works fine too. > > > > > > > > > > > > > > > > > > > > > Jut to be complete. I use the following two approaches to view > the vector: > > > > > > > > > > > > > > call VecView(x_vec_in,PETSC_VIEWER_STDOUT_WORLD,ierr_local) > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > call VecGetArrayF90(x_vec_in,tmp,ierr_local) > > > > > > > > > > > > > > > > > > > > > m = 1 > > > > > > > ! Loop over all elements > > > > > > > do ielem = elem_low, elem_high > > > > > > > ! Loop over all nodes in the element > > > > > > > do inode = 1, nodesperelem > > > > > > > !Loop over all equations > > > > > > > do ieq = 1, nequations > > > > > > > write(*,*) > m,index_list(m),x_vec_local(m),tmp(m) > > > > > > > ! Update m index > > > > > > > m = m+1 > > > > > > > end do > > > > > > > end do > > > > > > > end do > > > > > > > > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Matteo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Matteo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Matteo > > > > > > > > > > > > > > > > > > > > -- > > > > Matteo > > > > > > > > > > > > > > > -- > > > Matteo > > > > > > > > > > > > -- > > > Matteo > > > > > > -- Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From rupp at mcs.anl.gov Mon Jun 10 11:15:28 2013 From: rupp at mcs.anl.gov (Karl Rupp) Date: Mon, 10 Jun 2013 11:15:28 -0500 Subject: [petsc-users] CGNS In-Reply-To: <51B5ED58.3010405@lycos.com> References: <51B5ED58.3010405@lycos.com> Message-ID: <51B5FBA0.90509@mcs.anl.gov> Hi Kostas, do you mean CGNS as a mesh file format for DMPlex? Best regards, Karli On 06/10/2013 10:14 AM, Konstantinos Kontzialis wrote: > Dear all, > > How do I install PETSc with support for cgns? > > Kostas From ckontzialis at lycos.com Mon Jun 10 12:07:40 2013 From: ckontzialis at lycos.com (Konstantinos Kontzialis) Date: Mon, 10 Jun 2013 14:07:40 -0300 Subject: [petsc-users] CGNS In-Reply-To: References: Message-ID: <51B607DC.2080008@lycos.com> On 6/10/2013 2:00 PM, petsc-users-request at mcs.anl.gov wrote: > Send petsc-users mailing list submissions to > petsc-users at mcs.anl.gov > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > or, via email, send a message with subject or body 'help' to > petsc-users-request at mcs.anl.gov > > You can reach the person managing the list at > petsc-users-owner at mcs.anl.gov > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of petsc-users digest..." > > > Today's Topics: > > 1. Re: CGNS (Karl Rupp) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 10 Jun 2013 11:15:28 -0500 > From: Karl Rupp > To: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] CGNS > Message-ID: <51B5FBA0.90509 at mcs.anl.gov> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hi Kostas, > > do you mean CGNS as a mesh file format for DMPlex? > > Best regards, > Karli > > > On 06/10/2013 10:14 AM, Konstantinos Kontzialis wrote: >> Dear all, >> >> How do I install PETSc with support for cgns? >> >> Kostas > > > ------------------------------ > > _______________________________________________ > petsc-users mailing list > petsc-users at mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > > > End of petsc-users Digest, Vol 54, Issue 19 > ******************************************* Yes Karl. From balay at mcs.anl.gov Mon Jun 10 12:19:03 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 10 Jun 2013 12:19:03 -0500 (CDT) Subject: [petsc-users] CGNS In-Reply-To: <51B607DC.2080008@lycos.com> References: <51B607DC.2080008@lycos.com> Message-ID: On Mon, 10 Jun 2013, Konstantinos Kontzialis wrote: > On 6/10/2013 2:00 PM, petsc-users-request at mcs.anl.gov wrote: > > Message: 1 > > Date: Mon, 10 Jun 2013 11:15:28 -0500 > > From: Karl Rupp > > To: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] CGNS > > Message-ID: <51B5FBA0.90509 at mcs.anl.gov> > > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > > > Hi Kostas, > > > > do you mean CGNS as a mesh file format for DMPlex? > > > > Best regards, > > Karli > > > > > > On 06/10/2013 10:14 AM, Konstantinos Kontzialis wrote: > > > Dear all, > > > > > > How do I install PETSc with support for cgns? > > > > > > Kostas > > > Yes Karl. Is this what you are looking for? https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateCGNS.html PetscErrorCode DMPlexCreateCGNS(MPI_Comm comm, PetscInt cgid, PetscBool interpolate, DM *dm) Matt, Perhaps 'cgid' should be 'int' - and not 'PetscInt'? Satish From ckontzialis at lycos.com Mon Jun 10 12:22:44 2013 From: ckontzialis at lycos.com (Konstantinos Kontzialis) Date: Mon, 10 Jun 2013 14:22:44 -0300 Subject: [petsc-users] CGNS In-Reply-To: References: <51B607DC.2080008@lycos.com> Message-ID: <51B60B64.1080001@lycos.com> On 6/10/2013 2:19 PM, Satish Balay wrote: > On Mon, 10 Jun 2013, Konstantinos Kontzialis wrote: > >> On 6/10/2013 2:00 PM, petsc-users-request at mcs.anl.gov wrote: >>> Message: 1 >>> Date: Mon, 10 Jun 2013 11:15:28 -0500 >>> From: Karl Rupp >>> To: petsc-users at mcs.anl.gov >>> Subject: Re: [petsc-users] CGNS >>> Message-ID: <51B5FBA0.90509 at mcs.anl.gov> >>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >>> >>> Hi Kostas, >>> >>> do you mean CGNS as a mesh file format for DMPlex? >>> >>> Best regards, >>> Karli >>> >>> >>> On 06/10/2013 10:14 AM, Konstantinos Kontzialis wrote: >>>> Dear all, >>>> >>>> How do I install PETSc with support for cgns? >>>> >>>> Kostas >> Yes Karl. > Is this what you are looking for? > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateCGNS.html > > PetscErrorCode DMPlexCreateCGNS(MPI_Comm comm, PetscInt cgid, PetscBool interpolate, DM *dm) > > Matt, > > Perhaps 'cgid' should be 'int' - and not 'PetscInt'? > > Satish No, I need to know how to configure petsc in order to use the cgns format with it. From knepley at gmail.com Mon Jun 10 13:36:31 2013 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 10 Jun 2013 20:36:31 +0200 Subject: [petsc-users] CGNS In-Reply-To: <51B60B64.1080001@lycos.com> References: <51B607DC.2080008@lycos.com> <51B60B64.1080001@lycos.com> Message-ID: On Mon, Jun 10, 2013 at 7:22 PM, Konstantinos Kontzialis < ckontzialis at lycos.com> wrote: > On 6/10/2013 2:19 PM, Satish Balay wrote: > >> On Mon, 10 Jun 2013, Konstantinos Kontzialis wrote: >> >> On 6/10/2013 2:00 PM, petsc-users-request at mcs.anl.**govwrote: >>> >>>> Message: 1 >>>> Date: Mon, 10 Jun 2013 11:15:28 -0500 >>>> From: Karl Rupp >>>> To: petsc-users at mcs.anl.gov >>>> Subject: Re: [petsc-users] CGNS >>>> Message-ID: <51B5FBA0.90509 at mcs.anl.gov> >>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >>>> >>>> Hi Kostas, >>>> >>>> do you mean CGNS as a mesh file format for DMPlex? >>>> >>>> Best regards, >>>> Karli >>>> >>>> >>>> On 06/10/2013 10:14 AM, Konstantinos Kontzialis wrote: >>>> >>>>> Dear all, >>>>> >>>>> How do I install PETSc with support for cgns? >>>>> >>>>> Kostas >>>>> >>>> Yes Karl. >>> >> Is this what you are looking for? >> >> https://www.mcs.anl.gov/petsc/**petsc-current/docs/**manualpages/DM/** >> DMPlexCreateCGNS.html >> >> PetscErrorCode DMPlexCreateCGNS(MPI_Comm comm, PetscInt cgid, PetscBool >> interpolate, DM *dm) >> >> Matt, >> >> Perhaps 'cgid' should be 'int' - and not 'PetscInt'? >> >> Satish >> > No, I need to know how to configure petsc in order to use the cgns format > with it. > It does not work yet. I wrote it so Jed could test it. You have to manually define PETSC_HAVE_CGNS and add the relevant libraries. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Jun 10 13:47:11 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 10 Jun 2013 13:47:11 -0500 (CDT) Subject: [petsc-users] CGNS In-Reply-To: References: <51B607DC.2080008@lycos.com> <51B60B64.1080001@lycos.com> Message-ID: On Mon, 10 Jun 2013, Matthew Knepley wrote: > On Mon, Jun 10, 2013 at 7:22 PM, Konstantinos Kontzialis < > ckontzialis at lycos.com> wrote: > > No, I need to know how to configure petsc in order to use the cgns format > > with it. > > > > It does not work yet. I wrote it so Jed could test it. You have to manually > define PETSC_HAVE_CGNS > and add the relevant libraries. I gess one way to specify is to use configure options: CPPFLAGS='-DPETSC_HAVE_CGNS -I/path-to-cgns/include' LIBS=/path_to_cgns/lib/libcgns.a Satish From ztdepyahoo at 163.com Tue Jun 11 00:41:24 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Tue, 11 Jun 2013 13:41:24 +0800 (CST) Subject: [petsc-users] How to get the matrix diagonal value A(i,i) Message-ID: <74be6fe.3b85.13f31c0c49d.Coremail.ztdepyahoo@163.com> which function return the value at the diagonal of matrix A(i,i). Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 11 01:32:20 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 11 Jun 2013 08:32:20 +0200 Subject: [petsc-users] How to get the matrix diagonal value A(i,i) In-Reply-To: <74be6fe.3b85.13f31c0c49d.Coremail.ztdepyahoo@163.com> References: <74be6fe.3b85.13f31c0c49d.Coremail.ztdepyahoo@163.com> Message-ID: MatGetDiagonal or MatGetValue, depending on how you want it. On Jun 11, 2013 7:41 AM, "???" wrote: > which function return the value at the diagonal of matrix A(i,i). > > Regards. > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shuangshuang.Jin at pnnl.gov Tue Jun 11 15:22:04 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Tue, 11 Jun 2013 13:22:04 -0700 Subject: [petsc-users] MATELEMENTAL Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FD89B52DD@EMAIL04.pnl.gov> Hello, I'm trying to set my Jacobian matrix to MATELEMENTAL type. According to an Elemental matrix example code: /src/mat/examples/tests/ex38.c, I wrote my piece of code as following: ierr = MatCreate(PETSC_COMM_WORLD, &J); CHKERRQ(ierr); // J: Jacobian matrix ierr = MatSetSizes(J, 12, 12, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr); ierr = MatSetType(J, MATELEMENTAL); CHKERRQ(ierr); ierr = MatSetFromOptions(J); CHKERRQ(ierr); ierr = MatSetUp(J); CHKERRQ(ierr); PetscInt nrows,ncols; const PetscInt *rows,*cols; IS isrows,iscols; PetscScalar *v; ierr = MatGetOwnershipIS(J, &isrows, &iscols); CHKERRQ(ierr); // Set local matrix entries ierr = ISGetLocalSize(isrows, &nrows); CHKERRQ(ierr); ierr = ISGetIndices(isrows, &rows); CHKERRQ(ierr); ierr = ISGetLocalSize(iscols, &ncols); CHKERRQ(ierr); ierr = ISGetIndices(iscols, &cols); CHKERRQ(ierr); ierr = PetscMalloc(nrows*ncols*sizeof(*v), &v); CHKERRQ(ierr); printf("nrows=%d, ncols=%d\n",nrows,ncols); for (i = 0; i < nrows; i++) { for (j = 0; j < ncols; j++) { v[i*ncols+j] = jacVal[i][j]; // jacVal stores the value of the 12*12 Jacobian matrix } } ierr = MatSetValues(J, nrows, rows, ncols, cols, v, INSERT_VALUES); CHKERRQ(ierr); ierr = ISRestoreIndices(isrows, &rows); CHKERRQ(ierr); ierr = ISRestoreIndices(iscols, &cols); CHKERRQ(ierr); ierr = MatAssemblyBegin(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = ISDestroy(&isrows); CHKERRQ(ierr); ierr = ISDestroy(&iscols); CHKERRQ(ierr); This works for one processor. When it goes to multiple processor, it will fail. I notice that when the # of processor equals to 1, the nrows and ncols will be 12 each, which are the dimension of the Jacobian matrix. However, when the # of processor goes up, for example, to 2, nrows=24, ncols=12, then it will fail because the index of my jac[i][j] are withn [0, 11]. My question is, to use MATELEMENTAL, what is the rule to fit the values of my 12x12 matrix into a 24X12 arrays by the two level for loops? And I don't understand what the set means here: [d3m956 at olympus tests]$ mpiexec -n 1 ex38 -Cexp_view_ownership 1 -m 12 -n 12 Ownership of explicit C: Row index set: Number of indices in set 12 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 Column index set: Number of indices in set 12 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 nrows=12, ncols=12 [d3m956 at olympus tests]$ mpiexec -n 2 ex38 -Cexp_view_ownership 1 -m 12 -n 12 Ownership of explicit C: Row index set: [0] Number of indices in set 24 [0] 0 0 [0] 1 12 [0] 2 1 [0] 3 13 [0] 4 2 [0] 5 14 [0] 6 3 [0] 7 15 [0] 8 4 [0] 9 16 [0] 10 5 [0] 11 17 [0] 12 6 [0] 13 18 [0] 14 7 [0] 15 19 [0] 16 8 [0] 17 20 [0] 18 9 [0] 19 21 [0] 20 10 [0] 21 22 [0] 22 11 [0] 23 23 nrows=24, ncols=12 [1] Number of indices in set 24 [1] 0 0 [1] 1 12 [1] 2 1 [1] 3 13 [1] 4 2 [1] 5 14 [1] 6 3 [1] 7 15 [1] 8 4 [1] 9 16 [1] 10 5 [1] 11 17 [1] 12 6 [1] 13 18 [1] 14 7 [1] 15 19 [1] 16 8 [1] 17 20 [1] 18 9 [1] 19 21 [1] 20 10 [1] 21 22 [1] 22 11 [1] 23 23 Column index set: [0] Number of indices in set 12 [0] 0 0 [0] 1 1 [0] 2 2 [0] 3 3 [0] 4 4 [0] 5 5 [0] 6 6 [0] 7 7 [0] 8 8 [0] 9 9 [0] 10 10 [0] 11 11 [1] Number of indices in set 12 [1] 0 12 [1] 1 13 [1] 2 14 [1] 3 15 [1] 4 16 [1] 5 17 [1] 6 18 [1] 7 19 [1] 8 20 [1] 9 21 [1] 10 22 [1] 11 23 nrows=24, ncols=12 Thanks, Shuangshuang -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Jun 11 17:04:33 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 11 Jun 2013 17:04:33 -0500 Subject: [petsc-users] MATELEMENTAL In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FD89B52DD@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD89B52DD@EMAIL04.pnl.gov> Message-ID: Shuangshuang, In ex38.c, m and n are local block sizes, not global. When running np=2 on a matrix with global sizes 12 by 12, you should give mpiexec -n 2 ./ex38 -Cexp_view_ownership 1 -m 6 -n 6 Hong On Tue, Jun 11, 2013 at 3:22 PM, Jin, Shuangshuang < Shuangshuang.Jin at pnnl.gov> wrote: > Hello, I?m trying to set my Jacobian matrix to MATELEMENTAL type. > > According to an Elemental matrix example code: > /src/mat/examples/tests/ex38.c, I wrote my piece of code as following: > > ierr = MatCreate(PETSC_COMM_WORLD, &J); CHKERRQ(ierr); // J: Jacobian > matrix > ierr = MatSetSizes(J, 12, 12, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr); > ierr = MatSetType(J, MATELEMENTAL); CHKERRQ(ierr); > ierr = MatSetFromOptions(J); CHKERRQ(ierr); > ierr = MatSetUp(J); CHKERRQ(ierr); > > PetscInt nrows,ncols; > const PetscInt *rows,*cols; > IS isrows,iscols; > PetscScalar *v; > ierr = MatGetOwnershipIS(J, &isrows, &iscols); CHKERRQ(ierr); > > // Set local matrix entries > ierr = ISGetLocalSize(isrows, &nrows); CHKERRQ(ierr); > ierr = ISGetIndices(isrows, &rows); CHKERRQ(ierr); > ierr = ISGetLocalSize(iscols, &ncols); CHKERRQ(ierr); > ierr = ISGetIndices(iscols, &cols); CHKERRQ(ierr); > ierr = PetscMalloc(nrows*ncols*sizeof(*v), &v); CHKERRQ(ierr); > printf("nrows=%d, ncols=%d\n",nrows,ncols); > for (i = 0; i < nrows; i++) { > for (j = 0; j < ncols; j++) { > v[i*ncols+j] = jacVal[i][j]; // jacVal stores the value of the 12*12 > Jacobian matrix > } > } > ierr = MatSetValues(J, nrows, rows, ncols, cols, v, INSERT_VALUES); > CHKERRQ(ierr); > ierr = ISRestoreIndices(isrows, &rows); CHKERRQ(ierr); > ierr = ISRestoreIndices(iscols, &cols); CHKERRQ(ierr); > ierr = MatAssemblyBegin(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatAssemblyEnd(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = ISDestroy(&isrows); CHKERRQ(ierr); > ierr = ISDestroy(&iscols); CHKERRQ(ierr); > > This works for one processor. When it goes to multiple processor, it will > fail. > > I notice that when the # of processor equals to 1, the nrows and ncols > will be 12 each, which are the dimension of the Jacobian matrix. > However, when the # of processor goes up, for example, to 2, nrows=24, > ncols=12, then it will fail because the index of my jac[i][j] are withn [0, > 11]. > > My question is, to use MATELEMENTAL, what is the rule to fit the values of > my 12x12 matrix into a 24X12 arrays by the two level for loops? > > And I don?t understand what the set means here: > [d3m956 at olympus tests]$ mpiexec -n 1 ex38 -Cexp_view_ownership 1 -m 12 -n > 12 > Ownership of explicit C: > Row index set: > Number of indices in set 12 > 0 0 > 1 1 > 2 2 > 3 3 > 4 4 > 5 5 > 6 6 > 7 7 > 8 8 > 9 9 > 10 10 > 11 11 > Column index set: > Number of indices in set 12 > 0 0 > 1 1 > 2 2 > 3 3 > 4 4 > 5 5 > 6 6 > 7 7 > 8 8 > 9 9 > 10 10 > 11 11 > nrows=12, ncols=12 > > [d3m956 at olympus tests]$ mpiexec -n 2 ex38 -Cexp_view_ownership 1 -m 12 -n > 12 > Ownership of explicit C: > Row index set: > [0] Number of indices in set 24 > [0] 0 0 > [0] 1 12 > [0] 2 1 > [0] 3 13 > [0] 4 2 > [0] 5 14 > [0] 6 3 > [0] 7 15 > [0] 8 4 > [0] 9 16 > [0] 10 5 > [0] 11 17 > [0] 12 6 > [0] 13 18 > [0] 14 7 > [0] 15 19 > [0] 16 8 > [0] 17 20 > [0] 18 9 > [0] 19 21 > [0] 20 10 > [0] 21 22 > [0] 22 11 > [0] 23 23 > nrows=24, ncols=12 > [1] Number of indices in set 24 > [1] 0 0 > [1] 1 12 > [1] 2 1 > [1] 3 13 > [1] 4 2 > [1] 5 14 > [1] 6 3 > [1] 7 15 > [1] 8 4 > [1] 9 16 > [1] 10 5 > [1] 11 17 > [1] 12 6 > [1] 13 18 > [1] 14 7 > [1] 15 19 > [1] 16 8 > [1] 17 20 > [1] 18 9 > [1] 19 21 > [1] 20 10 > [1] 21 22 > [1] 22 11 > [1] 23 23 > Column index set: > [0] Number of indices in set 12 > [0] 0 0 > [0] 1 1 > [0] 2 2 > [0] 3 3 > [0] 4 4 > [0] 5 5 > [0] 6 6 > [0] 7 7 > [0] 8 8 > [0] 9 9 > [0] 10 10 > [0] 11 11 > [1] Number of indices in set 12 > [1] 0 12 > [1] 1 13 > [1] 2 14 > [1] 3 15 > [1] 4 16 > [1] 5 17 > [1] 6 18 > [1] 7 19 > [1] 8 20 > [1] 9 21 > [1] 10 22 > [1] 11 23 > nrows=24, ncols=12 > > Thanks, > Shuangshuang > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shuangshuang.Jin at pnnl.gov Tue Jun 11 17:50:50 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Tue, 11 Jun 2013 15:50:50 -0700 Subject: [petsc-users] MATELEMENTAL References: <6778DE83AB681D49BFC2CD850610FEB1018FD89B52DD@EMAIL04.pnl.gov> Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FD89B5359@EMAIL04.pnl.gov> Sorry, I forgot to reply to the user group again. Here's my question again. Thanks, Shuangshuang From: Jin, Shuangshuang Sent: Tuesday, June 11, 2013 3:43 PM To: 'Hong Zhang' Subject: RE: [petsc-users] MATELEMENTAL Hi, Hong, does that mean in my code where my J is 12x12, If I use one processor, I should set the matrix size as: ierr = MatSetSizes(J, 12, 12, 12, 12); CHKERRQ(ierr); If I use two processor, I should write: ierr = MatSetSizes(J, 6, 6, 12, 12); CHKERRQ(ierr); and for three processors, should be: ierr = MatSetSizes(J, 4, 4, 12, 12); CHKERRQ(ierr); which means, the number of local rows = the number of global rows / the number of processors? Thanks, Shuangshuang From: Hong Zhang [mailto:hzhang at mcs.anl.gov] Sent: Tuesday, June 11, 2013 3:05 PM To: Jin, Shuangshuang Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] MATELEMENTAL Shuangshuang, In ex38.c, m and n are local block sizes, not global. When running np=2 on a matrix with global sizes 12 by 12, you should give mpiexec -n 2 ./ex38 -Cexp_view_ownership 1 -m 6 -n 6 Hong On Tue, Jun 11, 2013 at 3:22 PM, Jin, Shuangshuang > wrote: Hello, I'm trying to set my Jacobian matrix to MATELEMENTAL type. According to an Elemental matrix example code: /src/mat/examples/tests/ex38.c, I wrote my piece of code as following: ierr = MatCreate(PETSC_COMM_WORLD, &J); CHKERRQ(ierr); // J: Jacobian matrix ierr = MatSetSizes(J, 12, 12, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr); ierr = MatSetType(J, MATELEMENTAL); CHKERRQ(ierr); ierr = MatSetFromOptions(J); CHKERRQ(ierr); ierr = MatSetUp(J); CHKERRQ(ierr); PetscInt nrows,ncols; const PetscInt *rows,*cols; IS isrows,iscols; PetscScalar *v; ierr = MatGetOwnershipIS(J, &isrows, &iscols); CHKERRQ(ierr); // Set local matrix entries ierr = ISGetLocalSize(isrows, &nrows); CHKERRQ(ierr); ierr = ISGetIndices(isrows, &rows); CHKERRQ(ierr); ierr = ISGetLocalSize(iscols, &ncols); CHKERRQ(ierr); ierr = ISGetIndices(iscols, &cols); CHKERRQ(ierr); ierr = PetscMalloc(nrows*ncols*sizeof(*v), &v); CHKERRQ(ierr); printf("nrows=%d, ncols=%d\n",nrows,ncols); for (i = 0; i < nrows; i++) { for (j = 0; j < ncols; j++) { v[i*ncols+j] = jacVal[i][j]; // jacVal stores the value of the 12*12 Jacobian matrix } } ierr = MatSetValues(J, nrows, rows, ncols, cols, v, INSERT_VALUES); CHKERRQ(ierr); ierr = ISRestoreIndices(isrows, &rows); CHKERRQ(ierr); ierr = ISRestoreIndices(iscols, &cols); CHKERRQ(ierr); ierr = MatAssemblyBegin(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = ISDestroy(&isrows); CHKERRQ(ierr); ierr = ISDestroy(&iscols); CHKERRQ(ierr); This works for one processor. When it goes to multiple processor, it will fail. I notice that when the # of processor equals to 1, the nrows and ncols will be 12 each, which are the dimension of the Jacobian matrix. However, when the # of processor goes up, for example, to 2, nrows=24, ncols=12, then it will fail because the index of my jac[i][j] are withn [0, 11]. My question is, to use MATELEMENTAL, what is the rule to fit the values of my 12x12 matrix into a 24X12 arrays by the two level for loops? And I don't understand what the set means here: [d3m956 at olympus tests]$ mpiexec -n 1 ex38 -Cexp_view_ownership 1 -m 12 -n 12 Ownership of explicit C: Row index set: Number of indices in set 12 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 Column index set: Number of indices in set 12 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 nrows=12, ncols=12 [d3m956 at olympus tests]$ mpiexec -n 2 ex38 -Cexp_view_ownership 1 -m 12 -n 12 Ownership of explicit C: Row index set: [0] Number of indices in set 24 [0] 0 0 [0] 1 12 [0] 2 1 [0] 3 13 [0] 4 2 [0] 5 14 [0] 6 3 [0] 7 15 [0] 8 4 [0] 9 16 [0] 10 5 [0] 11 17 [0] 12 6 [0] 13 18 [0] 14 7 [0] 15 19 [0] 16 8 [0] 17 20 [0] 18 9 [0] 19 21 [0] 20 10 [0] 21 22 [0] 22 11 [0] 23 23 nrows=24, ncols=12 [1] Number of indices in set 24 [1] 0 0 [1] 1 12 [1] 2 1 [1] 3 13 [1] 4 2 [1] 5 14 [1] 6 3 [1] 7 15 [1] 8 4 [1] 9 16 [1] 10 5 [1] 11 17 [1] 12 6 [1] 13 18 [1] 14 7 [1] 15 19 [1] 16 8 [1] 17 20 [1] 18 9 [1] 19 21 [1] 20 10 [1] 21 22 [1] 22 11 [1] 23 23 Column index set: [0] Number of indices in set 12 [0] 0 0 [0] 1 1 [0] 2 2 [0] 3 3 [0] 4 4 [0] 5 5 [0] 6 6 [0] 7 7 [0] 8 8 [0] 9 9 [0] 10 10 [0] 11 11 [1] Number of indices in set 12 [1] 0 12 [1] 1 13 [1] 2 14 [1] 3 15 [1] 4 16 [1] 5 17 [1] 6 18 [1] 7 19 [1] 8 20 [1] 9 21 [1] 10 22 [1] 11 23 nrows=24, ncols=12 Thanks, Shuangshuang -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Jun 11 22:25:32 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 11 Jun 2013 22:25:32 -0500 Subject: [petsc-users] MATELEMENTAL In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FD89B5359@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD89B52DD@EMAIL04.pnl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD89B5359@EMAIL04.pnl.gov> Message-ID: Shuangshuang : Use ierr = MatSetSizes(J, PETSC_DECIDE,PETSC_DECIDE, 12, 12); CHKERRQ(ierr); for any number of processors (np), or ierr = MatSetSizes(J, 6, 6, PETSC_DECIDE,PETSC_DECIDE); CHKERRQ(ierr); for np=2 or ierr = MatSetSizes(J, 4, 4, PETSC_DECIDE,PETSC_DECIDE); CHKERRQ(ierr); for np=3 ** > > Hi, Hong, does that mean in my code where my J is 12x12,**** > > ** ** > > If I use one processor, I should set the matrix size as: **** > > ierr = MatSetSizes(J, 12, 12, 12, 12); CHKERRQ(ierr);**** > > If I use two processor, I should write:**** > > ierr = MatSetSizes(J, 6, 6, 12, 12); CHKERRQ(ierr);**** > > and for three processors, should be:**** > > ierr = MatSetSizes(J, 4, 4, 12, 12); CHKERRQ(ierr);**** > > ** ** > > which means, the number of local rows = the number of global rows / the > number of processors? > Yes. Hong > *From:* Hong Zhang [mailto:hzhang at mcs.anl.gov ] > *Sent:* Tuesday, June 11, 2013 3:05 PM > *To:* Jin, Shuangshuang > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] MATELEMENTAL**** > > ** ** > > Shuangshuang,**** > > In ex38.c, m and n are local block sizes, not global. When running np=2 on > a matrix with global sizes 12 by 12, you should give**** > > mpiexec -n 2 ./ex38 -Cexp_view_ownership 1 -m 6 -n 6**** > > ** ** > > Hong**** > > ** ** > > ** ** > > On Tue, Jun 11, 2013 at 3:22 PM, Jin, Shuangshuang < > Shuangshuang.Jin at pnnl.gov> wrote:**** > > Hello, I?m trying to set my Jacobian matrix to MATELEMENTAL type.**** > > **** > > According to an Elemental matrix example code: > /src/mat/examples/tests/ex38.c, I wrote my piece of code as following:**** > > **** > > ierr = MatCreate(PETSC_COMM_WORLD, &J); CHKERRQ(ierr); // J: Jacobian > matrix**** > > ierr = MatSetSizes(J, 12, 12, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr); > **** > > ierr = MatSetType(J, MATELEMENTAL); CHKERRQ(ierr);**** > > ierr = MatSetFromOptions(J); CHKERRQ(ierr);**** > > ierr = MatSetUp(J); CHKERRQ(ierr);**** > > **** > > PetscInt nrows,ncols;**** > > const PetscInt *rows,*cols;**** > > IS isrows,iscols;**** > > PetscScalar *v;**** > > ierr = MatGetOwnershipIS(J, &isrows, &iscols); CHKERRQ(ierr);**** > > **** > > // Set local matrix entries**** > > ierr = ISGetLocalSize(isrows, &nrows); CHKERRQ(ierr);**** > > ierr = ISGetIndices(isrows, &rows); CHKERRQ(ierr);**** > > ierr = ISGetLocalSize(iscols, &ncols); CHKERRQ(ierr);**** > > ierr = ISGetIndices(iscols, &cols); CHKERRQ(ierr);**** > > ierr = PetscMalloc(nrows*ncols*sizeof(*v), &v); CHKERRQ(ierr);**** > > printf("nrows=%d, ncols=%d\n",nrows,ncols);**** > > for (i = 0; i < nrows; i++) {**** > > for (j = 0; j < ncols; j++) {**** > > v[i*ncols+j] = jacVal[i][j]; // jacVal stores the value of the 12*12 > Jacobian matrix**** > > }**** > > }**** > > ierr = MatSetValues(J, nrows, rows, ncols, cols, v, INSERT_VALUES); > CHKERRQ(ierr);**** > > ierr = ISRestoreIndices(isrows, &rows); CHKERRQ(ierr);**** > > ierr = ISRestoreIndices(iscols, &cols); CHKERRQ(ierr);**** > > ierr = MatAssemblyBegin(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);**** > > ierr = MatAssemblyEnd(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);**** > > ierr = ISDestroy(&isrows); CHKERRQ(ierr);**** > > ierr = ISDestroy(&iscols); CHKERRQ(ierr);**** > > **** > > This works for one processor. When it goes to multiple processor, it will > fail. **** > > **** > > I notice that when the # of processor equals to 1, the nrows and ncols > will be 12 each, which are the dimension of the Jacobian matrix.**** > > However, when the # of processor goes up, for example, to 2, nrows=24, > ncols=12, then it will fail because the index of my jac[i][j] are withn [0, > 11].**** > > **** > > My question is, to use MATELEMENTAL, what is the rule to fit the values of > my 12x12 matrix into a 24X12 arrays by the two level for loops?**** > > **** > > And I don?t understand what the set means here:**** > > [d3m956 at olympus tests]$ mpiexec -n 1 ex38 -Cexp_view_ownership 1 -m 12 -n > 12**** > > Ownership of explicit C:**** > > Row index set:**** > > Number of indices in set 12**** > > 0 0**** > > 1 1**** > > 2 2**** > > 3 3**** > > 4 4**** > > 5 5**** > > 6 6**** > > 7 7**** > > 8 8**** > > 9 9**** > > 10 10**** > > 11 11**** > > Column index set:**** > > Number of indices in set 12**** > > 0 0**** > > 1 1**** > > 2 2**** > > 3 3**** > > 4 4**** > > 5 5**** > > 6 6**** > > 7 7**** > > 8 8**** > > 9 9**** > > 10 10**** > > 11 11**** > > nrows=12, ncols=12**** > > **** > > [d3m956 at olympus tests]$ mpiexec -n 2 ex38 -Cexp_view_ownership 1 -m 12 -n > 12**** > > Ownership of explicit C:**** > > Row index set:**** > > [0] Number of indices in set 24**** > > [0] 0 0**** > > [0] 1 12**** > > [0] 2 1**** > > [0] 3 13**** > > [0] 4 2**** > > [0] 5 14**** > > [0] 6 3**** > > [0] 7 15**** > > [0] 8 4**** > > [0] 9 16**** > > [0] 10 5**** > > [0] 11 17**** > > [0] 12 6**** > > [0] 13 18**** > > [0] 14 7**** > > [0] 15 19**** > > [0] 16 8**** > > [0] 17 20**** > > [0] 18 9**** > > [0] 19 21**** > > [0] 20 10**** > > [0] 21 22**** > > [0] 22 11**** > > [0] 23 23**** > > nrows=24, ncols=12**** > > [1] Number of indices in set 24**** > > [1] 0 0**** > > [1] 1 12**** > > [1] 2 1**** > > [1] 3 13**** > > [1] 4 2**** > > [1] 5 14**** > > [1] 6 3**** > > [1] 7 15**** > > [1] 8 4**** > > [1] 9 16**** > > [1] 10 5**** > > [1] 11 17**** > > [1] 12 6**** > > [1] 13 18**** > > [1] 14 7**** > > [1] 15 19**** > > [1] 16 8**** > > [1] 17 20**** > > [1] 18 9**** > > [1] 19 21**** > > [1] 20 10**** > > [1] 21 22**** > > [1] 22 11**** > > [1] 23 23**** > > Column index set:**** > > [0] Number of indices in set 12**** > > [0] 0 0**** > > [0] 1 1**** > > [0] 2 2**** > > [0] 3 3**** > > [0] 4 4**** > > [0] 5 5**** > > [0] 6 6**** > > [0] 7 7**** > > [0] 8 8**** > > [0] 9 9**** > > [0] 10 10**** > > [0] 11 11**** > > [1] Number of indices in set 12**** > > [1] 0 12**** > > [1] 1 13**** > > [1] 2 14**** > > [1] 3 15**** > > [1] 4 16**** > > [1] 5 17**** > > [1] 6 18**** > > [1] 7 19**** > > [1] 8 20**** > > [1] 9 21**** > > [1] 10 22**** > > [1] 11 23**** > > nrows=24, ncols=12**** > > **** > > Thanks,**** > > Shuangshuang**** > > **** > > **** > > **** > > **** > > **** > > **** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Jun 12 02:37:35 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 12 Jun 2013 09:37:35 +0200 Subject: [petsc-users] Compile flags for a Blue Gene\Q In-Reply-To: References: <87mwrdwklg.fsf@mcs.anl.gov> Message-ID: <871u87kce8.fsf@mcs.anl.gov> Please always use "reply-all" so that your messages go to the list. This is standard mailing list etiquette. It is important to preserve threading for people who find this discussion later and so that we do not waste our time re-answering the same questions that have already been answered in private side-conversations. You'll likely get an answer faster that way too. Christian Klettner writes: > Dear Jed, > We have been testing your suggestion and found that we can not use ESSL > 5.1.1 which is installed on their system. From the installation page on > the PETSc website it does suggest that ESSL is difficult to bind in. Are > you using ESSL 5.1.1-0.beta to get around this issue? No, it was just what was available when I built that configuration. > Do you see much improvement when going from a 'standard' BLAS to ESSL > blas for sparse matrix multiplications? No, BLAS is not used in sparse matrix multiplication. The BLAS implementation usually makes very little difference since Level-1 operations are most of what is used by sparse iterative methods. Sparse direct solvers eventually produce large supernodes, so BLAS matters there. From balay at mcs.anl.gov Wed Jun 12 08:58:54 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 12 Jun 2013 08:58:54 -0500 (CDT) Subject: [petsc-users] petsc-3.4.1.tar.gz now available Message-ID: Dear PETSc users, The patch release petsc-3.4.1 is now available for download. http://www.mcs.anl.gov/petsc/download/index.html Some of the changes include: * KSPPIPECG: bug fix for PRECONDITIONED and UNPRECONDITIONED norms * Clean up communicator attribute deletion (works around bad MS-MPI attribute handling) * Remove reference to PETSC_COMM_WORLD from DMPlex and TSARKIMEX * SNES: bug fix when MatFDColoring is configured by user * SNESFAS: allow SNESView before setup * HDF5: add missing Fortran bindings * Xcode support restored * A few documentation fixes and improved diagnostic output Satish From zhenglun.wei at gmail.com Wed Jun 12 14:44:23 2013 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Wed, 12 Jun 2013 14:44:23 -0500 Subject: [petsc-users] about BDDC preconditioner Message-ID: <51B8CF97.9010204@gmail.com> Dear All, I hope you're having a nice day. I'm trying to use BDDC as a preconditioner. The executable command is: /mpiexec -np 8 ./ex45 -pc_type bddc -ksp_type cg -ksp_rtol 1.0e-7 -log_summary// ///However, it say: /[//0]PETSC ERROR: --------------------- Error Message ------------------------------------// //[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type:// //see http://www.mcs.anl.gov/petsc/documentation/installation.html#external!// //[0]PETSC ERROR: Unable to find requested PC type bddc!// //[0]PETSC ERROR: ------------------------------------------------------------------------// //[0]PETSC ERROR: Petsc Development GIT revision: 05acd661167488e2bf402ce2590acc1e0469ea17 GIT Date: 2013-04-15 22:45:57 -0500// //[0]PETSC ERROR: See docs/changes/index.html for recent updates.// //[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.// //[0]PETSC ERROR: See docs/index.html for manual pages.// //[0]PETSC ERROR: ------------------------------------------------------------------------// //[0]PETSC ERROR: ./ex45 on a linux-gnu-c-nodebug named m005 by zlwei Wed Jun 12 14:39:42 2013// //[0]PETSC ERROR: Libraries linked from /bio/work1/zlwei/PETSc/petsc-dev/linux-gnu-c-nodebug/lib// //[0]PETSC ERROR: Configure run at Tue Apr 16 11:59:49 2013// //[0]PETSC ERROR: Configure options --download-f-blas-lapack --download-hypre --download-mpich --with-cc=gcc --with-debugging=no --with-fc=gfortran PETSC_ARCH=linux-gnu-c-nodebug// //[0]PETSC ERROR: ------------------------------------------------------------------------// //[0]PETSC ERROR: PCSetType() line 68 in /bio/work1/zlwei/PETSc/petsc-dev/src/ksp/pc/interface/pcset.c// //[0]PETSC ERROR: PCSetFromOptions() line 163 in /bio/work1/zlwei/PETSc/petsc-dev/src/ksp/pc/interface/pcset.c// //[0]PETSC ERROR: KSPSetFromOptions() line 357 in /bio/work1/zlwei/PETSc/petsc-dev/src/ksp/ksp/interface/itcl.c// //[0]PETSC ERROR: main() line 135 in src/ksp/ksp/examples/tutorials/ex45.c// //application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0// //[cli_0]: aborting job:// //application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0/ What should I do for this? thanks, Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dharmareddy84 at gmail.com Wed Jun 12 16:08:28 2013 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Wed, 12 Jun 2013 16:08:28 -0500 Subject: [petsc-users] [Need Help] Time dependent split operator method Message-ID: Hello, I am trying to implement a split operator method to solve time dependent Schrodinger euqation. I use a semi implicit discretization for the time operator. I have not used petsc TS before, it will be helpful if you can give me some pointers to implementing this algorithm: Actual equation: i*hbar*ddt(x) = (H1+H2)*x (linear problem) A1 = i*hbar/dt * IdentityMat - H1/2 B1 = i*hbar/dt * IdentityMat + H2/2 A2 = i*hbar/dt * IdentityMat - H2/2 B2 = i*hbar/dt * IdentityMat + H1/2 x1,x2 are vectors do ic=1,NumTimeSteps solve substep1 : A1*x1 = B1*x2 solve substep2 : A2*x2 = B2*x1 copy solution: x(ic*dt) = x2 end do I have following constraints: 1.) I Want the time step to be fixed user specified value 2.) Matrices A1 and A2 can be reorder to be tri or penta diagonal. I would like to use this for fast direct solve. Typical matrix sizes 10^5 to 10^6 with 3 to 5 non zeros per row. 3.) based on the time it take for my matlab code, which seem to use umfpack, i think a direct solve will be fast enough for my needs. petsc TS has the theta method which seems like the one i should be using but its not clear to me how to do the stepping Thanks Reddy -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. Homepage: https://webspace.utexas.edu/~dpr342 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Jun 13 02:58:29 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 13 Jun 2013 09:58:29 +0200 Subject: [petsc-users] about BDDC preconditioner In-Reply-To: <51B8CF97.9010204@gmail.com> References: <51B8CF97.9010204@gmail.com> Message-ID: <871u86igre.fsf@mcs.anl.gov> "Zhenglun (Alan) Wei" writes: > Dear All, > I hope you're having a nice day. > I'm trying to use BDDC as a preconditioner. The executable command is: > /mpiexec -np 8 ./ex45 -pc_type bddc -ksp_type cg -ksp_rtol 1.0e-7 > -log_summary// This example does not assemble in MATIS format, so it cannot be used with BDDC. > ///However, it say: > /[//0]PETSC ERROR: --------------------- Error Message > ------------------------------------// > //[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing > external package needed for type:// > //see > http://www.mcs.anl.gov/petsc/documentation/installation.html#external!// > //[0]PETSC ERROR: Unable to find requested PC type bddc!// Configure --with-pcbddc. This will be enabled by default soon. From s.zampini at cineca.it Thu Jun 13 03:41:50 2013 From: s.zampini at cineca.it (Stefano Zampini) Date: Thu, 13 Jun 2013 10:41:50 +0200 Subject: [petsc-users] about BDDC preconditioner In-Reply-To: <871u86igre.fsf@mcs.anl.gov> References: <51B8CF97.9010204@gmail.com> <871u86igre.fsf@mcs.anl.gov> Message-ID: ex45 uses DMDA, so it cannot be used to test for PCBDDC. Instead, after configuring -with-pcbddc=1, you can use src/ksp/ksp/examples/tutorials/ex59.c (Laplacian discretized with spectral elements) 2013/6/13 Jed Brown > "Zhenglun (Alan) Wei" writes: > > > Dear All, > > I hope you're having a nice day. > > I'm trying to use BDDC as a preconditioner. The executable command is: > > /mpiexec -np 8 ./ex45 -pc_type bddc -ksp_type cg -ksp_rtol 1.0e-7 > > -log_summary// > > This example does not assemble in MATIS format, so it cannot be used > with BDDC. > > > ///However, it say: > > /[//0]PETSC ERROR: --------------------- Error Message > > ------------------------------------// > > //[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing > > external package needed for type:// > > //see > > http://www.mcs.anl.gov/petsc/documentation/installation.html#external!// > > //[0]PETSC ERROR: Unable to find requested PC type bddc!// > > Configure --with-pcbddc. > > This will be enabled by default soon. > -- Ph. D. Stefano Zampini CINECA SuperComputing Applications and Innovations Department - SCAI Via dei Tizii, 6 00185 Roma - ITALY ------------------------------------------------------------------------------------------------------------------------ Email: s.zampini at cineca.it SkypeID: stefano.zampini GoogleTalk: stefano.zampini at gmail.com Tel: +39 06.44486.707 ------------------------------------------------------------------------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dharmareddy84 at gmail.com Thu Jun 13 16:00:51 2013 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Thu, 13 Jun 2013 16:00:51 -0500 Subject: [petsc-users] configure crash Message-ID: Hello, I am trying to configure petsc with scalar type complex. I get a configure crash. Can you please help me figure out whats going on. I was able to configure petsc with same options but with scalar type=real. I am using petsc 3.3.6 as i need to use slepc and slepc complains about incompatibility with petsc 3.4 I need to use scalar-type=complex, umfpack and mumps, and packages for using DMPLEx (triangle, tetgen, scientific python, fiat and generator) Thanks Reddy -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. Homepage: https://webspace.utexas.edu/~dpr342 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 3571748 bytes Desc: not available URL: From zhenglun.wei at gmail.com Thu Jun 13 16:39:41 2013 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Thu, 13 Jun 2013 16:39:41 -0500 Subject: [petsc-users] Rules of choosing KSPType Message-ID: <51BA3C1D.4060608@gmail.com> Dear All, I hope you're having a nice day. Compared the user manual of PETSc 3.1 and 3.4, I found that there are several 'new' KSP types are added. Then, I have some confusions on which KSP solver I need to pick. To my knowledge, if the matrix is symmetric, positive definite, the CG can be used. Otherwise, I have to use GMRES. My questions are: 1, What is the major difference among 'Conjugate Gradient', 'BiConjugate Gradient' and 'Conjugate Gradient Squared'? As I tested, the CGS is slightly (5%) faster than CG on solving a Poisson equation with uniform Cartesian grid with PC = GAMG. The BiCG is not tested because it seems not be compatible with PC = GAMG. 2, What is the major difference among GMRES, fGMRES and dGMRES. Specially, I want to solve a Poisson equation with non-uniform Cartesian grid. Theoretically, which 'GMRES' method would be the best for my case? thanks, Alan From jedbrown at mcs.anl.gov Thu Jun 13 16:52:14 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 13 Jun 2013 23:52:14 +0200 Subject: [petsc-users] Rules of choosing KSPType In-Reply-To: <51BA3C1D.4060608@gmail.com> References: <51BA3C1D.4060608@gmail.com> Message-ID: <87a9mthe5t.fsf@mcs.anl.gov> "Zhenglun (Alan) Wei" writes: > Dear All, > I hope you're having a nice day. > Compared the user manual of PETSc 3.1 and 3.4, I found that there are > several 'new' KSP types are added. Then, I have some confusions on which > KSP solver I need to pick. To my knowledge, if the matrix is symmetric, > positive definite, the CG can be used. Yes, if you also have an SPD preconditioner. (This is often the case for SPD problems, but not always.) > Otherwise, I have to use GMRES. My questions are: 1, What is the > major difference among 'Conjugate Gradient', 'BiConjugate Gradient' > and 'Conjugate Gradient Squared'? The orthogonality relationship is different. This is covered in any book on iterative methods. You can download Saad's book from his website, for example. > As I tested, the CGS is slightly (5%) faster than CG on solving a > Poisson equation with uniform Cartesian grid with PC = GAMG. The BiCG > is not tested because it seems not be compatible with PC = GAMG. 2, > What is the major difference among GMRES, fGMRES and dGMRES. FGMRES is "flexible" -- the preconditioner can be a nonlinear function (often a few iterations of another iterative method). DGMRES is "deflated" -- it can identify some vectors to enrich the correction, similar to a coarse grid. > Specially, I want to solve a Poisson equation with non-uniform > Cartesian grid. If you use a finite element discretization, this will be SPD. Otherwise, it'll be "morally SPD", and any non-symmetric iteration should be fine. From bsmith at mcs.anl.gov Thu Jun 13 19:45:19 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 13 Jun 2013 19:45:19 -0500 Subject: [petsc-users] configure crash In-Reply-To: References: Message-ID: Hmm, didn't really crash, somehow it was killed working away generating fortran stubs. Can you try running it again? Barry On Jun 13, 2013, at 4:00 PM, Dharmendar Reddy wrote: > Hello, > I am trying to configure petsc with scalar type complex. I get a configure crash. Can you please help me figure out whats going on. I was able to configure petsc with same options but with scalar type=real. > > I am using petsc 3.3.6 as i need to use slepc and slepc complains about incompatibility with petsc 3.4 > > I need to use scalar-type=complex, umfpack and mumps, and packages for using DMPLEx (triangle, tetgen, scientific python, fiat and generator) > > > > > Thanks > Reddy > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > Homepage: https://webspace.utexas.edu/~dpr342 > From dharmareddy84 at gmail.com Thu Jun 13 22:47:17 2013 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Thu, 13 Jun 2013 22:47:17 -0500 Subject: [petsc-users] configure crash In-Reply-To: References: Message-ID: Hello, I rerun the configure, it works. I had to remove the ctetgen, metis and parametis. Is there documentation on dependencies for external packages ? If not, it will be useful to have a required package list for a given external package. Thanks Reddy On Thu, Jun 13, 2013 at 7:45 PM, Barry Smith wrote: > > Hmm, didn't really crash, somehow it was killed working away generating > fortran stubs. Can you try running it again? > > Barry > > On Jun 13, 2013, at 4:00 PM, Dharmendar Reddy > wrote: > > > Hello, > > I am trying to configure petsc with scalar type complex. I get > a configure crash. Can you please help me figure out whats going on. I was > able to configure petsc with same options but with scalar type=real. > > > > I am using petsc 3.3.6 as i need to use slepc and slepc complains about > incompatibility with petsc 3.4 > > > > I need to use scalar-type=complex, umfpack and mumps, and packages for > using DMPLEx (triangle, tetgen, scientific python, fiat and generator) > > > > > > > > > > Thanks > > Reddy > > > > -- > > ----------------------------------------------------- > > Dharmendar Reddy Palle > > Graduate Student > > Microelectronics Research center, > > University of Texas at Austin, > > 10100 Burnet Road, Bldg. 160 > > MER 2.608F, TX 78758-4445 > > e-mail: dharmareddy84 at gmail.com > > Phone: +1-512-350-9082 > > United States of America. > > Homepage: https://webspace.utexas.edu/~dpr342 > > > > -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. Homepage: https://webspace.utexas.edu/~dpr342 -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Jun 14 11:29:20 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 14 Jun 2013 11:29:20 -0500 (CDT) Subject: [petsc-users] configure crash In-Reply-To: References: Message-ID: On Thu, 13 Jun 2013, Dharmendar Reddy wrote: > Hello, > I rerun the configure, it works. I had to remove the ctetgen, > metis and parametis I don't unserstand why you had to remove them. > Is there documentation on dependencies for external > packages ? If not, it will be useful to have a required package list for a > given external package. its currently documented in the appropriate .py file for that package. [and if the dependency is missing - you get a configure error] Satish From fsantost at student.ethz.ch Fri Jun 14 11:31:19 2013 From: fsantost at student.ethz.ch (Santos Teixeira Frederico) Date: Fri, 14 Jun 2013 16:31:19 +0000 Subject: [petsc-users] Error in PETSc with MUMPS on Windows. Message-ID: <682CC3CD7A208742B8C2D116C67199010B19D928@MBX13.d.ethz.ch> Hi, I got the error below when I tried to execute PETSc with MUMPS. Some important facts: 1) the following libraries/versions were compiled separatedly: METIS: 5.1.0, ParMETIS: 4.0.3, MUMPS: 4.10.0 and added to PETSc 3.4.1 along with ScaLAPACK 2.0.2 (plus BLAS, LAPACK, BLACS etc. - from latest MKL). 2) the same code works correctly on Windows with Pardiso (interfaced with PETSc). 3) the same code works correctly on Linux with Pardiso and the libraries provided by --download-***. 4) the MUMPS libraries and its dependencies (the same described below) were linked and correctly executed a test provided by the own library. Despite the lack of informations and debugger (Windows...), could you give some tip and/or guide me to provide you better informations? I appreciate any tip! Thanks a lot! Regards, Frederico. ====================================================================== Starting KSPSolve with MUMPS... Entering DMUMPS driver with JOB, N, NZ = 1 74355 2782474 DMUMPS 4.10.0 L U Solver for unsymmetric matrices Type of parallelism: Working host ****** ANALYSIS STEP ******** Resetting candidate strategy to 0 because NSLAVES=1 ... Structural symmetry (in percent)= 93 Density: NBdense, Average, Median = 0 39 29 ... No column permutation Ordering based on METIS [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] MatLUFactorSymbolic_AIJMUMPS line 880 src/mat/impls/aij/mpi/mumps/C:\PETSC-~1.1\src\mat\impls\aij\mpi\mumps\mumps.c [0]PETSC ERROR: [0] MatLUFactorSymbolic line 2820 src/mat/interface/C:\PETSC-~1.1\src\mat\INTERF~1\matrix.c [0]PETSC ERROR: [0] PCSetUp_LU line 99 src/ksp/pc/impls/factor/lu/C:\PETSC-~1.1\src\ksp\pc\impls\factor\lu\lu.c [0]PETSC ERROR: [0] PCSetUp line 868 src/ksp/pc/interface/C:\PETSC-~1.1\src\ksp\pc\INTERF~1\precon.c [0]PETSC ERROR: [0] KSPSetUp line 192 src/ksp/ksp/interface/C:\PETSC-~1.1\src\ksp\ksp\INTERF~1\itfunc.c [0]PETSC ERROR: [0] KSPSolve line 356 src/ksp/ksp/interface/C:\PETSC-~1.1\src\ksp\ksp\INTERF~1\itfunc.c [0]PETSC ERROR: [0] FluidSolverDirect::Solve line 1145 "unknowndirectory/"..\..\..\Source\FluidSolver\FluidSolverDirect\FluidSolverDirect.cpp [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: D:\Solvers-dev\Solvers\builds\bin\Debug\FluidSolverDirectCavityDriver.exe on a win-x64-msvc-mkl-real-release named FREDERICO-PC by Frederico Teixeira Fri Jun 14 17:53:16 2013 [0]PETSC ERROR: Libraries linked from /cygdrive/c/petsc-3.4.1/win-x64-msvc-mkl-real-release/lib [0]PETSC ERROR: Configure run at Fri Jun 14 15:34:42 2013 [0]PETSC ERROR: Configure options PETSC_ARCH=win-x64-msvc-mkl-real-dbg --with-cc="win32fe cl" --with-cxx="win32fe cl" --with-fc=0 --with-x=0 --with-debugging=1 --with-mpi-dir=/cygdrive/d/Solvers-dev/Solvers/libs/MPI --with-blas-lapack-lib="[/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_bl as95_lp64.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_lapack95_lp64.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_intel_lp64.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_intel_thread.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_core.lib]" --with-metis- lib=/cygdrive/d/Solvers-dev/Solvers/libs/METIS/win-x64-msvc/lib/metis.lib --with-metis-include=/cygdrive/d/Solvers-dev/Solvers/libs/METIS/include --with-parmetis-lib=/cygdrive/d/Solvers-dev/Solvers/libs/PARMETIS/win-x64-msvc/lib/parmetis.lib --with-parmetis-include=/cygdrive/d/Solvers-dev/Solvers/li bs/PARMETIS/include --with-scalapack-lib="[/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_scalapack_lp64.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_blacs_mpich2_lp64.lib]" --with-scalapack-include=/cygdrive/d/Solvers-dev/Solvers/libs/MKL/include --with-mumps-lib="[/cygdrive/d/Sol vers-dev/Solvers/libs/MUMPS/win-x64-msvc-mkl/lib/dmumps.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MUMPS/win-x64-msvc-mkl/lib/mumps-common.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MUMPS/win-x64-msvc-mkl/lib/pord.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MUMPS/win-x64-msvc-mkl/lib/dmumps-f.lib]" --with- mumps-include=/cygdrive/d/Solvers-dev/Solvers/libs/MUMPS/include2 --with-hypre-dir=/cygdrive/d/Solvers-dev/Solvers/libs/HYPRE -CFLAGS=-MD -CXXFLAGS=-MD --LIBS="/cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/ifconsol.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libifcore md.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libifportmd.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libiomp5md.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libirc.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libmmd.lib /cyg drive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/svml_dispmd.lib /cygdrive/d/Solvers-dev/Solvers/libs/MPI/lib/fmpich2.lib" --useThreads=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 job aborted: rank: node: exit code[: error message] 0: Frederico-PC: 59: process 0 exited without calling finalize From balay at mcs.anl.gov Fri Jun 14 12:00:25 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 14 Jun 2013 12:00:25 -0500 (CDT) Subject: [petsc-users] Error in PETSc with MUMPS on Windows. In-Reply-To: <682CC3CD7A208742B8C2D116C67199010B19D928@MBX13.d.ethz.ch> References: <682CC3CD7A208742B8C2D116C67199010B19D928@MBX13.d.ethz.ch> Message-ID: > [0]PETSC ERROR: [0] MatLUFactorSymbolic_AIJMUMPS line 880 src/mat/impls/aij/mpi/mumps/C:\PETSC-~1.1\src\mat\impls\aij\mpi\mumps\mumps.c So its crashing somewhere in MatLUFactorSymbolic_AIJMUMPS() - which has calls to mumps routines. > --with-mpi-dir=/cygdrive/d/Solvers-dev/Solvers/libs/MPI /cygdrive/d/Solvers-dev/Solvers/libs/MPI/lib/fmpich2.lib Looks like you are using mpich2 - but not standard install? And Its not clear how one would use blacs/scalapack from mkl. Is it supporsed to be compatible with this version of mpich you have? [I have no idea] Such mixing of mpi compiled codes [where packages are compiled with different variants of mpi - but combined into a single binary] can potentially have issues. And you can do debugging on windows. For sequential you do: msdev binary.exe [or whatever the current name for the developer studio is.Its 'devenv' for VC2008, and something elese for VC2012] And if you wish to debug parallely - you would have to compile the application with the developer studio project files - and perhaps follow instructions from: http://www.mpich.org/static/downloads/1.4.1p1/mpich2-1.4.1p1-windevguide.pdf Satish On Fri, 14 Jun 2013, Santos Teixeira Frederico wrote: > Hi, > > I got the error below when I tried to execute PETSc with MUMPS. Some important facts: > > 1) the following libraries/versions were compiled separatedly: METIS: 5.1.0, ParMETIS: 4.0.3, MUMPS: 4.10.0 and added to PETSc 3.4.1 along with ScaLAPACK 2.0.2 (plus BLAS, LAPACK, BLACS etc. - from latest MKL). > > 2) the same code works correctly on Windows with Pardiso (interfaced with PETSc). > > 3) the same code works correctly on Linux with Pardiso and the libraries provided by --download-***. > > 4) the MUMPS libraries and its dependencies (the same described below) were linked and correctly executed a test provided by the own library. > > Despite the lack of informations and debugger (Windows...), could you give some tip and/or guide me to provide you better informations? I appreciate any tip! > > Thanks a lot! > > Regards, > Frederico. > > ====================================================================== > > Starting KSPSolve with MUMPS... > Entering DMUMPS driver with JOB, N, NZ = 1 74355 2782474 > > DMUMPS 4.10.0 > L U Solver for unsymmetric matrices > Type of parallelism: Working host > > ****** ANALYSIS STEP ******** > > Resetting candidate strategy to 0 because NSLAVES=1 > > ... Structural symmetry (in percent)= 93 > Density: NBdense, Average, Median = 0 39 29 > ... No column permutation > Ordering based on METIS > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] MatLUFactorSymbolic_AIJMUMPS line 880 src/mat/impls/aij/mpi/mumps/C:\PETSC-~1.1\src\mat\impls\aij\mpi\mumps\mumps.c > [0]PETSC ERROR: [0] MatLUFactorSymbolic line 2820 src/mat/interface/C:\PETSC-~1.1\src\mat\INTERF~1\matrix.c > [0]PETSC ERROR: [0] PCSetUp_LU line 99 src/ksp/pc/impls/factor/lu/C:\PETSC-~1.1\src\ksp\pc\impls\factor\lu\lu.c > [0]PETSC ERROR: [0] PCSetUp line 868 src/ksp/pc/interface/C:\PETSC-~1.1\src\ksp\pc\INTERF~1\precon.c > [0]PETSC ERROR: [0] KSPSetUp line 192 src/ksp/ksp/interface/C:\PETSC-~1.1\src\ksp\ksp\INTERF~1\itfunc.c > [0]PETSC ERROR: [0] KSPSolve line 356 src/ksp/ksp/interface/C:\PETSC-~1.1\src\ksp\ksp\INTERF~1\itfunc.c > [0]PETSC ERROR: [0] FluidSolverDirect::Solve line 1145 "unknowndirectory/"..\..\..\Source\FluidSolver\FluidSolverDirect\FluidSolverDirect.cpp > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: D:\Solvers-dev\Solvers\builds\bin\Debug\FluidSolverDirectCavityDriver.exe on a win-x64-msvc-mkl-real-release named FREDERICO-PC by Frederico Teixeira Fri Jun 14 17:53:16 2013 > [0]PETSC ERROR: Libraries linked from /cygdrive/c/petsc-3.4.1/win-x64-msvc-mkl-real-release/lib > [0]PETSC ERROR: Configure run at Fri Jun 14 15:34:42 2013 > [0]PETSC ERROR: Configure options PETSC_ARCH=win-x64-msvc-mkl-real-dbg --with-cc="win32fe cl" --with-cxx="win32fe cl" --with-fc=0 --with-x=0 --with-debugging=1 --with-mpi-dir=/cygdrive/d/Solvers-dev/Solvers/libs/MPI --with-blas-lapack-lib="[/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_bl > as95_lp64.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_lapack95_lp64.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_intel_lp64.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_intel_thread.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_core.lib]" --with-metis- > lib=/cygdrive/d/Solvers-dev/Solvers/libs/METIS/win-x64-msvc/lib/metis.lib --with-metis-include=/cygdrive/d/Solvers-dev/Solvers/libs/METIS/include --with-parmetis-lib=/cygdrive/d/Solvers-dev/Solvers/libs/PARMETIS/win-x64-msvc/lib/parmetis.lib --with-parmetis-include=/cygdrive/d/Solvers-dev/Solvers/li > bs/PARMETIS/include --with-scalapack-lib="[/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_scalapack_lp64.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_blacs_mpich2_lp64.lib]" --with-scalapack-include=/cygdrive/d/Solvers-dev/Solvers/libs/MKL/include --with-mumps-lib="[/cygdrive/d/Sol > vers-dev/Solvers/libs/MUMPS/win-x64-msvc-mkl/lib/dmumps.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MUMPS/win-x64-msvc-mkl/lib/mumps-common.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MUMPS/win-x64-msvc-mkl/lib/pord.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MUMPS/win-x64-msvc-mkl/lib/dmumps-f.lib]" --with- > mumps-include=/cygdrive/d/Solvers-dev/Solvers/libs/MUMPS/include2 --with-hypre-dir=/cygdrive/d/Solvers-dev/Solvers/libs/HYPRE -CFLAGS=-MD -CXXFLAGS=-MD --LIBS="/cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/ifconsol.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libifcore > md.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libifportmd.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libiomp5md.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libirc.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libmmd.lib /cyg > drive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/svml_dispmd.lib /cygdrive/d/Solvers-dev/Solvers/libs/MPI/lib/fmpich2.lib" --useThreads=0 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > job aborted: > rank: node: exit code[: error message] > 0: Frederico-PC: 59: process 0 exited without calling finalize > From dharmareddy84 at gmail.com Fri Jun 14 13:15:41 2013 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Fri, 14 Jun 2013 13:15:41 -0500 Subject: [petsc-users] configure crash In-Reply-To: References: Message-ID: Hello, I get the Following error when i use ctetgen: (have a look at the attached log file, i copied a snippet here). Also, metis and parametis give error when i configure with scalar-type=complex. I was using MUMPS which requires either parmetis or pt-scotch. I was able to cinfigure with ptscotch. At this point, i am just seting up my code, an i do not know which combination for direct solver will give a good result. I was hoping to configure with the required packages and then experiment. ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Cannot use ctetgen with complex numbers it is not coded for this capability ******************************************************************************* File "./config/configure.py", line 293, in petsc_configure framework.configure(out = sys.stdout) File "/home1/00924/Reddy135/LocalApps/petsc/config/BuildSystem/config/framework.py", line 933, in configure child.configure() File "/home1/00924/Reddy135/LocalApps/petsc/config/BuildSystem/config/package.py", line 552, in configure self.consistencyChecks() File "/home1/00924/Reddy135/LocalApps/petsc/config/PETSc/package.py", line 38, in consistencyChecks raise RuntimeError('Cannot use '+self.name+' with complex numbers it is not coded for this capability') On Fri, Jun 14, 2013 at 11:29 AM, Satish Balay wrote: > On Thu, 13 Jun 2013, Dharmendar Reddy wrote: > > > Hello, > > I rerun the configure, it works. I had to remove the ctetgen, > > metis and parametis > > I don't unserstand why you had to remove them. > > > Is there documentation on dependencies for external > > packages ? If not, it will be useful to have a required package list for > a > > given external package. > > its currently documented in the appropriate .py file for that package. > [and if the dependency is missing - you get a configure error] > > Satish > -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. Homepage: https://webspace.utexas.edu/~dpr342 -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Jun 14 13:33:50 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 14 Jun 2013 13:33:50 -0500 (CDT) Subject: [petsc-users] configure crash In-Reply-To: References: Message-ID: On Fri, 14 Jun 2013, Dharmendar Reddy wrote: > Hello, > I get the Following error when i use ctetgen: (have a look at the > attached log file, i copied a snippet here). Perhaps ctetgen would work with compilex. [Matt can confirm]. You can do the following to enable it. diff --git a/config/PETSc/packages/ctetgen.py b/config/PETSc/packages/ctetgen.py index 71ec5e6..f09d952 100644 --- a/config/PETSc/packages/ctetgen.py +++ b/config/PETSc/packages/ctetgen.py @@ -9,6 +9,7 @@ class Configure(PETSc.package.NewPackage): self.functions = [] self.includes = [] self.liblist = [['libctetgen.a']] + self.complex = 1 self.compileCtetgen = 0 return > Also, metis and parametis give > error when i configure with scalar-type=complex. It should work. What error do you get? Send logs. Satish > I was using MUMPS which > requires either parmetis or pt-scotch. I was able to cinfigure with > ptscotch. At this point, i am just seting up my code, an i do not know > which combination for direct solver will give a good result. I was hoping > to configure with the required packages and then experiment. > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > ------------------------------------------------------------------------------- > Cannot use ctetgen with complex numbers it is not coded for this capability > ******************************************************************************* > File "./config/configure.py", line 293, in petsc_configure > framework.configure(out = sys.stdout) > File > "/home1/00924/Reddy135/LocalApps/petsc/config/BuildSystem/config/framework.py", > line 933, in configure > child.configure() > File > "/home1/00924/Reddy135/LocalApps/petsc/config/BuildSystem/config/package.py", > line 552, in configure > self.consistencyChecks() > File "/home1/00924/Reddy135/LocalApps/petsc/config/PETSc/package.py", > line 38, in consistencyChecks > raise RuntimeError('Cannot use '+self.name+' with complex numbers it is > not coded for this capability') > > > > > On Fri, Jun 14, 2013 at 11:29 AM, Satish Balay wrote: > > > On Thu, 13 Jun 2013, Dharmendar Reddy wrote: > > > > > Hello, > > > I rerun the configure, it works. I had to remove the ctetgen, > > > metis and parametis > > > > I don't unserstand why you had to remove them. > > > > > Is there documentation on dependencies for external > > > packages ? If not, it will be useful to have a required package list for > > a > > > given external package. > > > > its currently documented in the appropriate .py file for that package. > > [and if the dependency is missing - you get a configure error] > > > > Satish > > > > > > From dharmareddy84 at gmail.com Fri Jun 14 13:54:33 2013 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Fri, 14 Jun 2013 13:54:33 -0500 Subject: [petsc-users] configure crash In-Reply-To: References: Message-ID: Hello, I rerun the configure with metis and parametis, it works now. I guess something went wrong when i got the configure crash messages and i saw the error messages related to metis and parmetis. But now it works. I will check the ctetgen with the patch you and it would make sense if the ctetgen works with scalar-type complex, i would expect the data transaction between petsc and tegen to use PetscReal. thanks Reddy On Fri, Jun 14, 2013 at 1:33 PM, Satish Balay wrote: > On Fri, 14 Jun 2013, Dharmendar Reddy wrote: > > > Hello, > > I get the Following error when i use ctetgen: (have a look at > the > > attached log file, i copied a snippet here). > > Perhaps ctetgen would work with compilex. [Matt can confirm]. You can > do the following to enable it. > > diff --git a/config/PETSc/packages/ctetgen.py > b/config/PETSc/packages/ctetgen.py > index 71ec5e6..f09d952 100644 > --- a/config/PETSc/packages/ctetgen.py > +++ b/config/PETSc/packages/ctetgen.py > @@ -9,6 +9,7 @@ class Configure(PETSc.package.NewPackage): > self.functions = [] > self.includes = [] > self.liblist = [['libctetgen.a']] > + self.complex = 1 > self.compileCtetgen = 0 > return > > > > Also, metis and parametis give > > error when i configure with scalar-type=complex. > > It should work. What error do you get? Send logs. > > Satish > > > I was using MUMPS which > > requires either parmetis or pt-scotch. I was able to cinfigure with > > ptscotch. At this point, i am just seting up my code, an i do not know > > which combination for direct solver will give a good result. I was hoping > > to configure with the required packages and then experiment. > > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > details): > > > ------------------------------------------------------------------------------- > > Cannot use ctetgen with complex numbers it is not coded for this > capability > > > ******************************************************************************* > > File "./config/configure.py", line 293, in petsc_configure > > framework.configure(out = sys.stdout) > > File > > > "/home1/00924/Reddy135/LocalApps/petsc/config/BuildSystem/config/framework.py", > > line 933, in configure > > child.configure() > > File > > > "/home1/00924/Reddy135/LocalApps/petsc/config/BuildSystem/config/package.py", > > line 552, in configure > > self.consistencyChecks() > > File "/home1/00924/Reddy135/LocalApps/petsc/config/PETSc/package.py", > > line 38, in consistencyChecks > > raise RuntimeError('Cannot use '+self.name+' with complex numbers > it is > > not coded for this capability') > > > > > > > > > > On Fri, Jun 14, 2013 at 11:29 AM, Satish Balay > wrote: > > > > > On Thu, 13 Jun 2013, Dharmendar Reddy wrote: > > > > > > > Hello, > > > > I rerun the configure, it works. I had to remove the > ctetgen, > > > > metis and parametis > > > > > > I don't unserstand why you had to remove them. > > > > > > > Is there documentation on dependencies for external > > > > packages ? If not, it will be useful to have a required package list > for > > > a > > > > given external package. > > > > > > its currently documented in the appropriate .py file for that package. > > > [and if the dependency is missing - you get a configure error] > > > > > > Satish > > > > > > > > > > > > > -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. Homepage: https://webspace.utexas.edu/~dpr342 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jefonseca at gmail.com Sat Jun 15 00:53:19 2013 From: jefonseca at gmail.com (Jim Fonseca) Date: Sat, 15 Jun 2013 01:53:19 -0400 Subject: [petsc-users] PETSC_VERSION_LT not giving expected result Message-ID: Hi, In http://www.mcs.anl.gov/petsc/petsc-current/include/petscversion.h there is: #define PETSC_VERSION_LT(MAJOR,MINOR,SUBMINOR) \ (PETSC_VERSION_RELEASE == 0 && \ (PETSC_VERSION_MAJOR < (MAJOR) || \ (PETSC_VERSION_MAJOR == (MAJOR) && \ (PETSC_VERSION_MINOR < (MINOR) || \ (PETSC_VERSION_MINOR == (MINOR) && \ (PETSC_VERSION_SUBMINOR < (SUBMINOR))))))) Won't this always be false for release versions? We're trying to move from 3.2 and the following is evaluating to true. (I've put the new additions to petscversion.h into that file in our older builds). #if PETSC_VERSION_GE(3,3,0) //we should not get here, but we do when petscversion.h has #define PETSC_VERSION_RELEASE 1 #define PETSC_VERSION_MAJOR 3 #define PETSC_VERSION_MINOR 2 #define PETSC_VERSION_SUBMINOR 0 #define PETSC_VERSION_PATCH 6 Thanks, Jim -- Jim Fonseca, PhD Research Scientist Network for Computational Nanotechnology Purdue University 765-496-6495 www.jimfonseca.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Jun 15 13:51:01 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 15 Jun 2013 20:51:01 +0200 Subject: [petsc-users] PETSC_VERSION_LT not giving expected result In-Reply-To: References: Message-ID: Yikes, that should be RELEASE==1. Looks like I totally botched this: https://bitbucket.org/petsc/petsc/commits/03354681f90b4c94467f09fd38f90b774d661d87 Will fix when I have a keyboard. But how can you have this macro with 3.2? On Jun 15, 2013 7:54 AM, "Jim Fonseca" wrote: > Hi, > In http://www.mcs.anl.gov/petsc/petsc-current/include/petscversion.h > there is: > > > > #define PETSC_VERSION_LT(MAJOR,MINOR,SUBMINOR) \ > (PETSC_VERSION_RELEASE == 0 && \ > (PETSC_VERSION_MAJOR < (MAJOR) || \ > (PETSC_VERSION_MAJOR == (MAJOR) && \ > (PETSC_VERSION_MINOR < (MINOR) || \ > (PETSC_VERSION_MINOR == (MINOR) && \ > (PETSC_VERSION_SUBMINOR < (SUBMINOR))))))) > > > Won't this always be false for release versions? > > We're trying to move from 3.2 and the following is evaluating to true. (I've put the new additions to petscversion.h into that file in our older builds). > > #if PETSC_VERSION_GE(3,3,0) > > //we should not get here, but we do > > > when petscversion.h has > > #define PETSC_VERSION_RELEASE 1 > #define PETSC_VERSION_MAJOR 3 > #define PETSC_VERSION_MINOR 2 > #define PETSC_VERSION_SUBMINOR 0 > #define PETSC_VERSION_PATCH 6 > > Thanks, > > Jim > > > -- > Jim Fonseca, PhD > Research Scientist > Network for Computational Nanotechnology > Purdue University > 765-496-6495 > www.jimfonseca.com > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Jun 16 14:15:04 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 16 Jun 2013 21:15:04 +0200 Subject: [petsc-users] PETSC_VERSION_LT not giving expected result In-Reply-To: References: Message-ID: <87vc5d7tqf.fsf@mcs.anl.gov> Please always use "reply-all" so that your messages go to the list. This is standard mailing list etiquette. It is important to preserve threading for people who find this discussion later and so that we do not waste our time re-answering the same questions that have already been answered in private side-conversations. You'll likely get an answer faster that way too. Jim Fonseca writes: > Hi Jed, > Thanks. Well, I was just going to paste the new directive functions into > the petscversion.h we have for our current PETSc 3.2 builds. Okay, thanks for explaining. This is now fixed in 'maint' and 'master'. > Yes, I know you probably cringe when you read we are still using > that. :) I wanted to use PETSC_VERSION_GE moving forward since that > seemed a lot nicer than comparing the version numbers one by one (i.e > 3.2, 3.3, 3.4, petsc-dev) I want to try the make the transition from > 3.2 to 3.4 as smooth as possible since we run on a few systems and it > will be difficult to update all our builds simultaneously (some are > cluster modules, some use our own PETSc builds) and test them. Yup, understood. Depending on how your code includes PETSc headers, you can do something like this: #include /* or whatever */ #if !defined(PETSC_VERSION_LT) #define PETSC_VERSION_LT(MAJOR,MINOR,SUBMINOR) \ (PETSC_VERSION_RELEASE == 1 && \ (PETSC_VERSION_MAJOR < (MAJOR) || \ (PETSC_VERSION_MAJOR == (MAJOR) && \ (PETSC_VERSION_MINOR < (MINOR) || \ (PETSC_VERSION_MINOR == (MINOR) && \ (PETSC_VERSION_SUBMINOR < (SUBMINOR))))))) #endif ... other stuff for you application > > Thanks, > Jim > > > On Sat, Jun 15, 2013 at 2:51 PM, Jed Brown wrote: > >> Yikes, that should be RELEASE==1. Looks like I totally botched this: >> >> >> https://bitbucket.org/petsc/petsc/commits/03354681f90b4c94467f09fd38f90b774d661d87 >> >> Will fix when I have a keyboard. >> >> But how can you have this macro with 3.2? >> On Jun 15, 2013 7:54 AM, "Jim Fonseca" wrote: >> >>> Hi, >>> In http://www.mcs.anl.gov/petsc/petsc-current/include/petscversion.h >>> there is: >>> >>> >>> >>> >>> #define PETSC_VERSION_LT(MAJOR,MINOR,SUBMINOR) \ >>> (PETSC_VERSION_RELEASE == 0 && \ >>> (PETSC_VERSION_MAJOR < (MAJOR) || \ >>> (PETSC_VERSION_MAJOR == (MAJOR) && \ >>> (PETSC_VERSION_MINOR < (MINOR) || \ >>> (PETSC_VERSION_MINOR == (MINOR) && \ >>> (PETSC_VERSION_SUBMINOR < (SUBMINOR))))))) >>> >>> >>> Won't this always be false for release versions? >>> >>> We're trying to move from 3.2 and the following is evaluating to true. (I've put the new additions to petscversion.h into that file in our older builds). >>> >>> #if PETSC_VERSION_GE(3,3,0) >>> >>> //we should not get here, but we do >>> >>> >>> when petscversion.h has >>> >>> #define PETSC_VERSION_RELEASE 1 >>> #define PETSC_VERSION_MAJOR 3 >>> #define PETSC_VERSION_MINOR 2 >>> #define PETSC_VERSION_SUBMINOR 0 >>> #define PETSC_VERSION_PATCH 6 >>> >>> Thanks, >>> >>> Jim >>> >>> >>> -- >>> Jim Fonseca, PhD >>> Research Scientist >>> Network for Computational Nanotechnology >>> Purdue University >>> 765-496-6495 >>> www.jimfonseca.com >>> >>> >>> > > > -- > Jim Fonseca, PhD > Research Scientist > Network for Computational Nanotechnology > Purdue University > 765-496-6495 > www.jimfonseca.com From jefonseca at gmail.com Sun Jun 16 16:20:40 2013 From: jefonseca at gmail.com (Jim Fonseca) Date: Sun, 16 Jun 2013 17:20:40 -0400 Subject: [petsc-users] PETSC_VERSION_LT not giving expected result In-Reply-To: <87vc5d7tqf.fsf@mcs.anl.gov> References: <87vc5d7tqf.fsf@mcs.anl.gov> Message-ID: Of course. I clicked the wrong button. Thanks for the tips. Jim On Sun, Jun 16, 2013 at 3:15 PM, Jed Brown wrote: > Please always use "reply-all" so that your messages go to the list. > This is standard mailing list etiquette. It is important to preserve > threading for people who find this discussion later and so that we do > not waste our time re-answering the same questions that have already > been answered in private side-conversations. You'll likely get an > answer faster that way too. > > Jim Fonseca writes: > > > Hi Jed, > > Thanks. Well, I was just going to paste the new directive functions into > > the petscversion.h we have for our current PETSc 3.2 builds. > > Okay, thanks for explaining. This is now fixed in 'maint' and 'master'. > > > Yes, I know you probably cringe when you read we are still using > > that. :) I wanted to use PETSC_VERSION_GE moving forward since that > > seemed a lot nicer than comparing the version numbers one by one (i.e > > 3.2, 3.3, 3.4, petsc-dev) I want to try the make the transition from > > 3.2 to 3.4 as smooth as possible since we run on a few systems and it > > will be difficult to update all our builds simultaneously (some are > > cluster modules, some use our own PETSc builds) and test them. > > Yup, understood. Depending on how your code includes PETSc headers, you > can do something like this: > > #include /* or whatever */ > > #if !defined(PETSC_VERSION_LT) > #define PETSC_VERSION_LT(MAJOR,MINOR,SUBMINOR) \ > (PETSC_VERSION_RELEASE == 1 && \ > (PETSC_VERSION_MAJOR < (MAJOR) || \ > (PETSC_VERSION_MAJOR == (MAJOR) && \ > (PETSC_VERSION_MINOR < (MINOR) || \ > (PETSC_VERSION_MINOR == (MINOR) && \ > (PETSC_VERSION_SUBMINOR < (SUBMINOR))))))) > #endif > > ... other stuff for you application > > > > > Thanks, > > Jim > > > > > > On Sat, Jun 15, 2013 at 2:51 PM, Jed Brown wrote: > > > >> Yikes, that should be RELEASE==1. Looks like I totally botched this: > >> > >> > >> > https://bitbucket.org/petsc/petsc/commits/03354681f90b4c94467f09fd38f90b774d661d87 > >> > >> Will fix when I have a keyboard. > >> > >> But how can you have this macro with 3.2? > >> On Jun 15, 2013 7:54 AM, "Jim Fonseca" wrote: > >> > >>> Hi, > >>> In http://www.mcs.anl.gov/petsc/petsc-current/include/petscversion.h > >>> there is: > >>> > >>> > >>> > >>> > >>> #define PETSC_VERSION_LT(MAJOR,MINOR,SUBMINOR) \ > >>> (PETSC_VERSION_RELEASE == 0 && \ > >>> (PETSC_VERSION_MAJOR < (MAJOR) || \ > >>> (PETSC_VERSION_MAJOR == (MAJOR) && \ > >>> (PETSC_VERSION_MINOR < (MINOR) || \ > >>> (PETSC_VERSION_MINOR == (MINOR) && \ > >>> (PETSC_VERSION_SUBMINOR < (SUBMINOR))))))) > >>> > >>> > >>> Won't this always be false for release versions? > >>> > >>> We're trying to move from 3.2 and the following is evaluating to true. > (I've put the new additions to petscversion.h into that file in our older > builds). > >>> > >>> #if PETSC_VERSION_GE(3,3,0) > >>> > >>> //we should not get here, but we do > >>> > >>> > >>> when petscversion.h has > >>> > >>> #define PETSC_VERSION_RELEASE 1 > >>> #define PETSC_VERSION_MAJOR 3 > >>> #define PETSC_VERSION_MINOR 2 > >>> #define PETSC_VERSION_SUBMINOR 0 > >>> #define PETSC_VERSION_PATCH 6 > >>> > >>> Thanks, > >>> > >>> Jim > >>> > >>> > >>> -- > >>> Jim Fonseca, PhD > >>> Research Scientist > >>> Network for Computational Nanotechnology > >>> Purdue University > >>> 765-496-6495 > >>> www.jimfonseca.com > >>> > >>> > >>> > > > > > > -- > > Jim Fonseca, PhD > > Research Scientist > > Network for Computational Nanotechnology > > Purdue University > > 765-496-6495 > > www.jimfonseca.com > -- Jim Fonseca, PhD Research Scientist Network for Computational Nanotechnology Purdue University 765-496-6495 www.jimfonseca.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Jun 17 03:31:24 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 17 Jun 2013 10:31:24 +0200 Subject: [petsc-users] Deflated GMRES in PETSc In-Reply-To: References: Message-ID: <87bo75kujn.fsf@mcs.anl.gov> [Cc'ing petsc-users since people may be interested.] Matteo Parsani writes: > Hi Jed, > do you know if in PETSc we can start a Krylov iteration with problematic > eigenmodes in the Krylov subspace at the beginning of the iteration? Deflation is identical to a Galerkin coarse grid correction. I would use PCMG to create a two-level method. Write your deflation space into the prolongation operator and use Galerkin. You can compose a PCMG with other preconditioners. I would leave DGMRES to only deal with adaptive identification of those problematic modes. Note that the DGMRES basis vectors will be global, but when you provide vectors a priori, you are free to localize them. > Machinery should be there already because we see that this is used when > GMRES stagnates. > > Thanks, > > -- > Matteo From domenico_lahaye at yahoo.com Mon Jun 17 03:42:48 2013 From: domenico_lahaye at yahoo.com (domenico lahaye) Date: Mon, 17 Jun 2013 01:42:48 -0700 (PDT) Subject: [petsc-users] Deflated GMRES in PETSc In-Reply-To: <87bo75kujn.fsf@mcs.anl.gov> References: <87bo75kujn.fsf@mcs.anl.gov> Message-ID: <1371458568.12757.YahooMailNeo@web125706.mail.ne1.yahoo.com> FYI: an example of the implementation of deflated GMRES in PETSc can be found at: http://onlinelibrary.wiley.com/doi/10.1002/nla.1882/full Possibly this helps. Kind wishes, Domenico Lahaye. ________________________________ From: Jed Brown To: Matteo Parsani Cc: Mark H. Carpenter ; petsc-users Sent: Monday, June 17, 2013 10:31 AM Subject: Re: [petsc-users] Deflated GMRES in PETSc [Cc'ing petsc-users since people may be interested.] Matteo Parsani writes: > Hi Jed, > do you know if in PETSc we can start a Krylov iteration with problematic > eigenmodes in the Krylov subspace at the beginning of the iteration? Deflation is identical to a Galerkin coarse grid correction.? I would use PCMG to create a two-level method.? Write your deflation space into the prolongation operator and use Galerkin.? You can compose a PCMG with other preconditioners.? I would leave DGMRES to only deal with adaptive identification of those problematic modes.? Note that the DGMRES basis vectors will be global, but when you provide vectors a priori, you are free to localize them. > Machinery should be there already because we see that this is used when > GMRES stagnates. > > Thanks, > > -- > Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From frtr at fysik.dtu.dk Mon Jun 17 03:59:49 2013 From: frtr at fysik.dtu.dk (Frederik Treue) Date: Mon, 17 Jun 2013 10:59:49 +0200 Subject: [petsc-users] petscviewerbinaryread Message-ID: <1371459589.16479.15.camel@frtr-laptop> Hi guys, is petscviewerbinaryread working? The examples given at the webpage either fails with out of memory error issues (ex65 and ex65dm) or doesn't compile (ex61) for me? Oddly I have problems only when trying to use mpi. My code: ierr=DMDACreate1d(PETSC_COMM_WORLD,DMDA_BOUNDARY_GHOSTED,156,1,1,PETSC_NULL,&da);CHKERRQ(ierr); ierr=DMCreateGlobalVector(da,&ReRead);CHKERRQ(ierr); ierr=VecAssemblyBegin(ReRead);CHKERRQ(ierr); ierr=VecAssemblyEnd(ReRead);CHKERRQ(ierr); double *workarray=(double*)malloc(156*sizeof(double)); ierr = VecGetArray(ReRead,&workarray); CHKERRQ(ierr); ierr=PetscViewerBinaryOpen(PETSC_COMM_WORLD,"./testvector",FILE_MODE_READ,&fileview);CHKERRQ(ierr); ierr=PetscViewerBinaryRead(fileview,&dummy,1,PETSC_SCALAR);CHKERRQ(ierr); ierr=PetscViewerBinaryRead(fileview,(void*)workarray,156,PETSC_SCALAR);CHKERRQ(ierr); printf("TEST: %g\n",workarray[144]); ierr=VecRestoreArray(ReRead,&workarray); VecView(ReRead,PETSC_VIEWER_DRAW_WORLD); This works fine as long as I'm on a single processor. The file read also works on mpi with 2 processors, as evidenced by the result of the printf statement (which gives the right result from both processors). However the VecView statement fails with "Vector not generated from a DMDA!". Why? And second question: How do I generalize to 2d? Can I give a 2d array to PetscViewerBinaryRead and expect it to work? And how should it be malloc'ed? Or does one give a 1d array to PetscViewerBinaryRead and let VecRestoreArray do the rest? Or what? /Frederik Treue From knepley at gmail.com Mon Jun 17 04:06:01 2013 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Jun 2013 11:06:01 +0200 Subject: [petsc-users] petscviewerbinaryread In-Reply-To: <1371459589.16479.15.camel@frtr-laptop> References: <1371459589.16479.15.camel@frtr-laptop> Message-ID: On Mon, Jun 17, 2013 at 10:59 AM, Frederik Treue wrote: > Hi guys, > > is petscviewerbinaryread working? The examples given at the webpage > either fails with out of memory error issues (ex65 and ex65dm) or > doesn't compile (ex61) for me? > > Oddly I have problems only when trying to use mpi. My code: > > > ierr=DMDACreate1d(PETSC_COMM_WORLD,DMDA_BOUNDARY_GHOSTED,156,1,1,PETSC_NULL,&da);CHKERRQ(ierr); > > ierr=DMCreateGlobalVector(da,&ReRead);CHKERRQ(ierr); > > ierr=VecAssemblyBegin(ReRead);CHKERRQ(ierr); > ierr=VecAssemblyEnd(ReRead);CHKERRQ(ierr); > > double *workarray=(double*)malloc(156*sizeof(double)); > > ierr = VecGetArray(ReRead,&workarray); CHKERRQ(ierr); > 1) This does not make sense. You allocate an array, but then overwrite that array with the one inside the vector ReRead. > > ierr=PetscViewerBinaryOpen(PETSC_COMM_WORLD,"./testvector",FILE_MODE_READ,&fileview);CHKERRQ(ierr); > > ierr=PetscViewerBinaryRead(fileview,&dummy,1,PETSC_SCALAR);CHKERRQ(ierr); > > > ierr=PetscViewerBinaryRead(fileview,(void*)workarray,156,PETSC_SCALAR);CHKERRQ(ierr); > printf("TEST: %g\n",workarray[144]); > ierr=VecRestoreArray(ReRead,&workarray); > 2) In parallel, the local array in your vector ReRead will be smaller than the global size 156. Thus this read also does not makes sense. Matt > VecView(ReRead,PETSC_VIEWER_DRAW_WORLD); > > This works fine as long as I'm on a single processor. The file read also > works on mpi with 2 processors, as evidenced by the result of the printf > statement (which gives the right result from both processors). However > the VecView statement fails with "Vector not generated from a DMDA!". > Why? > > And second question: How do I generalize to 2d? Can I give a 2d array to > PetscViewerBinaryRead and expect it to work? And how should it be > malloc'ed? Or does one give a 1d array to PetscViewerBinaryRead and let > VecRestoreArray do the rest? Or what? > > /Frederik Treue > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From frtr at fysik.dtu.dk Mon Jun 17 07:47:28 2013 From: frtr at fysik.dtu.dk (Frederik Treue) Date: Mon, 17 Jun 2013 14:47:28 +0200 Subject: [petsc-users] petscviewerbinaryread In-Reply-To: References: <1371459589.16479.15.camel@frtr-laptop> Message-ID: <1371473248.16479.23.camel@frtr-laptop> OK, so I got a little further: Now I can read 1D fields on any amount of processors, and 2D fields on 1 processor :). My code: MPI_Comm_size(PETSC_COMM_WORLD,&cs); MPI_Comm_rank(MPI_COMM_WORLD,&rank); ierr = VecGetArray(dummy,&dump); CHKERRQ(ierr); ierr = VecGetArray(ReRead,&readarray); CHKERRQ(ierr); ierr=PetscViewerBinaryOpen(PETSC_COMM_WORLD,"./testvector",FILE_MODE_READ,&fileview);CHKERRQ(ierr); for (i=0;i On Mon, Jun 17, 2013 at 10:59 AM, Frederik Treue > wrote: > Hi guys, > > is petscviewerbinaryread working? The examples given at the > webpage > either fails with out of memory error issues (ex65 and ex65dm) > or > doesn't compile (ex61) for me? > > Oddly I have problems only when trying to use mpi. My code: > > ierr=DMDACreate1d(PETSC_COMM_WORLD,DMDA_BOUNDARY_GHOSTED,156,1,1,PETSC_NULL,&da);CHKERRQ(ierr); > > ierr=DMCreateGlobalVector(da,&ReRead);CHKERRQ(ierr); > > ierr=VecAssemblyBegin(ReRead);CHKERRQ(ierr); > ierr=VecAssemblyEnd(ReRead);CHKERRQ(ierr); > > double *workarray=(double*)malloc(156*sizeof(double)); > > ierr = VecGetArray(ReRead,&workarray); CHKERRQ(ierr); > > > 1) This does not make sense. You allocate an array, but then overwrite > that > array with the one inside the vector ReRead. > > ierr=PetscViewerBinaryOpen(PETSC_COMM_WORLD,"./testvector",FILE_MODE_READ,&fileview);CHKERRQ(ierr); > > ierr=PetscViewerBinaryRead(fileview,&dummy,1,PETSC_SCALAR);CHKERRQ(ierr); > > ierr=PetscViewerBinaryRead(fileview,(void*)workarray,156,PETSC_SCALAR);CHKERRQ(ierr); > printf("TEST: %g\n",workarray[144]); > ierr=VecRestoreArray(ReRead,&workarray); > > > 2) In parallel, the local array in your vector ReRead will be smaller > than the global size 156. Thus this read also > does not makes sense. > > > Matt > > VecView(ReRead,PETSC_VIEWER_DRAW_WORLD); > > This works fine as long as I'm on a single processor. The file > read also > works on mpi with 2 processors, as evidenced by the result of > the printf > statement (which gives the right result from both processors). > However > the VecView statement fails with "Vector not generated from a > DMDA!". > Why? > > And second question: How do I generalize to 2d? Can I give a > 2d array to > PetscViewerBinaryRead and expect it to work? And how should it > be > malloc'ed? Or does one give a 1d array to > PetscViewerBinaryRead and let > VecRestoreArray do the rest? Or what? > > /Frederik Treue > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From knepley at gmail.com Mon Jun 17 09:13:29 2013 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Jun 2013 16:13:29 +0200 Subject: [petsc-users] petscviewerbinaryread In-Reply-To: <1371473248.16479.23.camel@frtr-laptop> References: <1371459589.16479.15.camel@frtr-laptop> <1371473248.16479.23.camel@frtr-laptop> Message-ID: On Mon, Jun 17, 2013 at 2:47 PM, Frederik Treue wrote: > OK, so I got a little further: Now I can read 1D fields on any amount of > processors, and 2D fields on 1 processor :). My code: > What are you trying to do? Why not just use VecView() and VecLoad(), which work in parallel, and are scalable. I doubt you want to reproduce that code. Matt > MPI_Comm_size(PETSC_COMM_WORLD,&cs); > MPI_Comm_rank(MPI_COMM_WORLD,&rank); > > ierr = VecGetArray(dummy,&dump); CHKERRQ(ierr); > ierr = VecGetArray(ReRead,&readarray); CHKERRQ(ierr); > > ierr=PetscViewerBinaryOpen(PETSC_COMM_WORLD,"./testvector",FILE_MODE_READ,&fileview);CHKERRQ(ierr); > > for (i=0;i printf("Rank BEFORE: %d\n",rank); > > ierr=PetscViewerBinaryRead(fileview,(void*)dump,nx*ny/cs,PETSC_SCALAR);CHKERRQ(ierr); > } > > > ierr=PetscViewerBinaryRead(fileview,(void*)readarray,nx*ny/cs,PETSC_SCALAR);CHKERRQ(ierr); > > for (i=rank+1;i printf("Rank: AFTER: %d\n",rank); > > ierr=PetscViewerBinaryRead(fileview,(void*)dump,nx*ny/cs,PETSC_SCALAR);CHKERRQ(ierr); > } > > However, this fails for 2D with more than one processor: The resulting > vector is garbled and I get memory corruption. Am I on the right track, > or is there another way to achieve an MPI version of binary read? The > above code seems somewhat cumbersome... > > /Frederik Treue > > On Mon, 2013-06-17 at 11:06 +0200, Matthew Knepley wrote: > > On Mon, Jun 17, 2013 at 10:59 AM, Frederik Treue > > wrote: > > Hi guys, > > > > is petscviewerbinaryread working? The examples given at the > > webpage > > either fails with out of memory error issues (ex65 and ex65dm) > > or > > doesn't compile (ex61) for me? > > > > Oddly I have problems only when trying to use mpi. My code: > > > > > ierr=DMDACreate1d(PETSC_COMM_WORLD,DMDA_BOUNDARY_GHOSTED,156,1,1,PETSC_NULL,&da);CHKERRQ(ierr); > > > > ierr=DMCreateGlobalVector(da,&ReRead);CHKERRQ(ierr); > > > > ierr=VecAssemblyBegin(ReRead);CHKERRQ(ierr); > > ierr=VecAssemblyEnd(ReRead);CHKERRQ(ierr); > > > > double *workarray=(double*)malloc(156*sizeof(double)); > > > > ierr = VecGetArray(ReRead,&workarray); CHKERRQ(ierr); > > > > > > 1) This does not make sense. You allocate an array, but then overwrite > > that > > array with the one inside the vector ReRead. > > > > > ierr=PetscViewerBinaryOpen(PETSC_COMM_WORLD,"./testvector",FILE_MODE_READ,&fileview);CHKERRQ(ierr); > > > > > ierr=PetscViewerBinaryRead(fileview,&dummy,1,PETSC_SCALAR);CHKERRQ(ierr); > > > > > ierr=PetscViewerBinaryRead(fileview,(void*)workarray,156,PETSC_SCALAR);CHKERRQ(ierr); > > printf("TEST: %g\n",workarray[144]); > > ierr=VecRestoreArray(ReRead,&workarray); > > > > > > 2) In parallel, the local array in your vector ReRead will be smaller > > than the global size 156. Thus this read also > > does not makes sense. > > > > > > Matt > > > > VecView(ReRead,PETSC_VIEWER_DRAW_WORLD); > > > > This works fine as long as I'm on a single processor. The file > > read also > > works on mpi with 2 processors, as evidenced by the result of > > the printf > > statement (which gives the right result from both processors). > > However > > the VecView statement fails with "Vector not generated from a > > DMDA!". > > Why? > > > > And second question: How do I generalize to 2d? Can I give a > > 2d array to > > PetscViewerBinaryRead and expect it to work? And how should it > > be > > malloc'ed? Or does one give a 1d array to > > PetscViewerBinaryRead and let > > VecRestoreArray do the rest? Or what? > > > > /Frederik Treue > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fsantost at student.ethz.ch Mon Jun 17 10:40:50 2013 From: fsantost at student.ethz.ch (Santos Teixeira Frederico) Date: Mon, 17 Jun 2013 15:40:50 +0000 Subject: [petsc-users] Error in PETSc with MUMPS on Windows. In-Reply-To: References: <682CC3CD7A208742B8C2D116C67199010B19D928@MBX13.d.ethz.ch>, Message-ID: <682CC3CD7A208742B8C2D116C67199010B1A030E@MBX13.d.ethz.ch> >> [0]PETSC ERROR: [0] MatLUFactorSymbolic_AIJMUMPS line 880 src/mat/impls/aij/mpi/mumps/C:\PETSC-~1.1\src\mat\impls\aij\mpi\mumps\mumps.c > So its crashing somewhere in MatLUFactorSymbolic_AIJMUMPS() - which > has calls to mumps routines. I found out that the problems was the call PetscMUMPS_c(&mumps_id) (file mumps.c, line 959). So, a problem with MUMPS... I contacted the MUMPS team and they suggest me to downgrade to METIS 4.X (and ParMETIS 3.X), once METIS 5.X is not supported yet. By the way, which are the METIS and ParMETIS versions that PETSc install with "--download-***=1"? I used this option on Linux and everything worked normally. > > --with-mpi-dir=/cygdrive/d/Solvers-dev/Solvers/libs/MPI /cygdrive/d/Solvers-dev/Solvers/libs/MPI/lib/fmpich2.lib > Looks like you are using mpich2 - but not standard install? Yes, this is the standard mpich2. > And Its not clear how one would use blacs/scalapack from mkl. Is it > supporsed to be compatible with this version of mpich you have? [I > have no idea] MKL website says that it compiles with MPICH2 1.x.x. > Such mixing of mpi compiled codes [where packages are compiled with > different variants of mpi - but combined into a single binary] can > potentially have issues. > And you can do debugging on windows. For sequential you do: > msdev binary.exe > [or whatever the current name for the developer studio is.Its 'devenv' for VC2008, and something elese for VC2012] > And if you wish to debug parallely - you would have to compile the > application with the developer studio project files - and perhaps > follow instructions from: > http://www.mpich.org/static/downloads/1.4.1p1/mpich2-1.4.1p1-windevguide.pdf Thanks the tips! They were very useful! > Satish Regards, Frederico. On Fri, 14 Jun 2013, Santos Teixeira Frederico wrote: > Hi, > > I got the error below when I tried to execute PETSc with MUMPS. Some important facts: > > 1) the following libraries/versions were compiled separatedly: METIS: 5.1.0, ParMETIS: 4.0.3, MUMPS: 4.10.0 and added to PETSc 3.4.1 along with ScaLAPACK 2.0.2 (plus BLAS, LAPACK, BLACS etc. - from latest MKL). > > 2) the same code works correctly on Windows with Pardiso (interfaced with PETSc). > > 3) the same code works correctly on Linux with Pardiso and the libraries provided by --download-***. > > 4) the MUMPS libraries and its dependencies (the same described below) were linked and correctly executed a test provided by the own library. > > Despite the lack of informations and debugger (Windows...), could you give some tip and/or guide me to provide you better informations? I appreciate any tip! > > Thanks a lot! > > Regards, > Frederico. > > ====================================================================== > > Starting KSPSolve with MUMPS... > Entering DMUMPS driver with JOB, N, NZ = 1 74355 2782474 > > DMUMPS 4.10.0 > L U Solver for unsymmetric matrices > Type of parallelism: Working host > > ****** ANALYSIS STEP ******** > > Resetting candidate strategy to 0 because NSLAVES=1 > > ... Structural symmetry (in percent)= 93 > Density: NBdense, Average, Median = 0 39 29 > ... No column permutation > Ordering based on METIS > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] MatLUFactorSymbolic_AIJMUMPS line 880 src/mat/impls/aij/mpi/mumps/C:\PETSC-~1.1\src\mat\impls\aij\mpi\mumps\mumps.c > [0]PETSC ERROR: [0] MatLUFactorSymbolic line 2820 src/mat/interface/C:\PETSC-~1.1\src\mat\INTERF~1\matrix.c > [0]PETSC ERROR: [0] PCSetUp_LU line 99 src/ksp/pc/impls/factor/lu/C:\PETSC-~1.1\src\ksp\pc\impls\factor\lu\lu.c > [0]PETSC ERROR: [0] PCSetUp line 868 src/ksp/pc/interface/C:\PETSC-~1.1\src\ksp\pc\INTERF~1\precon.c > [0]PETSC ERROR: [0] KSPSetUp line 192 src/ksp/ksp/interface/C:\PETSC-~1.1\src\ksp\ksp\INTERF~1\itfunc.c > [0]PETSC ERROR: [0] KSPSolve line 356 src/ksp/ksp/interface/C:\PETSC-~1.1\src\ksp\ksp\INTERF~1\itfunc.c > [0]PETSC ERROR: [0] FluidSolverDirect::Solve line 1145 "unknowndirectory/"..\..\..\Source\FluidSolver\FluidSolverDirect\FluidSolverDirect.cpp > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: D:\Solvers-dev\Solvers\builds\bin\Debug\FluidSolverDirectCavityDriver.exe on a win-x64-msvc-mkl-real-release named FREDERICO-PC by Frederico Teixeira Fri Jun 14 17:53:16 2013 > [0]PETSC ERROR: Libraries linked from /cygdrive/c/petsc-3.4.1/win-x64-msvc-mkl-real-release/lib > [0]PETSC ERROR: Configure run at Fri Jun 14 15:34:42 2013 > [0]PETSC ERROR: Configure options PETSC_ARCH=win-x64-msvc-mkl-real-dbg --with-cc="win32fe cl" --with-cxx="win32fe cl" --with-fc=0 --with-x=0 --with-debugging=1 --with-mpi-dir=/cygdrive/d/Solvers-dev/Solvers/libs/MPI --with-blas-lapack-lib="[/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_bl > as95_lp64.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_lapack95_lp64.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_intel_lp64.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_intel_thread.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_core.lib]" --with-metis- > lib=/cygdrive/d/Solvers-dev/Solvers/libs/METIS/win-x64-msvc/lib/metis.lib --with-metis-include=/cygdrive/d/Solvers-dev/Solvers/libs/METIS/include --with-parmetis-lib=/cygdrive/d/Solvers-dev/Solvers/libs/PARMETIS/win-x64-msvc/lib/parmetis.lib --with-parmetis-include=/cygdrive/d/Solvers-dev/Solvers/li > bs/PARMETIS/include --with-scalapack-lib="[/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_scalapack_lp64.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MKL/x64/lib/mkl_blacs_mpich2_lp64.lib]" --with-scalapack-include=/cygdrive/d/Solvers-dev/Solvers/libs/MKL/include --with-mumps-lib="[/cygdrive/d/Sol > vers-dev/Solvers/libs/MUMPS/win-x64-msvc-mkl/lib/dmumps.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MUMPS/win-x64-msvc-mkl/lib/mumps-common.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MUMPS/win-x64-msvc-mkl/lib/pord.lib,/cygdrive/d/Solvers-dev/Solvers/libs/MUMPS/win-x64-msvc-mkl/lib/dmumps-f.lib]" --with- > mumps-include=/cygdrive/d/Solvers-dev/Solvers/libs/MUMPS/include2 --with-hypre-dir=/cygdrive/d/Solvers-dev/Solvers/libs/HYPRE -CFLAGS=-MD -CXXFLAGS=-MD --LIBS="/cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/ifconsol.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libifcore > md.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libifportmd.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libiomp5md.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libirc.lib /cygdrive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/libmmd.lib /cyg > drive/d/Solvers-dev/Solvers/libs/IntelCompiler/x64/lib/svml_dispmd.lib /cygdrive/d/Solvers-dev/Solvers/libs/MPI/lib/fmpich2.lib" --useThreads=0 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > job aborted: > rank: node: exit code[: error message] > 0: Frederico-PC: 59: process 0 exited without calling finalize > From a.vergottis at gmail.com Mon Jun 17 11:37:28 2013 From: a.vergottis at gmail.com (Anthony Vergottis) Date: Mon, 17 Jun 2013 17:37:28 +0100 Subject: [petsc-users] KSP Solver Initial Non Zero Guess Message-ID: Dear All, If I am solving a linear system uses the KSP solver context i.e. *ierr = KSPSolve(ksp,RHS,**Solution**);* and I require an non zero initial guess for the system is it sufficient to call *ierr = KSPSetInitialGuessNonzero(ksp,nonzeroguess);* and supply the vector *Solution* with the values I require? will the initial non zero guess be taken as the values I specified in the vector *Solution*? or is there something else that must be done to achieve this? Just need to make sure of this. Thanks in advance for any help. Regards, Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jun 17 13:06:39 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 17 Jun 2013 13:06:39 -0500 Subject: [petsc-users] KSP Solver Initial Non Zero Guess In-Reply-To: References: Message-ID: <8E7DA05C-512F-4014-840B-D2CE5BC1E524@mcs.anl.gov> Yes. Barry Normally KSPSolve() zeros the solution vector on entry. If you have called KSPSetInitialGuessNonzero(ksp,PETSC_TRUE) then it skips that zeroing step. On Jun 17, 2013, at 11:37 AM, Anthony Vergottis wrote: > Dear All, > > If I am solving a linear system uses the KSP solver context i.e. ierr = KSPSolve(ksp,RHS,Solution); and I require an non zero initial guess for the system is it sufficient to call ierr = KSPSetInitialGuessNonzero(ksp,nonzeroguess); and supply the vector > Solution with the values I require? will the initial non zero guess be taken as the values I specified in the vector Solution? or is there something else that must be done to achieve this? > > Just need to make sure of this. Thanks in advance for any help. > > Regards, > Anthony From bclyons at princeton.edu Tue Jun 18 14:10:06 2013 From: bclyons at princeton.edu (Brendan C Lyons) Date: Tue, 18 Jun 2013 15:10:06 -0400 Subject: [petsc-users] SEGV on KSPSolve with mutliple processors Message-ID: Hi everyone, I've run into a strange problem in my Fortran 90 code where it runs fine with 1 processor, but then throws a segmentation fault on KSPSolve() when I try to run it in parallel. I'm using PETSc 3.3 with the SuperLU direct solver for the sequential case and SuperLU_dist for the parallel case. I've called KSPView before and after KSPSolve. I'll put the KSPView output for the sequential and parallel cases and the crash info for the parallel case below (with some details of my system redacted). Any help would be appreciated. If you need any other information, I'm happy to provide it. Thank you, ~Brendan ------------------------------ KSPView() before sequential solve: KSP Object: 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using DEFAULT norm type for convergence test PC Object: 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=11760, cols=11760 total: nonzeros=506586, allocated nonzeros=509061 total number of mallocs used during MatSetValues calls =0 not using I-node routines KSPView() after sequential solve: KSP Object: 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 0, needed 0 Factored matrix follows: Matrix Object: 1 MPI processes type: seqaij rows=11760, cols=11760 package used to perform factorization: superlu total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 SuperLU run parameters: Equil: NO ColPerm: 3 IterRefine: 0 SymmetricMode: NO DiagPivotThresh: 1 PivotGrowth: NO ConditionNumber: NO RowPerm: 0 ReplaceTinyPivot: NO PrintStat: NO lwork: 0 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=11760, cols=11760 total: nonzeros=506586, allocated nonzeros=509061 total number of mallocs used during MatSetValues calls =0 not using I-node routines KSPView() before parallel solve: KSP Object: 2 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using DEFAULT norm type for convergence test PC Object: 2 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural linear system matrix = precond matrix: Solving Electron Matrix Equation Matrix Object: 2 MPI processes type: mpiaij rows=11760, cols=11760 total: nonzeros=506586, allocated nonzeros=520821 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Crash info for parallel solve: [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [1]PETSC ERROR: to get more information on the crash. [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Signal received! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 6, Mon Feb 11 12:26:34 CST 2013 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: on a path-ompi named [1]PETSC ERROR: Libraries linked from [1]PETSC ERROR: Configure run at Thu Mar 21 14:19:42 2013 [1]PETSC ERROR: Configure options --PETSC_ARCH=path-ompi --PETSC_DIR= --CFLAGS="-fPIC -O -mp" --CXXFLAGS="-fPIC -O -mp" --FFLAGS="-fPIC -O -mp" --with-debugging=0 --with-dynamic-loadin=no --with-mpi=1 --with-mpi-dir= --with-superlu=1 --with-superlu-dir= --with-blas-lapack-lib="" --with-scalapack=1 --with-scalapack-dir= --with-superlu_dist=1 --with-superlu_dist-dir= --with-metis=1 --with-metis-dir= --with-parmetis=1 --with-parmetis-dir= --with-blacs-lib="" --with-blacs-include= --with-hypre=1 --download-hypre=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Jun 18 14:15:20 2013 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 18 Jun 2013 21:15:20 +0200 Subject: [petsc-users] SEGV on KSPSolve with mutliple processors In-Reply-To: References: Message-ID: You should recompile your code using a debug build of petsc so you get some meaningful info from the stack trace when the Segv occurs. Dave On Tuesday, 18 June 2013, Brendan C Lyons wrote: > Hi everyone, > > I've run into a strange problem in my Fortran 90 code where it runs fine > with 1 processor, but then throws a segmentation fault on KSPSolve() when I > try to run it in parallel. I'm using PETSc 3.3 with the SuperLU direct > solver for the sequential case and SuperLU_dist for the parallel case. > I've called KSPView before and after KSPSolve. I'll put the KSPView > output for the sequential and parallel cases and the crash info for the > parallel case below (with some details of my system redacted). Any help > would be appreciated. If you need any other information, I'm happy to > provide it. > > Thank you, > > ~Brendan > ------------------------------ > > KSPView() before sequential solve: > > KSP Object: 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: 1 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=11760, cols=11760 > total: nonzeros=506586, allocated nonzeros=509061 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > KSPView() after sequential solve: > > KSP Object: 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: 1 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 0, needed 0 > Factored matrix follows: > Matrix Object: 1 MPI processes > type: seqaij > rows=11760, cols=11760 > package used to perform factorization: superlu > total: nonzeros=0, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls =0 > SuperLU run parameters: > Equil: NO > ColPerm: 3 > IterRefine: 0 > SymmetricMode: NO > DiagPivotThresh: 1 > PivotGrowth: NO > ConditionNumber: NO > RowPerm: 0 > ReplaceTinyPivot: NO > PrintStat: NO > lwork: 0 > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=11760, cols=11760 > total: nonzeros=506586, allocated nonzeros=509061 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > > KSPView() before parallel solve: > > KSP Object: 2 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: 2 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > linear system matrix = precond matrix: > Solving Electron Matrix Equation > Matrix Object: 2 MPI processes > type: mpiaij > rows=11760, cols=11760 > total: nonzeros=506586, allocated nonzeros=520821 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > Crash info for parallel solve: > > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [1]PETSC ERROR: to get more information on the crash. > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Signal received! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 6, Mon Feb 11 12:26:34 > CST 2013 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: on a path-ompi named > [1]PETSC ERROR: Libraries linked from > [1]PETSC ERROR: Configure run at Thu Mar 21 14:19:42 2013 > [1]PETSC ERROR: Configure options --PETSC_ARCH=path-ompi > --PETSC_DIR= --CFLAGS="-fPIC -O -mp" --CXXFLAGS="-fPIC -O -mp" > --FFLAGS="-fPIC -O -mp" --with-debugging=0 --with-dynamic-loadin=no > --with-mpi=1 --with-mpi-dir= --with-superlu=1 > --with-superlu-dir= --with-blas-lapack-lib="" > --with-scalapack=1 --with-scalapack-dir= --with-superlu_dist=1 > --with-superlu_dist-dir= --with-metis=1 > --with-metis-dir= --with-parmetis=1 > --with-parmetis-dir= --with-blacs-lib="" > --with-blacs-include= --with-hypre=1 --download-hypre=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 18 15:52:30 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 18 Jun 2013 15:52:30 -0500 Subject: [petsc-users] SEGV on KSPSolve with mutliple processors In-Reply-To: References: Message-ID: If possible you would also benefit from running the debug version under valgrind http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind it is possible memory corruption has taken place before the point where the code crashes, valgrind will help identify any memory corruption as soon as it takes place. Barry On Jun 18, 2013, at 2:15 PM, Dave May wrote: > You should recompile your code using a debug build of petsc so you get some meaningful info from the stack trace when the Segv occurs. > > Dave > > > On Tuesday, 18 June 2013, Brendan C Lyons wrote: > Hi everyone, > > I've run into a strange problem in my Fortran 90 code where it runs fine with 1 processor, but then throws a segmentation fault on KSPSolve() when I try to run it in parallel. I'm using PETSc 3.3 with the SuperLU direct solver for the sequential case and SuperLU_dist for the parallel case. I've called KSPView before and after KSPSolve. I'll put the KSPView output for the sequential and parallel cases and the crash info for the parallel case below (with some details of my system redacted). Any help would be appreciated. If you need any other information, I'm happy to provide it. > > Thank you, > > ~Brendan > ------------------------------ > > KSPView() before sequential solve: > > KSP Object: 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: 1 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=11760, cols=11760 > total: nonzeros=506586, allocated nonzeros=509061 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > KSPView() after sequential solve: > > KSP Object: 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: 1 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 0, needed 0 > Factored matrix follows: > Matrix Object: 1 MPI processes > type: seqaij > rows=11760, cols=11760 > package used to perform factorization: superlu > total: nonzeros=0, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls =0 > SuperLU run parameters: > Equil: NO > ColPerm: 3 > IterRefine: 0 > SymmetricMode: NO > DiagPivotThresh: 1 > PivotGrowth: NO > ConditionNumber: NO > RowPerm: 0 > ReplaceTinyPivot: NO > PrintStat: NO > lwork: 0 > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=11760, cols=11760 > total: nonzeros=506586, allocated nonzeros=509061 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > > KSPView() before parallel solve: > > KSP Object: 2 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using DEFAULT norm type for convergence test > PC Object: 2 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > linear system matrix = precond matrix: > Solving Electron Matrix Equation > Matrix Object: 2 MPI processes > type: mpiaij > rows=11760, cols=11760 > total: nonzeros=506586, allocated nonzeros=520821 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > Crash info for parallel solve: > > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [1]PETSC ERROR: to get more information on the crash. > [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > [1]PETSC ERROR: Signal received! > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 6, Mon Feb 11 12:26:34 CST 2013 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: on a path-ompi named > [1]PETSC ERROR: Libraries linked from > [1]PETSC ERROR: Configure run at Thu Mar 21 14:19:42 2013 > [1]PETSC ERROR: Configure options --PETSC_ARCH=path-ompi --PETSC_DIR= --CFLAGS="-fPIC -O -mp" --CXXFLAGS="-fPIC -O -mp" --FFLAGS="-fPIC -O -mp" --with-debugging=0 --with-dynamic-loadin=no --with-mpi=1 --with-mpi-dir= --with-superlu=1 --with-superlu-dir= --with-blas-lapack-lib="" --with-scalapack=1 --with-scalapack-dir= --with-superlu_dist=1 --with-superlu_dist-dir= --with-metis=1 --with-metis-dir= --with-parmetis=1 --with-parmetis-dir= --with-blacs-lib="" --with-blacs-include= --with-hypre=1 --download-hypre=1 > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > > From lyh03259.aps at gmail.com Tue Jun 18 22:46:46 2013 From: lyh03259.aps at gmail.com (Yonghui) Date: Tue, 18 Jun 2013 22:46:46 -0500 Subject: [petsc-users] PETSc finite difference performance question (calculating gradient square) Message-ID: <008001ce6c9f$a0ae9940$e20bcbc0$@gmail.com> Hi PETSC users and developers, I am new to PETSC and I just want to have a communication with users and developers for a question I am concern. All I am concern is the performance of calculating Nabla_square in a parallel program with PETSC implemented. For example, say one have a 3 dimension function: f(x,y,z). We want nabla^2 f. i.e. (d^2/dx^2+ d^2/dy^2+ d^2/dz^2)f(x,y,z). (Make problem simpler assuming d^2/dxdy*f=0.) But on a computer, what we can do is usually finite difference, Thus in order to calculate the nabla^2*f on point x, y, z, we need all its nearest neighbors. That is to say, at leaste we need x+1, x-1, y+1, y-1, z+1 and z -1 (assuming 2nd order central). In the RAM, all number are stored in a ONE dimensional array. So the number is stored like this: ..f(x-1, y-1,z-1), f(x, y-1,z-1), f(x+1, y-1,z-1),.. f(x-1, y,z-1), f(x, y,z-1), f(x+1, y,z-1),.. ,f(x-1, y+1,z-1), f(x, y+1,z-1), f(x+1, y+1,z-1),.. So in order to work it out, the first thing is to pick out the numbers needed for the calculation. So each time one need to load some numbers in to the RAM, and pick out 1 or 2 or 3 out and discard the rest of them.. Until all the numbers one need are prepared for the calculation. It is clear that some there is a huge wasting here: the number that needed for the point cannot be loaded for immediately. Thus, the bottle neck is not the computing time but the loading time. Q: So how does PETSc library handle such kind of problem? Could you please explain it to me, if you understand how does it happened? I'll appreciate any explanation. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jefonseca at gmail.com Wed Jun 19 00:21:08 2013 From: jefonseca at gmail.com (Jim Fonseca) Date: Wed, 19 Jun 2013 01:21:08 -0400 Subject: [petsc-users] MatMatMult size error In-Reply-To: References: Message-ID: Hi Hong, It's been a while but we have found that moving to PETSc3.4 solved this problem with MatMatMult() in PETSc3.3p5. Thanks, Jim On Tue, Jan 15, 2013 at 4:43 PM, Hong Zhang wrote: > Jim : > Can you switch to petsc-dev? MatMatMult() > has been updated significantly in petsc-dev > If you still see problem in petsc-dev, send us a short code that > produce the error. We'll check it. > > Hong > > > Hi, > > We are in the process of upgrading from Petsc 3.2 to 3.3p5. > > > > We are creating matrices A and B in this way. > > petsc_matrix = new Mat; > > ierr = MatCreateDense(comm, m, num_cols ,num_rows,num_cols,data,A); > > > > Elsewhere, we have this. It gets called a few times, and on the 4th time, > > the size of matrix is C is wrong. Please see the output below. What > could be > > the problem? > > C = new Mat; > > double fill = PETSC_DEFAULT; > > MatMatMult(A,B,MAT_INITIAL_MATRIX, fill, C); > > { > > int m,n; > > MatGetOwnershipRange(A, &m, &n); > > cerr << "A.m = " << m << "\n"; > > cerr << "A.n = " << n << "\n"; > > MatGetSize(A,&m,&n); > > cerr << "A global rows = " << m << "\n"; > > cerr << "A global cols = " << n << "\n"; > > > > MatGetOwnershipRange(B, &m, &n); > > cerr << "B.m = " << m << "\n"; > > cerr << "B.n = " << n << "\n"; > > MatGetSize(B,&m,&n); > > cerr << "B global rows = " << m << "\n"; > > cerr << "B global cols = " << n << "\n"; > > > > MatGetOwnershipRange(*C, &m, &n); > > cerr << "C.m = " << m << "\n"; > > cerr << "C.n = " << n << "\n"; > > > > MatGetSize(*C,&m,&n); > > cerr << "C global rows = " << m << "\n"; > > cerr << "C global cols = " << n << "\n"; > > > > } > > > > A.m = 0 > > A.n = 59 > > A global rows = 59 > > A global cols = 320 > > B.m = 0 > > B.n = 320 > > B global rows = 320 > > B global cols = 320 > > C.m = 0 > > C.n = 59 > > C global rows = 59 > > C global cols = 320 > > A.m = 0 > > A.n = 59 > > A global rows = 59 > > A global cols = 320 > > B.m = 0 > > B.n = 320 > > B global rows = 320 > > B global cols = 59 > > C.m = 10922 > > C.n = -1389327096 > > C global rows = -1389327112 > > C global cols = -1389327112 > > > > > > Thanks, > > Jim > > -- > > Jim Fonseca, PhD > > Research Scientist > > Network for Computational Nanotechnology > > Purdue University > > 765-496-6495 > > www.jimfonseca.com > > > > > -- Jim Fonseca, PhD Research Scientist Network for Computational Nanotechnology Purdue University 765-496-6495 www.jimfonseca.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Jun 19 00:58:19 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 19 Jun 2013 07:58:19 +0200 Subject: [petsc-users] PETSc finite difference performance question (calculating gradient square) In-Reply-To: <008001ce6c9f$a0ae9940$e20bcbc0$@gmail.com> References: <008001ce6c9f$a0ae9940$e20bcbc0$@gmail.com> Message-ID: <877ghqiqv8.fsf@mcs.anl.gov> Yonghui writes: > Hi PETSC users and developers, > > > > I am new to PETSC and I just want to have a communication with users and > developers for a question I am concern. All I am concern is the performance > of calculating Nabla_square in a parallel program with PETSC implemented. > > > > For example, say one have a 3 dimension function: f(x,y,z). We want nabla^2 > f. i.e. (d^2/dx^2+ d^2/dy^2+ d^2/dz^2)f(x,y,z). (Make problem simpler > assuming d^2/dxdy*f=0.) > > But on a computer, what we can do is usually finite difference, Thus in > order to calculate the nabla^2*f on point x, y, z, we need all its nearest > neighbors. > > That is to say, at leaste we need x+1, x-1, y+1, y-1, z+1 and z -1 (assuming > 2nd order central). > > In the RAM, all number are stored in a ONE dimensional array. So the number > is stored like this: ..f(x-1, y-1,z-1), f(x, y-1,z-1), f(x+1, y-1,z-1),.. > f(x-1, y,z-1), f(x, y,z-1), f(x+1, y,z-1),.. ,f(x-1, y+1,z-1), f(x, > y+1,z-1), f(x+1, y+1,z-1),.. > > So in order to work it out, the first thing is to pick out the numbers > needed for the calculation. So each time one need to load some numbers in to > the RAM, and pick out 1 or 2 or 3 out and discard the rest of them.. Until > all the numbers one need are prepared for the calculation. > > It is clear that some there is a huge wasting here: the number that needed > for the point cannot be loaded for immediately. Thus, the bottle neck is not > the computing time but the loading time. I'm not sure if you argument here is about latency or bandwidth, but it's true that for sparse operations, memory access is almost always the bottleneck. > Q: So how does PETSc library handle such kind of problem? Could you please > explain it to me, if you understand how does it happened? Some kernels have software prefetch to reduce the unpredicted cache misses and so that upon eviction from L1, only the useful stuff remains in higher level caches. Bandwidth is the main limitation in sparse computation and cannot be avoided by clever tricks unless the data structure is changed entirely. See the second figure here for an example of how dramatic such transformations can be: https://github.com/jedbrown/dohp/wiki/Dohp But if you go high-order and matrix-free like in that figure, you still have to deal with preconditioning somewhere. Unless you have the expertise and motivation to write custom matrix-free preconditioners (like some multigrid methods), then preconditioners will be constructed From assembled matrices, and the bandwidth issues will apply. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From s.kramer at imperial.ac.uk Wed Jun 19 07:22:31 2013 From: s.kramer at imperial.ac.uk (Stephan Kramer) Date: Wed, 19 Jun 2013 13:22:31 +0100 Subject: [petsc-users] MatZeroRowsColumns for mpibaij Message-ID: <51C1A287.7070303@imperial.ac.uk> Dear all, We have found the MatZeroRowsColumns() routine to be very useful for lifting boundary conditions. Unfortunately we also found that it isn't implemented for matrix type mpibaij and we're quite keen on keeping the baij structure as it seems to give us significant savings for DG problems (in particular the blocked sor). I had a quick look at the implementation of MatZeroRowsColums_MPIAIJ but couldn't really estimate how hard it would be to do something similar for mpibaij (an implementation for seqbaij exists already). Is this something that's likely to be looked at in the future? Stephan From hzhang at mcs.anl.gov Wed Jun 19 09:38:27 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 19 Jun 2013 09:38:27 -0500 Subject: [petsc-users] MatMatMult size error In-Reply-To: References: Message-ID: Glad to know the problem is fixed in the release. Report to us whenever you encounter problem in using PETSc. Hong On Wed, Jun 19, 2013 at 12:21 AM, Jim Fonseca wrote: > Hi Hong, > It's been a while but we have found that moving to PETSc3.4 solved this > problem with MatMatMult() in PETSc3.3p5. > Thanks, > Jim > > > On Tue, Jan 15, 2013 at 4:43 PM, Hong Zhang wrote: > >> Jim : >> Can you switch to petsc-dev? MatMatMult() >> has been updated significantly in petsc-dev >> If you still see problem in petsc-dev, send us a short code that >> produce the error. We'll check it. >> >> Hong >> >> > Hi, >> > We are in the process of upgrading from Petsc 3.2 to 3.3p5. >> > >> > We are creating matrices A and B in this way. >> > petsc_matrix = new Mat; >> > ierr = MatCreateDense(comm, m, num_cols >> ,num_rows,num_cols,data,A); >> > >> > Elsewhere, we have this. It gets called a few times, and on the 4th >> time, >> > the size of matrix is C is wrong. Please see the output below. What >> could be >> > the problem? >> > C = new Mat; >> > double fill = PETSC_DEFAULT; >> > MatMatMult(A,B,MAT_INITIAL_MATRIX, fill, C); >> > { >> > int m,n; >> > MatGetOwnershipRange(A, &m, &n); >> > cerr << "A.m = " << m << "\n"; >> > cerr << "A.n = " << n << "\n"; >> > MatGetSize(A,&m,&n); >> > cerr << "A global rows = " << m << "\n"; >> > cerr << "A global cols = " << n << "\n"; >> > >> > MatGetOwnershipRange(B, &m, &n); >> > cerr << "B.m = " << m << "\n"; >> > cerr << "B.n = " << n << "\n"; >> > MatGetSize(B,&m,&n); >> > cerr << "B global rows = " << m << "\n"; >> > cerr << "B global cols = " << n << "\n"; >> > >> > MatGetOwnershipRange(*C, &m, &n); >> > cerr << "C.m = " << m << "\n"; >> > cerr << "C.n = " << n << "\n"; >> > >> > MatGetSize(*C,&m,&n); >> > cerr << "C global rows = " << m << "\n"; >> > cerr << "C global cols = " << n << "\n"; >> > >> > } >> > >> > A.m = 0 >> > A.n = 59 >> > A global rows = 59 >> > A global cols = 320 >> > B.m = 0 >> > B.n = 320 >> > B global rows = 320 >> > B global cols = 320 >> > C.m = 0 >> > C.n = 59 >> > C global rows = 59 >> > C global cols = 320 >> > A.m = 0 >> > A.n = 59 >> > A global rows = 59 >> > A global cols = 320 >> > B.m = 0 >> > B.n = 320 >> > B global rows = 320 >> > B global cols = 59 >> > C.m = 10922 >> > C.n = -1389327096 >> > C global rows = -1389327112 >> > C global cols = -1389327112 >> > >> > >> > Thanks, >> > Jim >> > -- >> > Jim Fonseca, PhD >> > Research Scientist >> > Network for Computational Nanotechnology >> > Purdue University >> > 765-496-6495 >> > www.jimfonseca.com >> > >> > >> > > > > -- > Jim Fonseca, PhD > Research Scientist > Network for Computational Nanotechnology > Purdue University > 765-496-6495 > www.jimfonseca.com > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Wed Jun 19 11:24:03 2013 From: john.mousel at gmail.com (John Mousel) Date: Wed, 19 Jun 2013 11:24:03 -0500 Subject: [petsc-users] Destroy ghosted vector that was used to duplicate Message-ID: Is it legal to destroy a ghosted vector if it has been used in VecDuplicate? Something like: VecCreateGhost(...,vec1) VecDuplicate(vec1,vec2) VecDuplicate(vec1,vec3) VecDestroy(vec1) VecAXPY(vec2,1.0,vec3) Is this legal or are the scatters left hanging in the ether? John -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 19 14:42:48 2013 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 19 Jun 2013 21:42:48 +0200 Subject: [petsc-users] Destroy ghosted vector that was used to duplicate In-Reply-To: References: Message-ID: On Wed, Jun 19, 2013 at 6:24 PM, John Mousel wrote: > Is it legal to destroy a ghosted vector if it has been used in > VecDuplicate? > This is fine. MAtt > Something like: > > VecCreateGhost(...,vec1) > VecDuplicate(vec1,vec2) > VecDuplicate(vec1,vec3) > VecDestroy(vec1) > VecAXPY(vec2,1.0,vec3) > > Is this legal or are the scatters left hanging in the ether? > > > John > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Wed Jun 19 14:45:59 2013 From: john.mousel at gmail.com (John Mousel) Date: Wed, 19 Jun 2013 14:45:59 -0500 Subject: [petsc-users] Destroy ghosted vector that was used to duplicate In-Reply-To: References: Message-ID: Looking at the message queue graph in Totalview, it seems that I'm running up the number of persistent communications when I try to resize a vector by duplicating, copying, destroying, and calling VecCreateGhost. On Wed, Jun 19, 2013 at 2:42 PM, Matthew Knepley wrote: > On Wed, Jun 19, 2013 at 6:24 PM, John Mousel wrote: > >> Is it legal to destroy a ghosted vector if it has been used in >> VecDuplicate? >> > > This is fine. > > MAtt > > >> Something like: >> >> VecCreateGhost(...,vec1) >> VecDuplicate(vec1,vec2) >> VecDuplicate(vec1,vec3) >> VecDestroy(vec1) >> VecAXPY(vec2,1.0,vec3) >> >> Is this legal or are the scatters left hanging in the ether? >> >> >> John >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 19 17:59:44 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 19 Jun 2013 17:59:44 -0500 Subject: [petsc-users] Destroy ghosted vector that was used to duplicate In-Reply-To: References: Message-ID: <7E6FD3B5-16F3-4215-801F-0D556BA03802@mcs.anl.gov> John, When a VecCreateGhost() is duplicated the new vector shares the VecScatter with the original vector. In your example below vec2 and vec3 share the VecScatter. Hence the VecScatter has a reference count of 3. When you call VecDestroy(vec1) the reference count for that scatter falls to 2. Once vec2 and vec3 are destroyed the reference count for the VecScatter falls to zero and it is destroyed. Hence no memory (or MPI persistent communication resources) should be lost. Note that even if you do not plan to use the "local vector" part of vec2 and vec3 that VecScatter will still be around until you destroy vec2 and vec3 and any other Vecs obtained by VecDuplicating any of these. If you want to eliminate the VecScatter then don't use a VecDuplicate() on the ghosted vector instead create a non-ghosted vector. This is the way it is suppose to be anyways. Are you observing something different that could be reproduced in a somewhat reasonably small example? Barry On Jun 19, 2013, at 2:45 PM, John Mousel wrote: > Looking at the message queue graph in Totalview, it seems that I'm running up the number of persistent communications when I try to resize a vector by duplicating, copying, destroying, and calling VecCreateGhost. > > > On Wed, Jun 19, 2013 at 2:42 PM, Matthew Knepley wrote: > On Wed, Jun 19, 2013 at 6:24 PM, John Mousel wrote: > Is it legal to destroy a ghosted vector if it has been used in VecDuplicate? > > This is fine. > > MAtt > > Something like: > > VecCreateGhost(...,vec1) > VecDuplicate(vec1,vec2) > VecDuplicate(vec1,vec3) > VecDestroy(vec1) > VecAXPY(vec2,1.0,vec3) > > Is this legal or are the scatters left hanging in the ether? > > > John > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > From john.mousel at gmail.com Wed Jun 19 20:05:43 2013 From: john.mousel at gmail.com (John Mousel) Date: Wed, 19 Jun 2013 20:05:43 -0500 Subject: [petsc-users] Destroy ghosted vector that was used to duplicate In-Reply-To: <7E6FD3B5-16F3-4215-801F-0D556BA03802@mcs.anl.gov> References: <7E6FD3B5-16F3-4215-801F-0D556BA03802@mcs.anl.gov> Message-ID: I just figured out that I had a well hidden duplication of a duplication of the vector I was trying to resize. Hence the run-up in persistent communications. This kind of dependency is somewhat difficult to track down. Is there any way to check the vectors dependent on a particular scatter? On Wed, Jun 19, 2013 at 5:59 PM, Barry Smith wrote: > > John, > > When a VecCreateGhost() is duplicated the new vector shares the > VecScatter with the original vector. In your example below vec2 and vec3 > share the VecScatter. Hence the VecScatter has a reference count of 3. When > you call VecDestroy(vec1) the reference count for that scatter falls to 2. > Once vec2 and vec3 are destroyed the reference count for the VecScatter > falls to zero and it is destroyed. Hence no memory (or MPI persistent > communication resources) should be lost. > > Note that even if you do not plan to use the "local vector" part of > vec2 and vec3 that VecScatter will still be around until you destroy vec2 > and vec3 and any other Vecs obtained by VecDuplicating any of these. If you > want to eliminate the VecScatter then don't use a VecDuplicate() on the > ghosted vector instead create a non-ghosted vector. > > > This is the way it is suppose to be anyways. Are you observing > something different that could be reproduced in a somewhat reasonably small > example? > > Barry > > On Jun 19, 2013, at 2:45 PM, John Mousel wrote: > > > Looking at the message queue graph in Totalview, it seems that I'm > running up the number of persistent communications when I try to resize a > vector by duplicating, copying, destroying, and calling VecCreateGhost. > > > > > > On Wed, Jun 19, 2013 at 2:42 PM, Matthew Knepley > wrote: > > On Wed, Jun 19, 2013 at 6:24 PM, John Mousel > wrote: > > Is it legal to destroy a ghosted vector if it has been used in > VecDuplicate? > > > > This is fine. > > > > MAtt > > > > Something like: > > > > VecCreateGhost(...,vec1) > > VecDuplicate(vec1,vec2) > > VecDuplicate(vec1,vec3) > > VecDestroy(vec1) > > VecAXPY(vec2,1.0,vec3) > > > > Is this legal or are the scatters left hanging in the ether? > > > > > > John > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 19 22:13:54 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 19 Jun 2013 22:13:54 -0500 Subject: [petsc-users] Destroy ghosted vector that was used to duplicate In-Reply-To: References: <7E6FD3B5-16F3-4215-801F-0D556BA03802@mcs.anl.gov> Message-ID: <8DE71ED8-D434-492C-A9C3-F1855C8FA2B1@mcs.anl.gov> On Jun 19, 2013, at 8:05 PM, John Mousel wrote: > I just figured out that I had a well hidden duplication of a duplication of the vector I was trying to resize. Hence the run-up in persistent communications. This kind of dependency is somewhat difficult to track down. Is there any way to check the vectors dependent on a particular scatter? Unfortunately not. It would not be terribly difficult to graphically display all the objects and their dependencies but we don't have that currently. Barry > > > On Wed, Jun 19, 2013 at 5:59 PM, Barry Smith wrote: > > John, > > When a VecCreateGhost() is duplicated the new vector shares the VecScatter with the original vector. In your example below vec2 and vec3 share the VecScatter. Hence the VecScatter has a reference count of 3. When you call VecDestroy(vec1) the reference count for that scatter falls to 2. Once vec2 and vec3 are destroyed the reference count for the VecScatter falls to zero and it is destroyed. Hence no memory (or MPI persistent communication resources) should be lost. > > Note that even if you do not plan to use the "local vector" part of vec2 and vec3 that VecScatter will still be around until you destroy vec2 and vec3 and any other Vecs obtained by VecDuplicating any of these. If you want to eliminate the VecScatter then don't use a VecDuplicate() on the ghosted vector instead create a non-ghosted vector. > > > This is the way it is suppose to be anyways. Are you observing something different that could be reproduced in a somewhat reasonably small example? > > Barry > > On Jun 19, 2013, at 2:45 PM, John Mousel wrote: > > > Looking at the message queue graph in Totalview, it seems that I'm running up the number of persistent communications when I try to resize a vector by duplicating, copying, destroying, and calling VecCreateGhost. > > > > > > On Wed, Jun 19, 2013 at 2:42 PM, Matthew Knepley wrote: > > On Wed, Jun 19, 2013 at 6:24 PM, John Mousel wrote: > > Is it legal to destroy a ghosted vector if it has been used in VecDuplicate? > > > > This is fine. > > > > MAtt > > > > Something like: > > > > VecCreateGhost(...,vec1) > > VecDuplicate(vec1,vec2) > > VecDuplicate(vec1,vec3) > > VecDestroy(vec1) > > VecAXPY(vec2,1.0,vec3) > > > > Is this legal or are the scatters left hanging in the ether? > > > > > > John > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > From john.mousel at gmail.com Wed Jun 19 22:17:40 2013 From: john.mousel at gmail.com (John Mousel) Date: Wed, 19 Jun 2013 22:17:40 -0500 Subject: [petsc-users] Destroy ghosted vector that was used to duplicate In-Reply-To: <8DE71ED8-D434-492C-A9C3-F1855C8FA2B1@mcs.anl.gov> References: <7E6FD3B5-16F3-4215-801F-0D556BA03802@mcs.anl.gov> <8DE71ED8-D434-492C-A9C3-F1855C8FA2B1@mcs.anl.gov> Message-ID: That's what I suspected. Thanks for the help. I have my current issue fixed. John On Wed, Jun 19, 2013 at 10:13 PM, Barry Smith wrote: > > On Jun 19, 2013, at 8:05 PM, John Mousel wrote: > > > I just figured out that I had a well hidden duplication of a duplication > of the vector I was trying to resize. Hence the run-up in persistent > communications. This kind of dependency is somewhat difficult to track > down. Is there any way to check the vectors dependent on a particular > scatter? > > Unfortunately not. It would not be terribly difficult to graphically > display all the objects and their dependencies but we don't have that > currently. > > Barry > > > > > > > On Wed, Jun 19, 2013 at 5:59 PM, Barry Smith wrote: > > > > John, > > > > When a VecCreateGhost() is duplicated the new vector shares the > VecScatter with the original vector. In your example below vec2 and vec3 > share the VecScatter. Hence the VecScatter has a reference count of 3. When > you call VecDestroy(vec1) the reference count for that scatter falls to 2. > Once vec2 and vec3 are destroyed the reference count for the VecScatter > falls to zero and it is destroyed. Hence no memory (or MPI persistent > communication resources) should be lost. > > > > Note that even if you do not plan to use the "local vector" part of > vec2 and vec3 that VecScatter will still be around until you destroy vec2 > and vec3 and any other Vecs obtained by VecDuplicating any of these. If you > want to eliminate the VecScatter then don't use a VecDuplicate() on the > ghosted vector instead create a non-ghosted vector. > > > > > > This is the way it is suppose to be anyways. Are you observing > something different that could be reproduced in a somewhat reasonably small > example? > > > > Barry > > > > On Jun 19, 2013, at 2:45 PM, John Mousel wrote: > > > > > Looking at the message queue graph in Totalview, it seems that I'm > running up the number of persistent communications when I try to resize a > vector by duplicating, copying, destroying, and calling VecCreateGhost. > > > > > > > > > On Wed, Jun 19, 2013 at 2:42 PM, Matthew Knepley > wrote: > > > On Wed, Jun 19, 2013 at 6:24 PM, John Mousel > wrote: > > > Is it legal to destroy a ghosted vector if it has been used in > VecDuplicate? > > > > > > This is fine. > > > > > > MAtt > > > > > > Something like: > > > > > > VecCreateGhost(...,vec1) > > > VecDuplicate(vec1,vec2) > > > VecDuplicate(vec1,vec3) > > > VecDestroy(vec1) > > > VecAXPY(vec2,1.0,vec3) > > > > > > Is this legal or are the scatters left hanging in the ether? > > > > > > > > > John > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > -- Norbert Wiener > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Jun 19 17:09:32 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 20 Jun 2013 00:09:32 +0200 Subject: [petsc-users] Destroy ghosted vector that was used to duplicate In-Reply-To: References: Message-ID: <87sj0d7nxf.fsf@mcs.anl.gov> John Mousel writes: > Looking at the message queue graph in Totalview, it seems that I'm running > up the number of persistent communications when I try to resize a vector by > duplicating, copying, destroying, and calling VecCreateGhost. I don't know how you're measuring that. PETSc does not create new communicators per object. We create one PETSc comm per user comm, and all the objects share that. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From john.mousel at gmail.com Wed Jun 19 23:03:57 2013 From: john.mousel at gmail.com (John Mousel) Date: Wed, 19 Jun 2013 23:03:57 -0500 Subject: [petsc-users] Destroy ghosted vector that was used to duplicate In-Reply-To: <87sj0d7nxf.fsf@mcs.anl.gov> References: <87sj0d7nxf.fsf@mcs.anl.gov> Message-ID: I was looking at the number of persistent sends and receives associated with scatters for ghosted vectors using the Totalview Message Queue Graph. These were growing because I was not destroying a duplication of a duplication of a vector. I was not observing an incorrect number of communicators, which was just a single duplication of MPI_COMM_WORLD. John On Wed, Jun 19, 2013 at 5:09 PM, Jed Brown wrote: > John Mousel writes: > > > Looking at the message queue graph in Totalview, it seems that I'm > running > > up the number of persistent communications when I try to resize a vector > by > > duplicating, copying, destroying, and calling VecCreateGhost. > > I don't know how you're measuring that. PETSc does not create new > communicators per object. We create one PETSc comm per user comm, and > all the objects share that. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Jun 20 09:47:34 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 20 Jun 2013 09:47:34 -0500 Subject: [petsc-users] The multiplication of the transpose of a dense matrix(A^T) and a large sparse matrix(X1) In-Reply-To: <1497296706.9782.1370387738058.JavaMail.root@mailhub028.itcs.purdue.edu> References: <87r4ghlo64.fsf@mcs.anl.gov> <1497296706.9782.1370387738058.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: Joon, We have implemented MatTransposeMatMult() for aij and dense matrices in the master branch of petsc-dev https://bitbucket.org/petsc/petsc/commits/cf22a14bc6cad51fdcbb947b8ac26f3ebf0d4d04?at=master based on Jed's suggestion. You may give it a try and let us know if there is a problem. Hong On Tue, Jun 4, 2013 at 6:15 PM, Joon hee Choi wrote: > I am trying to compress the matrix. I think it is the best way, because it > may be not storable by MPI aij. > If you get better solution in your lab meeting, then please let me know. :) > Anyhow, thank you very much. > > Joon > > ----- Original Message ----- > From: "Jed Brown" > To: "Joon hee Choi" > Cc: petsc-users at mcs.anl.gov > Sent: Tuesday, June 4, 2013 2:35:15 PM > Subject: Re: [petsc-users] The multiplication of the transpose of a dense > matrix(A^T) and a large sparse matrix(X1) > > Joon hee Choi writes: > > > Thank you for your fast reply. > > I got the out-of-memory error from MatTranspose(X1, MAT_INITIAL_MATRIX, > &tempX1). > > Also, it may take so much time to create X1 as X1^T because nnz for X1 > cannot have 1200Tril elements because of memory. > > Yes, that is also the dimension of the result matrix, which would be > dense with the normal rules about products of sparse and dense matrices. > > Instead, you should compress the column space, perhaps by just removing > all the columns that have no nonzeros. > > > Is there a fast way to create X1^T? > > > > Thank you, > > Joon > > > > > > ----- Original Message ----- > > From: "Jed Brown" > > To: "Joon hee Choi" , petsc-users at mcs.anl.gov > > Sent: Tuesday, June 4, 2013 7:55:31 AM > > Subject: Re: [petsc-users] The multiplication of the transpose of a > dense matrix(A^T) and a large sparse matrix(X1) > > > > Joon hee Choi writes: > > > >> Hello, > >> > >> I am trying to multiply the transpose of a dense matrix(A) and a large > sparse matrix(X1). That is, A^T x X1. > >> > >> X1 is a 26Mil x 1200Tril, 144Mil non-zeros sparse matrix and A is a > 26Mil x 10 dense matrix. > >> > >> I know that sparse x dense is faster than dense x sparse when using > MatMatMult. Thus, I tried to implement the following code: > >> > >> ierr = MatTranspose(X1, MAT_INITIAL_MATRIX, &tempX1); > CHKERRQ(ierr); > >> ierr = MatMatMult(tempX1, A, MAT_INITIAL_MATRIX, 1.0, &MT); > >> ierr = MatDestroy(&tempX1); CHKERRQ(ierr); > >> ierr = MatTranspose(MT, MAT_INITIAL_MATRIX, &M); CHKERRQ(ierr); > >> ierr = MatDestroy(&MT); CHKERRQ(ierr); > >> > >> However, I got the "out-of-memory" error when implementing > >> MatTranspose(). > > > > Which MatTranspose? > > > >> I think this is because the number of columns of X1 is much larger > >> than that of rows of X1. If there is a fast way to calculate M = A^T > >> x X1, > > > > Hong, do you have time to implement MatTransposeMatMult_MPIAIJ_MPIDense? > > > > Can you create X1 as X1^T instead? > > > > If you want to keep storing X1 is you do now, you can either store it as > > ten vectors and use MatMultTranspose or you can pack it into one vector > and use > > > > MatCreateMAIJ(X1,10,&X1m); > > MatMultTranspose(X1m,Apacked,Bpacked); > > > > This is actually a better ordering for memory bandwidth. The MAIJ > > matrix does not need extra storage. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hsahasra at purdue.edu Thu Jun 20 15:24:14 2013 From: hsahasra at purdue.edu (Harshad Sahasrabudhe) Date: Thu, 20 Jun 2013 16:24:14 -0400 Subject: [petsc-users] Using MAGMA subroutines instead of LAPACK in SLEPc Message-ID: <51C364EE.1020402@purdue.edu> Hi, I'm trying to run the SLEPc Eigensolver on a machine with CPUs & GPUs. I found that a package called MAGMA implements LAPACK functions on heterogeneous systems (CPU + GPU). MAGMA has functions very similar to LAPACK so that we can go and replace a call to a LAPACK function with a call to a MAGMA function. Now my question is, where can I apply such a hack inside SLEPc/PETSc so that the Eigensolver in SLEPc actually uses MAGMA functions instead of LAPACK functions? I'm using the PETSc 3.3-p6 and SLEPc 3.3-p3 versions. Any help would be greatly appreciated! Thanks and Regards, Harshad From Franck.Houssen at cea.fr Thu Jun 20 15:24:01 2013 From: Franck.Houssen at cea.fr (HOUSSEN Franck) Date: Thu, 20 Jun 2013 20:24:01 +0000 Subject: [petsc-users] How to get bandwidth peak out of PETSc log ? Message-ID: Hello, I am new to PETSc. I have written a (MPI) PETSc code to solve an AX=B system : how to know the bandwidth peak for a given run ? The code does not scale as I would expect (doubling the number of MPI processes does not half the elapsed time) : I would like to understand if this behavior is related to a bad MPI parallelization (that I may be able to improve), or, to the fact that the bandwidth limit as been reached (and in this case, my understanding is that I can not do anything to improve neither the performance, nor the scaling). I would like to know what's going on and why ! Concerning the computer, I have tried to estimate the bandwidth peak with the "stream benchmark" : www.streambench.org/index.html. I get this : ~>./stream_c.exe ... Function Best Rate MB/s Avg time Min time Max time Copy: 11473.8 0.014110 0.013945 0.015064 Scale: 11421.2 0.014070 0.014009 0.014096 Add: 12974.4 0.018537 0.018498 0.018590 Triad: 12964.6 0.018683 0.018512 0.019277 As a conclusion, my understanding is that the bandwidth peak of my computer is about 12 200 MB / s (= average between 11 400 and 12 900) which is about 12200 / 1024 = 11.9 GB/s. Concerning PETSc, I tried to find (without success) a figure to compare to 11.9 GB/s. First, I tried to run "make streams" but it doesn't compile (I run petsc3.4.1. on Ubuntu 12.04). Then, I looked into the PETSc log (using -log_summary and -ksp_view) but I was not able to find out an estimate of the bandwidth out of it (I get information about time and flops but not about the amount of data transfered between MPI processes in MB). How can I get bandwidth peak out of PETSc log ? Is there a specific option for that or is this not possible ? I have attached the PETSc log (MPI run over 2 processes). Thanks, FH -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PETSc.log Type: text/x-log Size: 20458 bytes Desc: PETSc.log URL: From knepley at gmail.com Thu Jun 20 15:38:08 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 20 Jun 2013 22:38:08 +0200 Subject: [petsc-users] How to get bandwidth peak out of PETSc log ? In-Reply-To: References: Message-ID: On Thu, Jun 20, 2013 at 10:24 PM, HOUSSEN Franck wrote: > Hello, > > I am new to PETSc. > > I have written a (MPI) PETSc code to solve an AX=B system : how to know > the bandwidth peak for a given run ? The code does not scale as I would > expect (doubling the number of MPI processes does not half the elapsed > time) : I would like to understand if this behavior is related to a bad MPI > parallelization (that I may be able to improve), or, to the fact that the > bandwidth limit as been reached (and in this case, my understanding is that > I can not do anything to improve neither the performance, nor the scaling). > I would like to know what's going on and why ! > > Concerning the computer, I have tried to estimate the bandwidth peak with > the "stream benchmark" : www.streambench.org/index.html. I get this : > ~>./stream_c.exe > ... > Function Best Rate MB/s Avg time Min time Max time > Copy: 11473.8 0.014110 0.013945 0.015064 > Scale: 11421.2 0.014070 0.014009 0.014096 > Add: 12974.4 0.018537 0.018498 0.018590 > Triad: 12964.6 0.018683 0.018512 0.019277 > As a conclusion, my understanding is that the bandwidth peak of my > computer is about 12 200 MB / s (= average between 11 400 and 12 900) which > is about 12200 / 1024 = 11.9 GB/s. > If this is your maximum bandwidth, it limits the amount of computing you can do since you need to wait for the operands. Thus, for VecAXPY, you need 2 doubles from memory and you write 1 double when you do 2 flops, meaning you do 2 flops/24 bytes = 1 flops/12 bytes The peak achievable flops for VecAXPY on your machine is roughly 12.9 GB/s * 1 flops/12 bytes = 1.1 GF/s In your log, you are getting 1.5 GF/s, so you are at the bandwidth peak. Another way to see this is that VecMAXPY gets much better performance. If the performance were limited by flops, this would not be the case. Matt Concerning PETSc, I tried to find (without success) a figure to compare to > 11.9 GB/s. First, I tried to run "make streams" but it doesn't compile (I > run petsc3.4.1. on Ubuntu 12.04). Then, I looked into the PETSc log (using > -log_summary and -ksp_view) but I was not able to find out an estimate of > the bandwidth out of it (I get information about time and flops but not > about the amount of data transfered between MPI processes in MB). How can I > get bandwidth peak out of PETSc log ? Is there a specific option for that > or is this not possible ? > > I have attached the PETSc log (MPI run over 2 processes). > > Thanks, > > FH > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 20 15:42:54 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 20 Jun 2013 22:42:54 +0200 Subject: [petsc-users] Using MAGMA subroutines instead of LAPACK in SLEPc In-Reply-To: <51C364EE.1020402@purdue.edu> References: <51C364EE.1020402@purdue.edu> Message-ID: On Thu, Jun 20, 2013 at 10:24 PM, Harshad Sahasrabudhe wrote: > Hi, > > I'm trying to run the SLEPc Eigensolver on a machine with CPUs & GPUs. I > found that a package called MAGMA implements LAPACK functions on > heterogeneous systems (CPU + GPU). MAGMA has functions very similar to > LAPACK so that we can go and replace a call to a LAPACK function with a > call to a MAGMA function. > > Now my question is, where can I apply such a hack inside SLEPc/PETSc so > that the Eigensolver in SLEPc actually uses MAGMA functions instead of > LAPACK functions? I'm using the PETSc 3.3-p6 and SLEPc 3.3-p3 versions. > > Any help would be greatly appreciated! > You need to do this in PETSc configure. Use --with-f-blas-lapack-lib=[]. We have never tried this, so I don't know if they obey the entire interface. Thanks > Thanks and Regards, > Harshad > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knteran at sandia.gov Thu Jun 20 15:49:34 2013 From: knteran at sandia.gov (Teranishi, Keita) Date: Thu, 20 Jun 2013 20:49:34 +0000 Subject: [petsc-users] [EXTERNAL] Re: Using MAGMA subroutines instead of LAPACK in SLEPc In-Reply-To: Message-ID: Hi, As far as I know, Magma's API us not exactly the same as the original LAPACK. You may need to hack either Magma's Fortran API or internal SLEPC/PETSc source. Thanks, --------------------------------------------- Keita Teranishi R&D Principal Staff Member Sandia National Laboratories Livermore, CA 94551 From: Matthew Knepley > Date: Thursday, June 20, 2013 1:42 PM To: Harshad Sahasrabudhe > Cc: "petsc-users at mcs.anl.gov" > Subject: [EXTERNAL] Re: [petsc-users] Using MAGMA subroutines instead of LAPACK in SLEPc On Thu, Jun 20, 2013 at 10:24 PM, Harshad Sahasrabudhe > wrote: Hi, I'm trying to run the SLEPc Eigensolver on a machine with CPUs & GPUs. I found that a package called MAGMA implements LAPACK functions on heterogeneous systems (CPU + GPU). MAGMA has functions very similar to LAPACK so that we can go and replace a call to a LAPACK function with a call to a MAGMA function. Now my question is, where can I apply such a hack inside SLEPc/PETSc so that the Eigensolver in SLEPc actually uses MAGMA functions instead of LAPACK functions? I'm using the PETSc 3.3-p6 and SLEPc 3.3-p3 versions. Any help would be greatly appreciated! You need to do this in PETSc configure. Use --with-f-blas-lapack-lib=[]. We have never tried this, so I don't know if they obey the entire interface. Thanks Thanks and Regards, Harshad -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hsahasra at purdue.edu Thu Jun 20 15:54:35 2013 From: hsahasra at purdue.edu (Harshad Sahasrabudhe) Date: Thu, 20 Jun 2013 16:54:35 -0400 Subject: [petsc-users] [EXTERNAL] Re: Using MAGMA subroutines instead of LAPACK in SLEPc In-Reply-To: References: Message-ID: <51C36C0B.3060500@purdue.edu> Hi, Yes, MAGMA's API is not exactly the same. There are a few additional arguments in the functions related to number of GPU etc. So it cannot be directly linked instead of LAPACK. Hacking SLEPc/PETSc code seems to be the easier way to go. But I can't find the exact place where the LAPACK functions are called. Thanks, Harshad On 06/20/2013 04:49 PM, Teranishi, Keita wrote: > Hi, > > As far as I know, Magma's API us not exactly the same as the original > LAPACK. You may need to hack either Magma's Fortran API or internal > SLEPC/PETSc source. > > Thanks, > --------------------------------------------- > Keita Teranishi > R&D Principal Staff Member > Sandia National Laboratories > Livermore, CA 94551 > > From: Matthew Knepley > > Date: Thursday, June 20, 2013 1:42 PM > To: Harshad Sahasrabudhe > > Cc: "petsc-users at mcs.anl.gov " > > > Subject: [EXTERNAL] Re: [petsc-users] Using MAGMA subroutines instead > of LAPACK in SLEPc > > On Thu, Jun 20, 2013 at 10:24 PM, Harshad Sahasrabudhe > > wrote: > > Hi, > > I'm trying to run the SLEPc Eigensolver on a machine with CPUs & > GPUs. I found that a package called MAGMA implements LAPACK > functions on heterogeneous systems (CPU + GPU). MAGMA has > functions very similar to LAPACK so that we can go and replace a > call to a LAPACK function with a call to a MAGMA function. > > Now my question is, where can I apply such a hack inside > SLEPc/PETSc so that the Eigensolver in SLEPc actually uses MAGMA > functions instead of LAPACK functions? I'm using the PETSc 3.3-p6 > and SLEPc 3.3-p3 versions. > > Any help would be greatly appreciated! > > > You need to do this in PETSc configure. Use > --with-f-blas-lapack-lib=[]. We have never tried this, so > I don't > know if they obey the entire interface. > > Thanks > > Thanks and Regards, > Harshad > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jun 20 15:59:59 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 20 Jun 2013 15:59:59 -0500 (CDT) Subject: [petsc-users] [EXTERNAL] Re: Using MAGMA subroutines instead of LAPACK in SLEPc In-Reply-To: <51C36C0B.3060500@purdue.edu> References: <51C36C0B.3060500@purdue.edu> Message-ID: check include/petscblaslapack*.h You might have to add a new one [or modify one of the relavent one for your compiler] - similar to stuff in petscblaslapack_stdcall.h Satish On Thu, 20 Jun 2013, Harshad Sahasrabudhe wrote: > Hi, > > Yes, MAGMA's API is not exactly the same. There are a few additional arguments > in the functions related to number of GPU etc. So it cannot be directly linked > instead of LAPACK. > > Hacking SLEPc/PETSc code seems to be the easier way to go. But I can't find > the exact place where the LAPACK functions are called. > > Thanks, > Harshad > > On 06/20/2013 04:49 PM, Teranishi, Keita wrote: > > Hi, > > > > As far as I know, Magma's API us not exactly the same as the original > > LAPACK. You may need to hack either Magma's Fortran API or internal > > SLEPC/PETSc source. > > > > Thanks, > > --------------------------------------------- > > Keita Teranishi > > R&D Principal Staff Member > > Sandia National Laboratories > > Livermore, CA 94551 > > > > From: Matthew Knepley > > > Date: Thursday, June 20, 2013 1:42 PM > > To: Harshad Sahasrabudhe > > > Cc: "petsc-users at mcs.anl.gov " > > > > > Subject: [EXTERNAL] Re: [petsc-users] Using MAGMA subroutines instead of > > LAPACK in SLEPc > > > > On Thu, Jun 20, 2013 at 10:24 PM, Harshad Sahasrabudhe > > wrote: > > > > Hi, > > > > I'm trying to run the SLEPc Eigensolver on a machine with CPUs & > > GPUs. I found that a package called MAGMA implements LAPACK > > functions on heterogeneous systems (CPU + GPU). MAGMA has > > functions very similar to LAPACK so that we can go and replace a > > call to a LAPACK function with a call to a MAGMA function. > > > > Now my question is, where can I apply such a hack inside > > SLEPc/PETSc so that the Eigensolver in SLEPc actually uses MAGMA > > functions instead of LAPACK functions? I'm using the PETSc 3.3-p6 > > and SLEPc 3.3-p3 versions. > > > > Any help would be greatly appreciated! > > > > > > You need to do this in PETSc configure. Use --with-f-blas-lapack-lib=[ > libs>]. We have never tried this, so I don't > > know if they obey the entire interface. > > > > Thanks > > > > Thanks and Regards, > > Harshad > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments > > is infinitely more interesting than any results to which their experiments > > lead. > > -- Norbert Wiener > > From hsahasra at purdue.edu Thu Jun 20 16:09:39 2013 From: hsahasra at purdue.edu (Harshad Sahasrabudhe) Date: Thu, 20 Jun 2013 17:09:39 -0400 Subject: [petsc-users] [EXTERNAL] Re: Using MAGMA subroutines instead of LAPACK in SLEPc In-Reply-To: References: <51C36C0B.3060500@purdue.edu> Message-ID: <51C36F93.8030701@purdue.edu> Thanks! That's exactly what I was looking for! Harshad On 06/20/2013 04:59 PM, Satish Balay wrote: > check include/petscblaslapack*.h > > You might have to add a new one [or modify one of the relavent one for > your compiler] - similar to stuff in petscblaslapack_stdcall.h > > Satish > > On Thu, 20 Jun 2013, Harshad Sahasrabudhe wrote: > >> Hi, >> >> Yes, MAGMA's API is not exactly the same. There are a few additional arguments >> in the functions related to number of GPU etc. So it cannot be directly linked >> instead of LAPACK. >> >> Hacking SLEPc/PETSc code seems to be the easier way to go. But I can't find >> the exact place where the LAPACK functions are called. >> >> Thanks, >> Harshad >> >> On 06/20/2013 04:49 PM, Teranishi, Keita wrote: >>> Hi, >>> >>> As far as I know, Magma's API us not exactly the same as the original >>> LAPACK. You may need to hack either Magma's Fortran API or internal >>> SLEPC/PETSc source. >>> >>> Thanks, >>> --------------------------------------------- >>> Keita Teranishi >>> R&D Principal Staff Member >>> Sandia National Laboratories >>> Livermore, CA 94551 >>> >>> From: Matthew Knepley > >>> Date: Thursday, June 20, 2013 1:42 PM >>> To: Harshad Sahasrabudhe > >>> Cc: "petsc-users at mcs.anl.gov " >>> > >>> Subject: [EXTERNAL] Re: [petsc-users] Using MAGMA subroutines instead of >>> LAPACK in SLEPc >>> >>> On Thu, Jun 20, 2013 at 10:24 PM, Harshad Sahasrabudhe >> > wrote: >>> >>> Hi, >>> >>> I'm trying to run the SLEPc Eigensolver on a machine with CPUs & >>> GPUs. I found that a package called MAGMA implements LAPACK >>> functions on heterogeneous systems (CPU + GPU). MAGMA has >>> functions very similar to LAPACK so that we can go and replace a >>> call to a LAPACK function with a call to a MAGMA function. >>> >>> Now my question is, where can I apply such a hack inside >>> SLEPc/PETSc so that the Eigensolver in SLEPc actually uses MAGMA >>> functions instead of LAPACK functions? I'm using the PETSc 3.3-p6 >>> and SLEPc 3.3-p3 versions. >>> >>> Any help would be greatly appreciated! >>> >>> >>> You need to do this in PETSc configure. Use --with-f-blas-lapack-lib=[>> libs>]. We have never tried this, so I don't >>> know if they obey the entire interface. >>> >>> Thanks >>> >>> Thanks and Regards, >>> Harshad >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments >>> is infinitely more interesting than any results to which their experiments >>> lead. >>> -- Norbert Wiener >> From bsmith at mcs.anl.gov Thu Jun 20 16:20:24 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 20 Jun 2013 16:20:24 -0500 Subject: [petsc-users] How to get bandwidth peak out of PETSc log ? In-Reply-To: References: Message-ID: Please send also the -log_summary for 1 process. Note that in the run you provided the time spent in PETSc is about 25 percent of the total run. So how the "other" portion of the code scales will make a large difference for speedup, hence we need to see with a different number of processes. Barry On Jun 20, 2013, at 3:24 PM, HOUSSEN Franck wrote: > Hello, > > I am new to PETSc. > > I have written a (MPI) PETSc code to solve an AX=B system : how to know the bandwidth peak for a given run ? The code does not scale as I would expect (doubling the number of MPI processes does not half the elapsed time) : I would like to understand if this behavior is related to a bad MPI parallelization (that I may be able to improve), or, to the fact that the bandwidth limit as been reached (and in this case, my understanding is that I can not do anything to improve neither the performance, nor the scaling). I would like to know what's going on and why ! > > Concerning the computer, I have tried to estimate the bandwidth peak with the "stream benchmark" : www.streambench.org/index.html. I get this : > ~>./stream_c.exe > ... > Function Best Rate MB/s Avg time Min time Max time > Copy: 11473.8 0.014110 0.013945 0.015064 > Scale: 11421.2 0.014070 0.014009 0.014096 > Add: 12974.4 0.018537 0.018498 0.018590 > Triad: 12964.6 0.018683 0.018512 0.019277 > As a conclusion, my understanding is that the bandwidth peak of my computer is about 12 200 MB / s (= average between 11 400 and 12 900) which is about 12200 / 1024 = 11.9 GB/s. > > Concerning PETSc, I tried to find (without success) a figure to compare to 11.9 GB/s. First, I tried to run "make streams" but it doesn't compile (I run petsc3.4.1. on Ubuntu 12.04). Then, I looked into the PETSc log (using -log_summary and -ksp_view) but I was not able to find out an estimate of the bandwidth out of it (I get information about time and flops but not about the amount of data transfered between MPI processes in MB). How can I get bandwidth peak out of PETSc log ? Is there a specific option for that or is this not possible ? > > I have attached the PETSc log (MPI run over 2 processes). > > Thanks, > > FH > From ondrej.certik at gmail.com Thu Jun 20 18:57:33 2013 From: ondrej.certik at gmail.com (=?UTF-8?B?T25kxZllaiDEjGVydMOtaw==?=) Date: Thu, 20 Jun 2013 17:57:33 -0600 Subject: [petsc-users] Make PETSc not link with systemwide Atlas Message-ID: Hi, I want to strictly link to my own Lapack installed in $LAPACK: $ ls $LAPACK/lib cmake libblas.so liblapack.so pkgconfig So I configure petsc with: $PYTHON/bin/python ./config/configure.py --with-pic --with-clanguage=C --with-cc=$MPI/bin/mpicc --with-mpi-compilers --with-shared-libraries --prefix=$ARTIFACT --with-fortran=0 --with-blas-lapack-dir=$LAPACK/lib The full log is attached with exact paths etc. The problem is that petsc still picks up systemwide Atlas, as can be seen from the end of configure.log: BLAS/LAPACK: -Wl,-rpath,/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib -L/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib -llapack -Wl,-rpath,/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib -L/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib -lcblas -Wl,-rpath,/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib -L/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib -lf77blas -Wl,-rpath,/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib -L/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib -latlas as well as from ldd: ondrej at cekees-MacBookPro-ubuntu:~/repos/python-hpcmp2$ ldd opt/petsc/zlfe/lib/libpetsc.so linux-gate.so.1 => (0xb7742000) libX11.so.6 => /usr/lib/i386-linux-gnu/libX11.so.6 (0xb672d000) libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb6712000) liblapack.so => /home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib/liblapack.so (0xb5e6c000) libf77blas.so.3 => /usr/lib/libf77blas.so.3 (0xb5e4d000) libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xb5e0a000) libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xb5e05000) libmpich.so.10 => /home/ondrej/repos/python-hpcmp2/opt/mpich/c7uq/lib/libmpich.so.10 (0xb5b91000) libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb59dd000) libxcb.so.1 => /usr/lib/i386-linux-gnu/libxcb.so.1 (0xb59bb000) /lib/ld-linux.so.2 (0xb7743000) libblas.so => /home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib/libblas.so (0xb591b000) libgfortran.so.3 => /usr/lib/i386-linux-gnu/libgfortran.so.3 (0xb5823000) libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb5806000) libcblas.so.3 => /usr/lib/libcblas.so.3 (0xb57e5000) libatlas.so.3 => /usr/lib/libatlas.so.3 (0xb5504000) libmpl.so.1 => /home/ondrej/repos/python-hpcmp2/opt/mpich/c7uq/lib/libmpl.so.1 (0xb54fe000) librt.so.1 => /lib/i386-linux-gnu/librt.so.1 (0xb54f5000) libXau.so.6 => /usr/lib/i386-linux-gnu/libXau.so.6 (0xb54f1000) libXdmcp.so.6 => /usr/lib/i386-linux-gnu/libXdmcp.so.6 (0xb54e9000) libquadmath.so.0 => /usr/lib/i386-linux-gnu/libquadmath.so.0 (0xb5471000) as you can see, it actually does link to my blas/lapack, but also to systemwide cblas and atlas... I want the ldd to look like this (my other machine): ondrej at kittiwake:~/repos/python-hpcmp2((bca3099...))$ ldd local/lib/libpetsc.so linux-vdso.so.1 => (0x00007fff1dd1f000) libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007fe85f46f000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe85f252000) liblapack.so => /auto/netscratch/ondrej/opt/lapack/xq2x/lib/liblapack.so (0x00007fe85e768000) libblas.so => /auto/netscratch/ondrej/opt/lapack/xq2x/lib/libblas.so (0x00007fe85e4a8000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe85e1ac000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe85dfa7000) libmpich.so.10 => /auto/netscratch/ondrej/opt/mpich/c7uq/lib/libmpich.so.10 (0x00007fe85db3d000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe85d77e000) libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007fe85d55f000) /lib64/ld-linux-x86-64.so.2 (0x00007fe8609a9000) libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007fe85d248000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe85d032000) libmpl.so.1 => /auto/netscratch/ondrej/opt/mpich/c7uq/lib/libmpl.so.1 (0x00007fe85ce2b000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fe85cc23000) libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007fe85ca1f000) libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007fe85c819000) libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007fe85c5e3000) How can I do that? I read here: http://www.mcs.anl.gov/petsc/documentation/installation.html#blas-lapack that maybe I should try to use the "--with-blas-lapack-lib" option. But I wanted to first check if I am doing something wrong, or what the intended way to setup Lapack is. Thanks for any tips, Ondrej Certik -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log.gz Type: application/x-gzip Size: 204682 bytes Desc: not available URL: From balay at mcs.anl.gov Thu Jun 20 19:56:05 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 20 Jun 2013 19:56:05 -0500 (CDT) Subject: [petsc-users] Make PETSc not link with systemwide Atlas In-Reply-To: References: Message-ID: To strictly specify what you want - use --with-blas-lapack-lib i.e --with-blas-lapack-lib="-L${LAPACK}/lib -llapack -lblas" When you specify --with-blas-lapack-dir=$LAPACK/lib configure is guessing the library names in a perticular order. It so happens that it was guessing atlas before -lblas and compiler found atlas in std locations [and that got picked up] Satish On Thu, 20 Jun 2013, Ond?ej ?ert?k wrote: > Hi, > > I want to strictly link to my own Lapack installed in $LAPACK: > > $ ls $LAPACK/lib > cmake libblas.so liblapack.so pkgconfig > > So I configure petsc with: > > $PYTHON/bin/python ./config/configure.py --with-pic --with-clanguage=C > --with-cc=$MPI/bin/mpicc --with-mpi-compilers --with-shared-libraries > --prefix=$ARTIFACT --with-fortran=0 --with-blas-lapack-dir=$LAPACK/lib > > The full log is attached with exact paths etc. The problem is that > petsc still picks up systemwide Atlas, as can be seen from the end of > configure.log: > > BLAS/LAPACK: -Wl,-rpath,/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib > -L/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib -llapack > -Wl,-rpath,/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib > -L/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib -lcblas > -Wl,-rpath,/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib > -L/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib -lf77blas > -Wl,-rpath,/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib > -L/home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib -latlas > > as well as from ldd: > > ondrej at cekees-MacBookPro-ubuntu:~/repos/python-hpcmp2$ ldd > opt/petsc/zlfe/lib/libpetsc.so > linux-gate.so.1 => (0xb7742000) > libX11.so.6 => /usr/lib/i386-linux-gnu/libX11.so.6 (0xb672d000) > libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb6712000) > liblapack.so => > /home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib/liblapack.so > (0xb5e6c000) > libf77blas.so.3 => /usr/lib/libf77blas.so.3 (0xb5e4d000) > libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xb5e0a000) > libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xb5e05000) > libmpich.so.10 => > /home/ondrej/repos/python-hpcmp2/opt/mpich/c7uq/lib/libmpich.so.10 > (0xb5b91000) > libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb59dd000) > libxcb.so.1 => /usr/lib/i386-linux-gnu/libxcb.so.1 (0xb59bb000) > /lib/ld-linux.so.2 (0xb7743000) > libblas.so => > /home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib/libblas.so > (0xb591b000) > libgfortran.so.3 => /usr/lib/i386-linux-gnu/libgfortran.so.3 (0xb5823000) > libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb5806000) > libcblas.so.3 => /usr/lib/libcblas.so.3 (0xb57e5000) > libatlas.so.3 => /usr/lib/libatlas.so.3 (0xb5504000) > libmpl.so.1 => > /home/ondrej/repos/python-hpcmp2/opt/mpich/c7uq/lib/libmpl.so.1 > (0xb54fe000) > librt.so.1 => /lib/i386-linux-gnu/librt.so.1 (0xb54f5000) > libXau.so.6 => /usr/lib/i386-linux-gnu/libXau.so.6 (0xb54f1000) > libXdmcp.so.6 => /usr/lib/i386-linux-gnu/libXdmcp.so.6 (0xb54e9000) > libquadmath.so.0 => /usr/lib/i386-linux-gnu/libquadmath.so.0 (0xb5471000) > > as you can see, it actually does link to my blas/lapack, but also to > systemwide cblas and atlas... > > I want the ldd to look like this (my other machine): > > ondrej at kittiwake:~/repos/python-hpcmp2((bca3099...))$ ldd > local/lib/libpetsc.so linux-vdso.so.1 => (0x00007fff1dd1f000) > libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007fe85f46f000) > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > (0x00007fe85f252000) > liblapack.so => > /auto/netscratch/ondrej/opt/lapack/xq2x/lib/liblapack.so > (0x00007fe85e768000) > libblas.so => > /auto/netscratch/ondrej/opt/lapack/xq2x/lib/libblas.so > (0x00007fe85e4a8000) > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe85e1ac000) > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe85dfa7000) > libmpich.so.10 => > /auto/netscratch/ondrej/opt/mpich/c7uq/lib/libmpich.so.10 > (0x00007fe85db3d000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe85d77e000) > libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007fe85d55f000) > /lib64/ld-linux-x86-64.so.2 (0x00007fe8609a9000) > libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 > (0x00007fe85d248000) > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe85d032000) > libmpl.so.1 => > /auto/netscratch/ondrej/opt/mpich/c7uq/lib/libmpl.so.1 > (0x00007fe85ce2b000) > librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fe85cc23000) > libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007fe85ca1f000) > libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 > (0x00007fe85c819000) > libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 > (0x00007fe85c5e3000) > > > How can I do that? I read here: > > http://www.mcs.anl.gov/petsc/documentation/installation.html#blas-lapack > > that maybe I should try to use the "--with-blas-lapack-lib" option. > But I wanted to first check if I am doing something wrong, or what the > intended way to setup Lapack is. > > Thanks for any tips, > Ondrej Certik > From fd.kong at siat.ac.cn Thu Jun 20 22:16:08 2013 From: fd.kong at siat.ac.cn (Fande Kong) Date: Fri, 21 Jun 2013 11:16:08 +0800 Subject: [petsc-users] How to understand these error messages Message-ID: Hi all, How to understand this error message? Some things about mpi or something else? [3229]PETSC ERROR: --------------------- Error Message ------------------------------------ [3229]PETSC ERROR: Petsc has generated inconsistent data! [3229]PETSC ERROR: Negative MPI source!! [3229]PETSC ERROR: ------------------------------------------------------------------------ [3229]PETSC ERROR: Petsc Release Version 3.4.1, unknown [3229]PETSC ERROR: See docs/changes/index.html for recent updates. [3229]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [3229]PETSC ERROR: See docs/index.html for manual pages. [3229]PETSC ERROR: ------------------------------------------------------------------------ [3229]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-opt named ys0850 by fandek Thu Jun 20 07:41:52 2013 [3229]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-opt/lib [3229]PETSC ERROR: Configure run at Thu Jun 20 06:23:01 2013 [3229]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=no [3229]PETSC ERROR: ------------------------------------------------------------------------ [3229]PETSC ERROR: MatStashScatterGetMesg_Private() line 632 in /src/mat/utilsmatstash.c [3229]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c [3229]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c [3229]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp [3229]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp [3229]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp [3229]PETSC ERROR: DMmeshInitialize() line 76 in mgInitialize.cpp The details of the messages are attached. And log files for configure and make are also attached. Regards, -- Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: solidcube.982720.err Type: application/octet-stream Size: 400569 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure and make log.zip Type: application/zip Size: 773715 bytes Desc: not available URL: From knepley at gmail.com Fri Jun 21 00:02:31 2013 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 21 Jun 2013 07:02:31 +0200 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: Message-ID: On Fri, Jun 21, 2013 at 5:16 AM, Fande Kong wrote: > Hi all, > > How to understand this error message? Some things about mpi or something > else? > This says that you are trying to send values to a negative MPI rank. This has to be a memory overwrite somewhere in your code I think. Matt > [3229]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [3229]PETSC ERROR: Petsc has generated inconsistent data! > [3229]PETSC ERROR: Negative MPI source!! > [3229]PETSC ERROR: > ------------------------------------------------------------------------ > [3229]PETSC ERROR: Petsc Release Version 3.4.1, unknown > [3229]PETSC ERROR: See docs/changes/index.html for recent updates. > [3229]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [3229]PETSC ERROR: See docs/index.html for manual pages. > [3229]PETSC ERROR: > ------------------------------------------------------------------------ > [3229]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-opt named > ys0850 by fandek Thu Jun 20 07:41:52 2013 > [3229]PETSC ERROR: Libraries linked from > /glade/p/work/fandek/petsc/arch-linux2-cxx-opt/lib > [3229]PETSC ERROR: Configure run at Thu Jun 20 06:23:01 2013 > [3229]PETSC ERROR: Configure options --with-valgrind=1 > --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 > --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 > --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 > --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 > --with-debugging=no > [3229]PETSC ERROR: > ------------------------------------------------------------------------ > [3229]PETSC ERROR: MatStashScatterGetMesg_Private() line 632 in > /src/mat/utilsmatstash.c > [3229]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in > /src/mat/impls/aij/mpimpiaij.c > [3229]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c > [3229]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in > meshreorder.cpp > [3229]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in > meshreorder.cpp > [3229]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > [3229]PETSC ERROR: DMmeshInitialize() line 76 in mgInitialize.cpp > > > The details of the messages are attached. And log files for configure and > make are also attached. > > Regards, > -- > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fd.kong at siat.ac.cn Fri Jun 21 00:25:12 2013 From: fd.kong at siat.ac.cn (Fande Kong) Date: Fri, 21 Jun 2013 13:25:12 +0800 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: Message-ID: These messages come from MatStashScatterGetMesg_Private() line 632 in /src/mat/utilsmatstash.c. I do not send values. I just try to use MatAssemblyEnd/begin() to assembly stiffness matrix. On Fri, Jun 21, 2013 at 1:02 PM, Matthew Knepley wrote: > On Fri, Jun 21, 2013 at 5:16 AM, Fande Kong wrote: > >> Hi all, >> >> How to understand this error message? Some things about mpi or something >> else? >> > > This says that you are trying to send values to a negative MPI rank. This > has to be a > memory overwrite somewhere in your code I think. > > Matt > > >> [3229]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [3229]PETSC ERROR: Petsc has generated inconsistent data! >> [3229]PETSC ERROR: Negative MPI source!! >> [3229]PETSC ERROR: >> ------------------------------------------------------------------------ >> [3229]PETSC ERROR: Petsc Release Version 3.4.1, unknown >> [3229]PETSC ERROR: See docs/changes/index.html for recent updates. >> [3229]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [3229]PETSC ERROR: See docs/index.html for manual pages. >> [3229]PETSC ERROR: >> ------------------------------------------------------------------------ >> [3229]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-opt named >> ys0850 by fandek Thu Jun 20 07:41:52 2013 >> [3229]PETSC ERROR: Libraries linked from >> /glade/p/work/fandek/petsc/arch-linux2-cxx-opt/lib >> [3229]PETSC ERROR: Configure run at Thu Jun 20 06:23:01 2013 >> [3229]PETSC ERROR: Configure options --with-valgrind=1 >> --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 >> --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 >> --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 >> --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 >> --with-debugging=no >> [3229]PETSC ERROR: >> ------------------------------------------------------------------------ >> [3229]PETSC ERROR: MatStashScatterGetMesg_Private() line 632 in >> /src/mat/utilsmatstash.c >> [3229]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in >> /src/mat/impls/aij/mpimpiaij.c >> [3229]PETSC ERROR: MatAssemblyEnd() line 4939 in >> /src/mat/interfacematrix.c >> [3229]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in >> meshreorder.cpp >> [3229]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in >> meshreorder.cpp >> [3229]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp >> [3229]PETSC ERROR: DMmeshInitialize() line 76 in mgInitialize.cpp >> >> >> The details of the messages are attached. And log files for configure and >> make are also attached. >> >> Regards, >> -- >> Fande Kong >> ShenZhen Institutes of Advanced Technology >> Chinese Academy of Sciences >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 21 01:39:02 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 21 Jun 2013 08:39:02 +0200 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: Message-ID: <87bo703r3t.fsf@mcs.anl.gov> Fande Kong writes: > These messages come from MatStashScatterGetMesg_Private() line 632 in > /src/mat/utilsmatstash.c. I do not send values. I just try to > use MatAssemblyEnd/begin() to assembly stiffness matrix. It is most likely memory corruption caused by something earlier. Is it reproducible? (It looks like you're running at fairly large scale.) I wonder if something earlier misuses 32- versus 64-bit ints. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From Franck.Houssen at cea.fr Fri Jun 21 01:45:54 2013 From: Franck.Houssen at cea.fr (HOUSSEN Franck) Date: Fri, 21 Jun 2013 06:45:54 +0000 Subject: [petsc-users] RE : How to get bandwidth peak out of PETSc log ? In-Reply-To: References: , Message-ID: Hello, The log I attached was a very small test case (1 iteration) : I just wanted to get some information about PETSc log printings (time, flops...). I profiled the code over "big" test case with scalasca : I know I spend 95% of the time calling PETSc (on realistic test cases). Barry, I have attached 2 logs over 1 and 2 procs running a intermediate test case (not small but not big). >From PETSc_1proc.log, I get : Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 2.6379e+02 100.0% 9.8232e+10 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 2.943e+03 100.0% My understanding is that the relevant figure is 9.8232e+10 (flop) / 2.6379e+02 (sec) = 3.72e12 flops = 3723 gflops for 1 proc. >From PETSc_2procs.log, I get : Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.6733e+02 100.0% 1.0348e+11 100.0% 3.427e+04 100.0% 2.951e+04 100.0% 1.355e+04 100.0% My understanding is that the relevant figure is 1.0348e+11 (flop) / 1.6733e+02 (sec) = 6.18e12 flops = 6184 gflops for 2 procs. Am I correct ? My understanding is that if the code scales "well", when I double the number of MPI processes, I should double the flops which is not the case (I get 6184 gflops instead of 2*3723) : right ? wrong ? >From this, how can I know if I am at the bandwidth peak ? If I compare 6184 gflops to the (computer characteristic computed by Matt = 12.9 GB/s * 1 flops/12 bytes =) 1.1 GF/s, I get a huge difference that I am not sure to understand... I am not sure I can compare these numbers : I guess no ?! Did I miss something ? More over I realize that I get : ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ VecAXPY 14402 1.0 5.4951e+00 1.0 6.63e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 7 0 0 0 2 7 0 0 0 1206 (in PETSc_1proc.log => 1.2 gflops which is superior to the 1.1 gflops computed by Matt) VecAXPY 14402 1.0 4.3619e+00 1.0 3.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00 3 6 0 0 0 3 6 0 0 0 1520 (in PETSc_2procs.log => 1.5 gflops which is superior to the 1.1 gflops computed by Matt) My understanding is that I can conclude I am at the bandwidth peak if I rely on : VecAXPY 14402 1.0 5.4951e+00 1.0 6.63e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 7 0 0 0 2 7 0 0 0 1206 (in PETSc_1proc.log => 1.2 gflops which is superior to the 1.1 gflops computed by Matt) VecAXPY 14402 1.0 4.3619e+00 1.0 3.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00 3 6 0 0 0 3 6 0 0 0 1520 (in PETSc_2procs.log => 1.5 gflops which is superior to the 1.1 gflops computed by Matt) But I can not conclude anything when I rely on 3723 gflops for 1 proc / 6184 gflops for 2 procs. Can somebody help me to see clear on that ?! Thanks, FH ________________________________________ De : Barry Smith [bsmith at mcs.anl.gov] Date d'envoi : jeudi 20 juin 2013 23:20 ? : HOUSSEN Franck Cc: petsc-users at mcs.anl.gov Objet : Re: [petsc-users] How to get bandwidth peak out of PETSc log ? Please send also the -log_summary for 1 process. Note that in the run you provided the time spent in PETSc is about 25 percent of the total run. So how the "other" portion of the code scales will make a large difference for speedup, hence we need to see with a different number of processes. Barry On Jun 20, 2013, at 3:24 PM, HOUSSEN Franck wrote: > Hello, > > I am new to PETSc. > > I have written a (MPI) PETSc code to solve an AX=B system : how to know the bandwidth peak for a given run ? The code does not scale as I would expect (doubling the number of MPI processes does not half the elapsed time) : I would like to understand if this behavior is related to a bad MPI parallelization (that I may be able to improve), or, to the fact that the bandwidth limit as been reached (and in this case, my understanding is that I can not do anything to improve neither the performance, nor the scaling). I would like to know what's going on and why ! > > Concerning the computer, I have tried to estimate the bandwidth peak with the "stream benchmark" : www.streambench.org/index.html. I get this : > ~>./stream_c.exe > ... > Function Best Rate MB/s Avg time Min time Max time > Copy: 11473.8 0.014110 0.013945 0.015064 > Scale: 11421.2 0.014070 0.014009 0.014096 > Add: 12974.4 0.018537 0.018498 0.018590 > Triad: 12964.6 0.018683 0.018512 0.019277 > As a conclusion, my understanding is that the bandwidth peak of my computer is about 12 200 MB / s (= average between 11 400 and 12 900) which is about 12200 / 1024 = 11.9 GB/s. > > Concerning PETSc, I tried to find (without success) a figure to compare to 11.9 GB/s. First, I tried to run "make streams" but it doesn't compile (I run petsc3.4.1. on Ubuntu 12.04). Then, I looked into the PETSc log (using -log_summary and -ksp_view) but I was not able to find out an estimate of the bandwidth out of it (I get information about time and flops but not about the amount of data transfered between MPI processes in MB). How can I get bandwidth peak out of PETSc log ? Is there a specific option for that or is this not possible ? > > I have attached the PETSc log (MPI run over 2 processes). > > Thanks, > > FH > -------------- next part -------------- A non-text attachment was scrubbed... Name: PETSc_1proc.log Type: text/x-log Size: 12342 bytes Desc: PETSc_1proc.log URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PETSc_2procs.log Type: text/x-log Size: 12972 bytes Desc: PETSc_2procs.log URL: From knepley at gmail.com Fri Jun 21 01:56:28 2013 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 21 Jun 2013 08:56:28 +0200 Subject: [petsc-users] RE : How to get bandwidth peak out of PETSc log ? In-Reply-To: References: Message-ID: On Fri, Jun 21, 2013 at 8:45 AM, HOUSSEN Franck wrote: > Hello, > > The log I attached was a very small test case (1 iteration) : I just > wanted to get some information about PETSc log printings (time, flops...). > I profiled the code over "big" test case with scalasca : I know I spend 95% > of the time calling PETSc (on realistic test cases). > > Barry, > I have attached 2 logs over 1 and 2 procs running a intermediate test case > (not small but not big). > From PETSc_1proc.log, I get : > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages > --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts > %Total Avg %Total counts %Total > 0: Main Stage: 2.6379e+02 100.0% 9.8232e+10 100.0% 0.000e+00 > 0.0% 0.000e+00 0.0% 2.943e+03 100.0% > My understanding is that the relevant figure is 9.8232e+10 (flop) / > 2.6379e+02 (sec) = 3.72e12 flops = 3723 gflops for 1 proc. > From PETSc_2procs.log, I get : > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages > --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts > %Total Avg %Total counts %Total > 0: Main Stage: 1.6733e+02 100.0% 1.0348e+11 100.0% 3.427e+04 > 100.0% 2.951e+04 100.0% 1.355e+04 100.0% > My understanding is that the relevant figure is 1.0348e+11 (flop) / > 1.6733e+02 (sec) = 6.18e12 flops = 6184 gflops for 2 procs. > > Am I correct ? > > My understanding is that if the code scales "well", when I double the > number of MPI processes, I should double the flops which is not the case (I > get 6184 gflops instead of 2*3723) : right ? wrong ? > From this, how can I know if I am at the bandwidth peak ? > > If I compare 6184 gflops to the (computer characteristic computed by Matt > = 12.9 GB/s * 1 flops/12 bytes =) 1.1 GF/s, I get a huge difference that I > am not sure to understand... I am not sure I can compare these numbers : I > guess no ?! Did I miss something ? > The bandwidth limit depends on the operation. Some operations, like DGEMM have no limit, whereas others like VecAXPY do. That is why it is meaningless to just report flop numbers. > More over I realize that I get : > > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flops > --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len > Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s > > ------------------------------------------------------------------------------------------------------------------------ > VecAXPY 14402 1.0 5.4951e+00 1.0 6.63e+09 1.0 0.0e+00 0.0e+00 > 0.0e+00 2 7 0 0 0 2 7 0 0 0 1206 (in PETSc_1proc.log => 1.2 > gflops which is superior to the 1.1 gflops computed by Matt) > VecAXPY 14402 1.0 4.3619e+00 1.0 3.32e+09 1.0 0.0e+00 0.0e+00 > 0.0e+00 3 6 0 0 0 3 6 0 0 0 1520 (in PETSc_2procs.log => 1.5 > gflops which is superior to the 1.1 gflops computed by Matt) > > My understanding is that I can conclude I am at the bandwidth peak if I > rely on : > VecAXPY 14402 1.0 5.4951e+00 1.0 6.63e+09 1.0 0.0e+00 0.0e+00 > 0.0e+00 2 7 0 0 0 2 7 0 0 0 1206 (in PETSc_1proc.log => 1.2 > gflops which is superior to the 1.1 gflops computed by Matt) > VecAXPY 14402 1.0 4.3619e+00 1.0 3.32e+09 1.0 0.0e+00 0.0e+00 > 0.0e+00 3 6 0 0 0 3 6 0 0 0 1520 (in PETSc_2procs.log => 1.5 > gflops which is superior to the 1.1 gflops computed by Matt) > > But I can not conclude anything when I rely on 3723 gflops for 1 proc / > 6184 gflops for 2 procs. > > Can somebody help me to see clear on that ?! > What the above shows is that you get very little improvement from the extra process, so your machine does not have enough bandwidth to support those two 2 processors for this operation. Thus I would not expect to see speedup from 1 to 2 processors on this machine. Matt > Thanks, > > FH > > ________________________________________ > De : Barry Smith [bsmith at mcs.anl.gov] > Date d'envoi : jeudi 20 juin 2013 23:20 > ? : HOUSSEN Franck > Cc: petsc-users at mcs.anl.gov > Objet : Re: [petsc-users] How to get bandwidth peak out of PETSc log ? > > Please send also the -log_summary for 1 process. > > Note that in the run you provided the time spent in PETSc is about 25 > percent of the total run. So how the "other" portion of the code scales > will make a large difference for speedup, hence we need to see with a > different number of processes. > > Barry > > On Jun 20, 2013, at 3:24 PM, HOUSSEN Franck wrote: > > > Hello, > > > > I am new to PETSc. > > > > I have written a (MPI) PETSc code to solve an AX=B system : how to know > the bandwidth peak for a given run ? The code does not scale as I would > expect (doubling the number of MPI processes does not half the elapsed > time) : I would like to understand if this behavior is related to a bad MPI > parallelization (that I may be able to improve), or, to the fact that the > bandwidth limit as been reached (and in this case, my understanding is that > I can not do anything to improve neither the performance, nor the scaling). > I would like to know what's going on and why ! > > > > Concerning the computer, I have tried to estimate the bandwidth peak > with the "stream benchmark" : www.streambench.org/index.html. I get this : > > ~>./stream_c.exe > > ... > > Function Best Rate MB/s Avg time Min time Max time > > Copy: 11473.8 0.014110 0.013945 0.015064 > > Scale: 11421.2 0.014070 0.014009 0.014096 > > Add: 12974.4 0.018537 0.018498 0.018590 > > Triad: 12964.6 0.018683 0.018512 0.019277 > > As a conclusion, my understanding is that the bandwidth peak of my > computer is about 12 200 MB / s (= average between 11 400 and 12 900) which > is about 12200 / 1024 = 11.9 GB/s. > > > > Concerning PETSc, I tried to find (without success) a figure to compare > to 11.9 GB/s. First, I tried to run "make streams" but it doesn't compile > (I run petsc3.4.1. on Ubuntu 12.04). Then, I looked into the PETSc log > (using -log_summary and -ksp_view) but I was not able to find out an > estimate of the bandwidth out of it (I get information about time and flops > but not about the amount of data transfered between MPI processes in MB). > How can I get bandwidth peak out of PETSc log ? Is there a specific option > for that or is this not possible ? > > > > I have attached the PETSc log (MPI run over 2 processes). > > > > Thanks, > > > > FH > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fd.kong at siat.ac.cn Fri Jun 21 02:05:08 2013 From: fd.kong at siat.ac.cn (Fande Kong) Date: Fri, 21 Jun 2013 15:05:08 +0800 Subject: [petsc-users] How to understand these error messages In-Reply-To: <87bo703r3t.fsf@mcs.anl.gov> References: <87bo703r3t.fsf@mcs.anl.gov> Message-ID: The code works well with less cores. And It also works well with petsc-3.3-p7. But it does not work with petsc-3.4.1. Thus, If you can check the differences between petsc-3.3-p7 and petsc-3.4.1, you can figure out the reason. On Fri, Jun 21, 2013 at 2:39 PM, Jed Brown wrote: > Fande Kong writes: > > > These messages come from MatStashScatterGetMesg_Private() line 632 in > > /src/mat/utilsmatstash.c. I do not send values. I just try to > > use MatAssemblyEnd/begin() to assembly stiffness matrix. > > It is most likely memory corruption caused by something earlier. Is it > reproducible? (It looks like you're running at fairly large scale.) I > wonder if something earlier misuses 32- versus 64-bit ints. > -- Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 21 05:33:34 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 21 Jun 2013 12:33:34 +0200 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> Message-ID: <87vc573g8x.fsf@mcs.anl.gov> Fande Kong writes: > The code works well with less cores. And It also works well with > petsc-3.3-p7. But it does not work with petsc-3.4.1. Thus, If you can check > the differences between petsc-3.3-p7 and petsc-3.4.1, you can figure out > the reason. That is one way to start debugging, but there are no changes to the core MatStash code, and many, many changes to PETSc in total. The relevant snippet of code is here: if (stash->reproduce) { i = stash->reproduce_count++; ierr = MPI_Wait(stash->recv_waits+i,&recv_status);CHKERRQ(ierr); } else { ierr = MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);CHKERRQ(ierr); } if (recv_status.MPI_SOURCE < 0) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source!"); So MPI returns correctly (stash->reproduce will be FALSE unless you changed it). You could change the line above to the following: if (recv_status.MPI_SOURCE < 0) SETERRQ5(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source: stash->nrecvs=%D i=%d MPI_SOURCE=%d MPI_TAG=%d MPI_ERROR=%d", stash->nrecvs,i,recv_status.MPI_SOURCE,recv_status.MPI_TAG,recv_status.MPI_ERROR); It would help to debug --with-debugging=1, so that more checks for corrupt data are performed. You can still make the compiler optimize if it takes a long time to reach the error condition. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Fri Jun 21 05:41:11 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 21 Jun 2013 12:41:11 +0200 Subject: [petsc-users] RE : How to get bandwidth peak out of PETSc log ? In-Reply-To: References: Message-ID: <87sj0b3fw8.fsf@mcs.anl.gov> HOUSSEN Franck writes: > MatMult 13216 1.0 5.4614e+01 1.0 5.52e+10 1.0 0.0e+00 0.0e+00 0.0e+00 21 56 0 0 0 21 56 0 0 0 1010 Much of your time is spent here, which is bandwidth limited. It needs 6 bytes per flop (plus a little, if the vector is perfectly reused) , so this number is about 6 GB/s. > MatMult 13550 1.0 3.8204e+01 1.0 2.85e+10 1.0 2.7e+04 3.5e+04 0.0e+00 23 55 79 93 0 23 55 79 93 0 1494 Here with two processes, you have about 9 GB/s. Was your STREAM test (getting 11 GB/s) using multiple threads/processes? Can you send STREAM results for one and for two threads? 50% of STREAM is not very good (though it's actually the best you can do on some funny architectures), 70-85% is what we expect. If you're getting a low fraction of peak in MatMult, try reordering your matrix to have lower bandwidth. You can use MatGetOrdering with RCM for this. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From ztdepyahoo at 163.com Fri Jun 21 07:05:03 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Fri, 21 Jun 2013 20:05:03 +0800 (CST) Subject: [petsc-users] run examples with kdevelop Message-ID: <734479ee.11757.13f669f9a4c.Coremail.ztdepyahoo@163.com> Dear friends: I want to runmy code the kdevelop4.4, but could you provide me a cmake file for it. i can run it smoothly with the make file. but how to translate it into a cmake filel include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules ALL: main main: main.o chkopts -${CLINKER} -o main main.o ${PETSC_KSP_LIB} ${RM} main.o Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Franck.Houssen at cea.fr Fri Jun 21 07:10:05 2013 From: Franck.Houssen at cea.fr (HOUSSEN Franck) Date: Fri, 21 Jun 2013 12:10:05 +0000 Subject: [petsc-users] RE : RE : How to get bandwidth peak out of PETSc log ? In-Reply-To: <87sj0b3fw8.fsf@mcs.anl.gov> References: , <87sj0b3fw8.fsf@mcs.anl.gov> Message-ID: It seems performance are always the same whatever the number of threads. I ran all these tests on my laptop (which is not a "great supercomputer" !... As Matt noticed, and as I knew it !) FH ~/Bandwidth>export OMP_NUM_THREADS=1 ~/Bandwidth>./stream_c.exe > stream_1thread.log ~/Bandwidth>export OMP_NUM_THREADS=2 ~/Bandwidth>./stream_c.exe > stream_2threads.log ~/Bandwidth>export OMP_NUM_THREADS=4 ~/Bandwidth>./stream_c.exe > stream_4threads.log ~/Bandwidth>grep Triad *log stream_1thread.log:Triad: 13057.6 0.018427 0.018380 0.018539 stream_2threads.log:Triad: 13044.9 0.018496 0.018398 0.018936 stream_4threads.log:Triad: 13049.1 0.018449 0.018392 0.018561 ________________________________________ De : Jed Brown [five9a2 at gmail.com] de la part de Jed Brown [jedbrown at mcs.anl.gov] Date d'envoi : vendredi 21 juin 2013 12:41 ? : HOUSSEN Franck; Barry Smith Cc: petsc-users at mcs.anl.gov Objet : Re: [petsc-users] RE : How to get bandwidth peak out of PETSc log ? HOUSSEN Franck writes: > MatMult 13216 1.0 5.4614e+01 1.0 5.52e+10 1.0 0.0e+00 0.0e+00 0.0e+00 21 56 0 0 0 21 56 0 0 0 1010 Much of your time is spent here, which is bandwidth limited. It needs 6 bytes per flop (plus a little, if the vector is perfectly reused) , so this number is about 6 GB/s. > MatMult 13550 1.0 3.8204e+01 1.0 2.85e+10 1.0 2.7e+04 3.5e+04 0.0e+00 23 55 79 93 0 23 55 79 93 0 1494 Here with two processes, you have about 9 GB/s. Was your STREAM test (getting 11 GB/s) using multiple threads/processes? Can you send STREAM results for one and for two threads? 50% of STREAM is not very good (though it's actually the best you can do on some funny architectures), 70-85% is what we expect. If you're getting a low fraction of peak in MatMult, try reordering your matrix to have lower bandwidth. You can use MatGetOrdering with RCM for this. -------------- next part -------------- A non-text attachment was scrubbed... Name: stream_1thread.log Type: text/x-log Size: 1639 bytes Desc: stream_1thread.log URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: stream_2threads.log Type: text/x-log Size: 1639 bytes Desc: stream_2threads.log URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: stream_4threads.log Type: text/x-log Size: 1639 bytes Desc: stream_4threads.log URL: From jedbrown at mcs.anl.gov Fri Jun 21 07:16:48 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 21 Jun 2013 14:16:48 +0200 Subject: [petsc-users] RE : RE : How to get bandwidth peak out of PETSc log ? In-Reply-To: References: <87sj0b3fw8.fsf@mcs.anl.gov> Message-ID: <87ip173bgv.fsf@mcs.anl.gov> HOUSSEN Franck writes: > It seems performance are always the same whatever the number of threads. That's pretty common when there is only one memory bus. > I ran all these tests on my laptop (which is not a "great > supercomputer" !... As Matt noticed, and as I knew it !) Matt does all his "supercomputing" on his laptop. ;-) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Fri Jun 21 07:17:46 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 21 Jun 2013 14:17:46 +0200 Subject: [petsc-users] run examples with kdevelop In-Reply-To: <734479ee.11757.13f669f9a4c.Coremail.ztdepyahoo@163.com> References: <734479ee.11757.13f669f9a4c.Coremail.ztdepyahoo@163.com> Message-ID: <87fvwb3bf9.fsf@mcs.anl.gov> ??? writes: > Dear friends: > I want to runmy code the kdevelop4.4, but could you provide me a cmake file for it. > i can run it smoothly with the make file. but how to translate it into a cmake filel You can use FindPETSc.cmake from https://github.com/jedbrown/cmake-modules. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From ztdepyahoo at 163.com Fri Jun 21 07:29:35 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Fri, 21 Jun 2013 20:29:35 +0800 (CST) Subject: [petsc-users] run examples with kdevelop In-Reply-To: <87fvwb3bf9.fsf@mcs.anl.gov> References: <734479ee.11757.13f669f9a4c.Coremail.ztdepyahoo@163.com> <87fvwb3bf9.fsf@mcs.anl.gov> Message-ID: <6dcfb956.11a5f.13f66b61334.Coremail.ztdepyahoo@163.com> but how to use it? ? 2013-06-21 20:17:46?"Jed Brown" ??? >??? writes: > >> Dear friends: >> I want to runmy code the kdevelop4.4, but could you provide me a cmake file for it. >> i can run it smoothly with the make file. but how to translate it into a cmake filel > >You can use FindPETSc.cmake from https://github.com/jedbrown/cmake-modules. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 21 07:37:37 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 21 Jun 2013 14:37:37 +0200 Subject: [petsc-users] run examples with kdevelop In-Reply-To: <6dcfb956.11a5f.13f66b61334.Coremail.ztdepyahoo@163.com> References: <734479ee.11757.13f669f9a4c.Coremail.ztdepyahoo@163.com> <87fvwb3bf9.fsf@mcs.anl.gov> <6dcfb956.11a5f.13f66b61334.Coremail.ztdepyahoo@163.com> Message-ID: <87a9mj3ai6.fsf@mcs.anl.gov> ??? writes: > but how to use it? There is documentation in the file and you can find examples that use it. Usage is pretty much the same as other Find*.cmake modules. Consult the CMake documentation. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From rupp at mcs.anl.gov Fri Jun 21 08:13:01 2013 From: rupp at mcs.anl.gov (Karl Rupp) Date: Fri, 21 Jun 2013 08:13:01 -0500 Subject: [petsc-users] RE : RE : How to get bandwidth peak out of PETSc log ? In-Reply-To: References: , <87sj0b3fw8.fsf@mcs.anl.gov> Message-ID: <51C4515D.4060607@mcs.anl.gov> Hi Franck, > It seems performance are always the same whatever the number of threads. As Jed already noted, this is pretty common on such machines. Also, the recent Sandy Bridge CPUs from Intel can pretty much saturate the memory link with a single process/thread already, while former architectures often needed a second thread. Thus, if you have a Core i{3,5,7}-{2,3}xyz-CPU from Intel in your laptop, that's exactly what you observe. Best regards, Karli From cchoudary at rnet-tech.com Fri Jun 21 14:09:25 2013 From: cchoudary at rnet-tech.com (Chekuri Choudary) Date: Fri, 21 Jun 2013 15:09:25 -0400 Subject: [petsc-users] OpenFOAM and PETSc Message-ID: <002f01ce6eb2$d949ce90$8bdd6bb0$@rnet-tech.com> OpenFOAM is mentioned in the following slides on PETSc. What exactly is the relationship between OpenFOAM and PETSc? http://www.mcs.anl.gov/petsc/documentation/tutorials/ACTSTutorial2012.pdf How well is PETSc integrated with OpenFOAM? Is it possible to use most of the linear solvers and data types in PETSc for OpenFOAM based applications using the FoamedOver interface mentioned in the link below?(http://oversetgridsymposium.org/assets/pdf/presentations/1_2/Boger_Ov ersetSymposium2010.pdf) Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From ondrej.certik at gmail.com Sat Jun 22 14:03:36 2013 From: ondrej.certik at gmail.com (=?UTF-8?B?T25kxZllaiDEjGVydMOtaw==?=) Date: Sat, 22 Jun 2013 13:03:36 -0600 Subject: [petsc-users] Make PETSc not link with systemwide Atlas In-Reply-To: References: Message-ID: Satish, On Thu, Jun 20, 2013 at 6:56 PM, Satish Balay wrote: > To strictly specify what you want - use --with-blas-lapack-lib > > i.e --with-blas-lapack-lib="-L${LAPACK}/lib -llapack -lblas" > > When you specify --with-blas-lapack-dir=$LAPACK/lib configure is > guessing the library names in a perticular order. It so happens that > it was guessing atlas before -lblas and compiler found atlas in std > locations [and that got picked up] Indeed, this worked! Thanks a lot. For reference, here is the exact configure line that I am using: $PYTHON/bin/python ./config/configure.py --with-pic --with-clanguage=C --with-cc=$MPI/bin/mpicc --with-mpi-compilers --with-shared-libraries --prefix=$ARTIFACT --with-fortran=0 --with-blas-lapack-lib="-L$LAPACK/lib -llapack -lblas" it even sets RPATH correctly: patchelf --print-rpath opt/petsc/n5i4/lib/libpetsc.so /home/ondrej/repos/python-hpcmp2/opt/lapack/xq2x/lib:/home/ondrej/repos/python-hpcmp2/opt/mpich/c7uq/lib so everything works now. Thanks, Ondrej From fd.kong at siat.ac.cn Sun Jun 23 04:12:34 2013 From: fd.kong at siat.ac.cn (Fande Kong) Date: Sun, 23 Jun 2013 03:12:34 -0600 Subject: [petsc-users] How to understand these error messages In-Reply-To: <87vc573g8x.fsf@mcs.anl.gov> References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> Message-ID: Thanks Jed, I added your code into the petsc. I run my code with 10240 cores. I got the following error messages: [6724]PETSC ERROR: --------------------- Error Message ------------------------------------ [6724]PETSC ERROR: Petsc has generated inconsistent data! [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892! [6724]PETSC ERROR: ------------------------------------------------------------------------ [6724]PETSC ERROR: Petsc Release Version 3.4.1, unknown [6724]PETSC ERROR: See docs/changes/index.html for recent updates. [6724]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [6724]PETSC ERROR: See docs/index.html for manual pages. [6724]PETSC ERROR: ------------------------------------------------------------------------ [6724]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug named ys4350 by fandek Sun Jun 23 02:58:23 2013 [6724]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib [6724]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 [6724]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --d ownload-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=yes [6724]PETSC ERROR: ------------------------------------------------------------------------ [6724]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /src/mat/utilsmatstash.c [6724]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c [6724]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c [6724]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp [6724]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp [6724]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp [6724]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp [6724]PETSC ERROR: main() line 71 in linearElasticity3d.cpp Abort(77) on node 6724 (rank 6724 in comm 1140850688): application called MPI_Abort(MPI_COMM_WORLD, 77) - process 6724 [2921]PETSC ERROR: --------------------- Error Message ------------------------------------ [2921]PETSC ERROR: Petsc has generated inconsistent data! [2921]PETSC ERROR: Negative MPI source: stash->nrecvs=15 i=3 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=3825270! [2921]PETSC ERROR: ------------------------------------------------------------------------ [2921]PETSC ERROR: Petsc Release Version 3.4.1, unknown [2921]PETSC ERROR: See docs/changes/index.html for recent updates. [2921]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [2921]PETSC ERROR: See docs/index.html for manual pages. [2921]PETSC ERROR: ------------------------------------------------------------------------ [2921]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug named ys0270 by fandek Sun Jun 23 02:58:23 2013 [2921]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib [2921]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 [2921]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=yes [2921]PETSC ERROR: ------------------------------------------------------------------------ [2921]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /src/mat/utilsmatstash.c [2921]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c [2921]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c [2921]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp [2921]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp [2921]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp [2921]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp [2921]PETSC ERROR: main() line 71 in linearElasticity3d.cpp : On Fri, Jun 21, 2013 at 4:33 AM, Jed Brown wrote: > Fande Kong writes: > > > The code works well with less cores. And It also works well with > > petsc-3.3-p7. But it does not work with petsc-3.4.1. Thus, If you can > check > > the differences between petsc-3.3-p7 and petsc-3.4.1, you can figure out > > the reason. > > That is one way to start debugging, but there are no changes to the core > MatStash code, and many, many changes to PETSc in total. The relevant > snippet of code is here: > > if (stash->reproduce) { > i = stash->reproduce_count++; > ierr = MPI_Wait(stash->recv_waits+i,&recv_status);CHKERRQ(ierr); > } else { > ierr = > MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);CHKERRQ(ierr); > } > if (recv_status.MPI_SOURCE < 0) > SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source!"); > > So MPI returns correctly (stash->reproduce will be FALSE unless you > changed it). You could change the line above to the following: > > if (recv_status.MPI_SOURCE < 0) > SETERRQ5(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source: > stash->nrecvs=%D i=%d MPI_SOURCE=%d MPI_TAG=%d MPI_ERROR=%d", > > stash->nrecvs,i,recv_status.MPI_SOURCE,recv_status.MPI_TAG,recv_status.MPI_ERROR); > > > It would help to debug --with-debugging=1, so that more checks for > corrupt data are performed. You can still make the compiler optimize if > it takes a long time to reach the error condition. > -- Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Jun 23 11:27:44 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 23 Jun 2013 11:27:44 -0500 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> Message-ID: What kind of computer system are you running? What MPI does it use? These values are nonsense MPI_SOURCE=-32766 MPI_TAG=-32766 Is it possible to run the code with valgrind? Any chance of running the code with a different compiler? Barry On Jun 23, 2013, at 4:12 AM, Fande Kong wrote: > Thanks Jed, > > I added your code into the petsc. I run my code with 10240 cores. I got the following error messages: > > [6724]PETSC ERROR: --------------------- Error Message ------------------------------------ > [6724]PETSC ERROR: Petsc has generated inconsistent data! > [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892! > [6724]PETSC ERROR: ------------------------------------------------------------------------ > [6724]PETSC ERROR: Petsc Release Version 3.4.1, unknown > [6724]PETSC ERROR: See docs/changes/index.html for recent updates. > [6724]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [6724]PETSC ERROR: See docs/index.html for manual pages. > [6724]PETSC ERROR: ------------------------------------------------------------------------ > [6724]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug named ys4350 by fandek Sun Jun 23 02:58:23 2013 > [6724]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib > [6724]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 > [6724]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --d > ownload-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=yes > [6724]PETSC ERROR: ------------------------------------------------------------------------ > [6724]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /src/mat/utilsmatstash.c > [6724]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c > [6724]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c > [6724]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp > [6724]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp > [6724]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > [6724]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp > [6724]PETSC ERROR: main() line 71 in linearElasticity3d.cpp > Abort(77) on node 6724 (rank 6724 in comm 1140850688): application called MPI_Abort(MPI_COMM_WORLD, 77) - process 6724 > [2921]PETSC ERROR: --------------------- Error Message ------------------------------------ > [2921]PETSC ERROR: Petsc has generated inconsistent data! > [2921]PETSC ERROR: Negative MPI source: stash->nrecvs=15 i=3 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=3825270! > [2921]PETSC ERROR: ------------------------------------------------------------------------ > [2921]PETSC ERROR: Petsc Release Version 3.4.1, unknown > [2921]PETSC ERROR: See docs/changes/index.html for recent updates. > [2921]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [2921]PETSC ERROR: See docs/index.html for manual pages. > [2921]PETSC ERROR: ------------------------------------------------------------------------ > [2921]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug named ys0270 by fandek Sun Jun 23 02:58:23 2013 > [2921]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib > [2921]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 > [2921]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=yes > [2921]PETSC ERROR: ------------------------------------------------------------------------ > [2921]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /src/mat/utilsmatstash.c > [2921]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c > [2921]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c > [2921]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp > [2921]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp > [2921]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > [2921]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp > [2921]PETSC ERROR: main() line 71 in linearElasticity3d.cpp > : > > On Fri, Jun 21, 2013 at 4:33 AM, Jed Brown wrote: > Fande Kong writes: > > > The code works well with less cores. And It also works well with > > petsc-3.3-p7. But it does not work with petsc-3.4.1. Thus, If you can check > > the differences between petsc-3.3-p7 and petsc-3.4.1, you can figure out > > the reason. > > That is one way to start debugging, but there are no changes to the core > MatStash code, and many, many changes to PETSc in total. The relevant > snippet of code is here: > > if (stash->reproduce) { > i = stash->reproduce_count++; > ierr = MPI_Wait(stash->recv_waits+i,&recv_status);CHKERRQ(ierr); > } else { > ierr = MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);CHKERRQ(ierr); > } > if (recv_status.MPI_SOURCE < 0) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source!"); > > So MPI returns correctly (stash->reproduce will be FALSE unless you > changed it). You could change the line above to the following: > > if (recv_status.MPI_SOURCE < 0) SETERRQ5(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source: stash->nrecvs=%D i=%d MPI_SOURCE=%d MPI_TAG=%d MPI_ERROR=%d", > stash->nrecvs,i,recv_status.MPI_SOURCE,recv_status.MPI_TAG,recv_status.MPI_ERROR); > > > It would help to debug --with-debugging=1, so that more checks for > corrupt data are performed. You can still make the compiler optimize if > it takes a long time to reach the error condition. > > > > -- > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences From jedbrown at mcs.anl.gov Sun Jun 23 12:03:30 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 23 Jun 2013 12:03:30 -0500 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> Message-ID: <87zjugwyhp.fsf@mcs.anl.gov> Barry Smith writes: > What kind of computer system are you running? What MPI does it use? These values are nonsense MPI_SOURCE=-32766 MPI_TAG=-32766 From configure.log, this is Intel MPI. Can you ask their support what this error condition is supposed to mean? It's not clear to me that MPI_SOURCE or MPI_TAG contain any meaningful information (though it could be indicative of an internal overflow), but this value of MPI_ERROR should mean something. > Is it possible to run the code with valgrind? > > Any chance of running the code with a different compiler? > > Barry > > > > On Jun 23, 2013, at 4:12 AM, Fande Kong wrote: > >> Thanks Jed, >> >> I added your code into the petsc. I run my code with 10240 cores. I got the following error messages: >> >> [6724]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [6724]PETSC ERROR: Petsc has generated inconsistent data! >> [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892! >> [6724]PETSC ERROR: ------------------------------------------------------------------------ >> [6724]PETSC ERROR: Petsc Release Version 3.4.1, unknown >> [6724]PETSC ERROR: See docs/changes/index.html for recent updates. >> [6724]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [6724]PETSC ERROR: See docs/index.html for manual pages. >> [6724]PETSC ERROR: ------------------------------------------------------------------------ >> [6724]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug named ys4350 by fandek Sun Jun 23 02:58:23 2013 >> [6724]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib >> [6724]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 >> [6724]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --d >> ownload-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=yes >> [6724]PETSC ERROR: ------------------------------------------------------------------------ >> [6724]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /src/mat/utilsmatstash.c >> [6724]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c >> [6724]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c >> [6724]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp >> [6724]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp >> [6724]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp >> [6724]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp >> [6724]PETSC ERROR: main() line 71 in linearElasticity3d.cpp >> Abort(77) on node 6724 (rank 6724 in comm 1140850688): application called MPI_Abort(MPI_COMM_WORLD, 77) - process 6724 >> [2921]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [2921]PETSC ERROR: Petsc has generated inconsistent data! >> [2921]PETSC ERROR: Negative MPI source: stash->nrecvs=15 i=3 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=3825270! >> [2921]PETSC ERROR: ------------------------------------------------------------------------ >> [2921]PETSC ERROR: Petsc Release Version 3.4.1, unknown >> [2921]PETSC ERROR: See docs/changes/index.html for recent updates. >> [2921]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [2921]PETSC ERROR: See docs/index.html for manual pages. >> [2921]PETSC ERROR: ------------------------------------------------------------------------ >> [2921]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug named ys0270 by fandek Sun Jun 23 02:58:23 2013 >> [2921]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib >> [2921]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 >> [2921]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=yes >> [2921]PETSC ERROR: ------------------------------------------------------------------------ >> [2921]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /src/mat/utilsmatstash.c >> [2921]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c >> [2921]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c >> [2921]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp >> [2921]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp >> [2921]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp >> [2921]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp >> [2921]PETSC ERROR: main() line 71 in linearElasticity3d.cpp >> : >> >> On Fri, Jun 21, 2013 at 4:33 AM, Jed Brown wrote: >> Fande Kong writes: >> >> > The code works well with less cores. And It also works well with >> > petsc-3.3-p7. But it does not work with petsc-3.4.1. Thus, If you can check >> > the differences between petsc-3.3-p7 and petsc-3.4.1, you can figure out >> > the reason. >> >> That is one way to start debugging, but there are no changes to the core >> MatStash code, and many, many changes to PETSc in total. The relevant >> snippet of code is here: >> >> if (stash->reproduce) { >> i = stash->reproduce_count++; >> ierr = MPI_Wait(stash->recv_waits+i,&recv_status);CHKERRQ(ierr); >> } else { >> ierr = MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);CHKERRQ(ierr); >> } >> if (recv_status.MPI_SOURCE < 0) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source!"); >> >> So MPI returns correctly (stash->reproduce will be FALSE unless you >> changed it). You could change the line above to the following: >> >> if (recv_status.MPI_SOURCE < 0) SETERRQ5(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source: stash->nrecvs=%D i=%d MPI_SOURCE=%d MPI_TAG=%d MPI_ERROR=%d", >> stash->nrecvs,i,recv_status.MPI_SOURCE,recv_status.MPI_TAG,recv_status.MPI_ERROR); >> >> >> It would help to debug --with-debugging=1, so that more checks for >> corrupt data are performed. You can still make the compiler optimize if >> it takes a long time to reach the error condition. >> >> >> >> -- >> Fande Kong >> ShenZhen Institutes of Advanced Technology >> Chinese Academy of Sciences -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From rupp at mcs.anl.gov Sun Jun 23 13:37:18 2013 From: rupp at mcs.anl.gov (Karl Rupp) Date: Sun, 23 Jun 2013 13:37:18 -0500 Subject: [petsc-users] OpenFOAM and PETSc In-Reply-To: <002f01ce6eb2$d949ce90$8bdd6bb0$@rnet-tech.com> References: <002f01ce6eb2$d949ce90$8bdd6bb0$@rnet-tech.com> Message-ID: <51C7405E.7090501@mcs.anl.gov> Hi Chekuri, unfortunately I don't have a direct answer for you, as we cannot keep track of the details of all packages using PETSc. I suggest you give the usual OpenFOAM support channels a try, for example here: http://www.cfd-online.com/Forums/openfoam/ Best regards, Karli On 06/21/2013 02:09 PM, Chekuri Choudary wrote: > OpenFOAM is mentioned in the following slides on PETSc. What exactly is > the relationship between OpenFOAM and PETSc? > > http://www.mcs.anl.gov/petsc/documentation/tutorials/ACTSTutorial2012.pdf > > How well is PETSc integrated with OpenFOAM? Is it possible to use most > of the linear solvers and data types in PETSc for OpenFOAM based > applications using the FoamedOver interface mentioned in the link > below?(http://oversetgridsymposium.org/assets/pdf/presentations/1_2/Boger_OversetSymposium2010.pdf) > > Thanks > From fd.kong at siat.ac.cn Sun Jun 23 21:14:35 2013 From: fd.kong at siat.ac.cn (Fande Kong) Date: Mon, 24 Jun 2013 10:14:35 +0800 Subject: [petsc-users] How to understand these error messages In-Reply-To: <87zjugwyhp.fsf@mcs.anl.gov> References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> Message-ID: Thanks Barry, Thanks Jed, The computer I am using is Yellowstone http://en.wikipedia.org/wiki/Yellowstone_(supercomputer), or http://www2.cisl.ucar.edu/resources/yellowstone. The compiler is intel compiler. The mpi is IBM mpi which is a part of IBM PE. With less unknowns (about 5 \times 10^7), the code can correctly run. With unknowns (4 \times 10^8), the code produced the error messages. But with so large unknowns (4 \times 10^8), the code can also run with less cores. This is very strange. When I switch to gnu compiler, I can not install petsc, I got the following errors: ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Downloaded exodusii could not be used. Please check install in /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu ******************************************************************************* File "./config/configure.py", line 293, in petsc_configure framework.configure(out = sys.stdout) File "/glade/p/work/fandek/petsc/config/BuildSystem/config/framework.py", line 933, in configure child.configure() File "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line 556, in configure self.executeTest(self.configureLibrary) File "/glade/p/work/fandek/petsc/config/BuildSystem/config/base.py", line 115, in executeTest ret = apply(test, args,kargs) File "/glade/p/work/fandek/petsc/config/BuildSystem/config/packages/exodusii.py", line 36, in configureLibrary config.package.Package.configureLibrary(self) File "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line 484, in configureLibrary for location, directory, lib, incl in self.generateGuesses(): File "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line 238, in generateGuesses raise RuntimeError('Downloaded '+self.package+' could not be used. Please check install in '+d+'\n') The configure.log is attached. Regards, On Mon, Jun 24, 2013 at 1:03 AM, Jed Brown wrote: > Barry Smith writes: > > > What kind of computer system are you running? What MPI does it use? > These values are nonsense MPI_SOURCE=-32766 MPI_TAG=-32766 > > From configure.log, this is Intel MPI. Can you ask their support what > this error condition is supposed to mean? It's not clear to me that > MPI_SOURCE or MPI_TAG contain any meaningful information (though it > could be indicative of an internal overflow), but this value of > MPI_ERROR should mean something. > > > Is it possible to run the code with valgrind? > > > > Any chance of running the code with a different compiler? > > > > Barry > > > > > > > > On Jun 23, 2013, at 4:12 AM, Fande Kong wrote: > > > >> Thanks Jed, > >> > >> I added your code into the petsc. I run my code with 10240 cores. I got > the following error messages: > >> > >> [6724]PETSC ERROR: --------------------- Error Message > ------------------------------------ > >> [6724]PETSC ERROR: Petsc has generated inconsistent data! > >> [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11 > MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892! > >> [6724]PETSC ERROR: > ------------------------------------------------------------------------ > >> [6724]PETSC ERROR: Petsc Release Version 3.4.1, unknown > >> [6724]PETSC ERROR: See docs/changes/index.html for recent updates. > >> [6724]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >> [6724]PETSC ERROR: See docs/index.html for manual pages. > >> [6724]PETSC ERROR: > ------------------------------------------------------------------------ > >> [6724]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug named > ys4350 by fandek Sun Jun 23 02:58:23 2013 > >> [6724]PETSC ERROR: Libraries linked from > /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib > >> [6724]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 > >> [6724]PETSC ERROR: Configure options --with-valgrind=1 > --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 > --download-f-blas-lapack=1 --with-mpi=1 --d > >> ownload-parmetis=1 --download-metis=1 --with-64-bit-indices=1 > --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 > --download-hdf5=1 --with-debugging=yes > >> [6724]PETSC ERROR: > ------------------------------------------------------------------------ > >> [6724]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in > /src/mat/utilsmatstash.c > >> [6724]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in > /src/mat/impls/aij/mpimpiaij.c > >> [6724]PETSC ERROR: MatAssemblyEnd() line 4939 in > /src/mat/interfacematrix.c > >> [6724]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in > meshreorder.cpp > >> [6724]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in > meshreorder.cpp > >> [6724]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > >> [6724]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp > >> [6724]PETSC ERROR: main() line 71 in linearElasticity3d.cpp > >> Abort(77) on node 6724 (rank 6724 in comm 1140850688): application > called MPI_Abort(MPI_COMM_WORLD, 77) - process 6724 > >> [2921]PETSC ERROR: --------------------- Error Message > ------------------------------------ > >> [2921]PETSC ERROR: Petsc has generated inconsistent data! > >> [2921]PETSC ERROR: Negative MPI source: stash->nrecvs=15 i=3 > MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=3825270! > >> [2921]PETSC ERROR: > ------------------------------------------------------------------------ > >> [2921]PETSC ERROR: Petsc Release Version 3.4.1, unknown > >> [2921]PETSC ERROR: See docs/changes/index.html for recent updates. > >> [2921]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >> [2921]PETSC ERROR: See docs/index.html for manual pages. > >> [2921]PETSC ERROR: > ------------------------------------------------------------------------ > >> [2921]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug named > ys0270 by fandek Sun Jun 23 02:58:23 2013 > >> [2921]PETSC ERROR: Libraries linked from > /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib > >> [2921]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 > >> [2921]PETSC ERROR: Configure options --with-valgrind=1 > --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 > --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 > --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 > --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 > --with-debugging=yes > >> [2921]PETSC ERROR: > ------------------------------------------------------------------------ > >> [2921]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in > /src/mat/utilsmatstash.c > >> [2921]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in > /src/mat/impls/aij/mpimpiaij.c > >> [2921]PETSC ERROR: MatAssemblyEnd() line 4939 in > /src/mat/interfacematrix.c > >> [2921]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in > meshreorder.cpp > >> [2921]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in > meshreorder.cpp > >> [2921]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > >> [2921]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp > >> [2921]PETSC ERROR: main() line 71 in linearElasticity3d.cpp > >> : > >> > >> On Fri, Jun 21, 2013 at 4:33 AM, Jed Brown > wrote: > >> Fande Kong writes: > >> > >> > The code works well with less cores. And It also works well with > >> > petsc-3.3-p7. But it does not work with petsc-3.4.1. Thus, If you can > check > >> > the differences between petsc-3.3-p7 and petsc-3.4.1, you can figure > out > >> > the reason. > >> > >> That is one way to start debugging, but there are no changes to the core > >> MatStash code, and many, many changes to PETSc in total. The relevant > >> snippet of code is here: > >> > >> if (stash->reproduce) { > >> i = stash->reproduce_count++; > >> ierr = MPI_Wait(stash->recv_waits+i,&recv_status);CHKERRQ(ierr); > >> } else { > >> ierr = > MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);CHKERRQ(ierr); > >> } > >> if (recv_status.MPI_SOURCE < 0) > SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source!"); > >> > >> So MPI returns correctly (stash->reproduce will be FALSE unless you > >> changed it). You could change the line above to the following: > >> > >> if (recv_status.MPI_SOURCE < 0) > SETERRQ5(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source: > stash->nrecvs=%D i=%d MPI_SOURCE=%d MPI_TAG=%d MPI_ERROR=%d", > >> > stash->nrecvs,i,recv_status.MPI_SOURCE,recv_status.MPI_TAG,recv_status.MPI_ERROR); > >> > >> > >> It would help to debug --with-debugging=1, so that more checks for > >> corrupt data are performed. You can still make the compiler optimize if > >> it takes a long time to reach the error condition. > >> > >> > >> > >> -- > >> Fande Kong > >> ShenZhen Institutes of Advanced Technology > >> Chinese Academy of Sciences > -- Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.zip Type: application/zip Size: 159246 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Sun Jun 23 22:35:05 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 23 Jun 2013 21:35:05 -0600 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> Message-ID: <87ip14w592.fsf@mcs.anl.gov> Fande Kong writes: > Thanks Barry, > Thanks Jed, > > The computer I am using is Yellowstone > http://en.wikipedia.org/wiki/Yellowstone_(supercomputer), or > http://www2.cisl.ucar.edu/resources/yellowstone. The compiler is intel > compiler. The mpi is IBM mpi which is a part of IBM PE. Can you talk to IBM support about the error condition (MPI_SOURCE, MPI_TAG, MPI_ERROR)? > With less unknowns (about 5 \times 10^7), the code can correctly run. With > unknowns (4 \times 10^8), the code produced the error messages. But with > so large unknowns (4 \times 10^8), the code can also run with less cores. > This is very strange. Does the crash occur on the first iteration or later? > When I switch to gnu compiler, I can not install petsc, I got the following > errors: Looks like an environment problem: Possible ERROR while running linker: /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu/lib/libexodus.a(ex_err.o): In function `ex_err': ex_err.c:(.text+0xbc): undefined reference to `__intel_sse2_strcpy' ex_err.c:(.text+0xe3): undefined reference to `__intel_sse2_strcpy' /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu/lib/libexodus.a(ex_utils.o): In function `ex_put_names_internal': ex_utils.c:(.text+0x247): undefined reference to `__intel_sse2_strlen' ex_utils.c:(.text+0x292): undefined reference to `__intel_sse2_strlen' Did you load the wrong modules? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Jun 23 23:05:32 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 23 Jun 2013 23:05:32 -0500 Subject: [petsc-users] How to understand these error messages In-Reply-To: <87ip14w592.fsf@mcs.anl.gov> References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> <87ip14w592.fsf@mcs.anl.gov> Message-ID: <36B8A29F-D548-4D8D-B7A7-C39AC5A67430@mcs.anl.gov> On Jun 23, 2013, at 10:35 PM, Jed Brown wrote: > Fande Kong writes: > >> When I switch to gnu compiler, I can not install petsc, I got the following >> errors: > > Looks like an environment problem: > > Possible ERROR while running linker: /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu/lib/libexodus.a(ex_err.o): In function `ex_err': > ex_err.c:(.text+0xbc): undefined reference to `__intel_sse2_strcpy' > ex_err.c:(.text+0xe3): undefined reference to `__intel_sse2_strcpy' > /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu/lib/libexodus.a(ex_utils.o): In function `ex_put_names_internal': > ex_utils.c:(.text+0x247): undefined reference to `__intel_sse2_strlen' > ex_utils.c:(.text+0x292): undefined reference to `__intel_sse2_strlen' > > Did you load the wrong modules? Fande, The error you got listed below can happen when some of the code is compiled with the Intel compilers (which insert those funky symbols) but then linked with GNU which doesn't know about those symbols since they are hidden away inside the Intel compiler libraries. ex_err.c:(.text+0xbc): undefined reference to `__intel_sse2_strcpy' ex_err.c:(.text+0xe3): undefined reference to `__intel_sse2_strcpy' /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu/lib/libexodus.a(ex_utils.o): In function `ex_put_names_internal': ex_utils.c:(.text+0x247): undefined reference to `__intel_sse2_strlen' ex_utils.c:(.text+0x292): undefined reference to `__intel_sse2_strlen' Looking closer at the configure output I see sh: cd /glade/p/work/fandek/petsc/externalpackages/exodus-5.24/exodus && make -f Makefile.standalone libexodus.a RANLIB="/usr/bin/ranlib" AR="/usr/bin/ar cr" NETCDF="/glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu" CC="mpicc" CCOPTIONS=" -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O -DADDC_ " FC="mpif90" F77OPTIONS=" -fPIC -Wall -Wno-unused-variable -Wno-unused-dummy-argument -O " Executing: cd /glade/p/work/fandek/petsc/externalpackages/exodus-5.24/exodus && make -f Makefile.standalone libexodus.a RANLIB="/usr/bin/ranlib" AR="/usr/bin/ar cr" NETCDF="/glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu" CC="mpicc" CCOPTIONS=" -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O -DADDC_ " FC="mpif90" F77OPTIONS=" -fPIC -Wall -Wno-unused-variable -Wno-unused-dummy-argument -O " sh: echo "making exodus in cbind/src" making exodus in cbind/src (cd cbind/src && make -f Makefile.standalone "AR=/usr/bin/ar cr" "CC=mpicc" "CFLAGS=-O2 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O -DADDC_ -DVERBOSE -I../../cbind/include -I../../forbind/include -I/glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu/include " "RANLIB=/usr/bin/ranlib" libexodus.a) make[1]: Entering directory `/glade/p/work/fandek/petsc/externalpackages/exodus-5.24/exodus/cbind/src' make[1]: `libexodus.a' is up to date. make[1]: Leaving directory `/glade/p/work/fandek/petsc/externalpackages/exodus-5.24/exodus/cbind/src' cp cbind/src/libexodus.a . /usr/bin/ranlib libexodus.a it is NOT actually recompiling the libraries so the old library (compiled with Intel) is just reused resulting in the problem. Please do the following, rm -rf petsc/externalpackages/exodus-5.24 and then run the configure again and it should go through. Barryt From bsmith at mcs.anl.gov Sun Jun 23 23:27:55 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 23 Jun 2013 23:27:55 -0500 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> Message-ID: Fande, We've seen trouble before with IBM on large intel systems at scale. From the previous configure.log you sent I see sh: mpicc -show Executing: mpicc -show sh: /ncar/opt/intel/12.1.0.233/composer_xe_2011_sp1.11.339/bin/intel64/icc -I/glade/apps/el6/include -I/glade/apps/el6/usr/include -I/glade/apps/opt/netcdf/4.2/intel/default/include -Wl,-rpath,/ncar/opt/intel/12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64 -Wl,-rpath,/ncar/opt/intel/12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/ia32 -L/glade/apps/el6/usr/lib -L/glade/apps/el6/usr/lib64 -Wl,-rpath,/glade/apps/el6/usr/lib -Wl,-rpath,/glade/apps/el6/usr/lib64 -L/glade/apps/opt/netcdf/4.2/intel/default/lib -lnetcdf_c++4 -lnetcdff -lnetcdf -Wl,-rpath,/glade/apps/opt/netcdf/4.2/intel/default/lib -m64 -D__64BIT__ -Wl,--allow-shlib-undefined -Wl,--enable-new-dtags -Wl,-rpath,/opt/ibmhpc/pe1209/mpich2/intel/lib64 -Wl,-rpath,/ncar/opt/intel/12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64 -I/opt/ibmhpc/pe1209/mpich2/intel/include64 -I/opt/ibmhpc/pe1209/base/include -L/opt/ibmhpc/pe1209/mpich2/intel/lib64 -lmpi -ldl -L/ncar/opt/intel/12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64 -lirc -lpthread -lrt Note the -I/opt/ibmhpc/pe1209/base/include -L/opt/ibmhpc/pe1209/mpich2/intel/lib64 -lmpi which is probably some IBM hack job of some ancient mpich2 Now the page http://www2.cisl.ucar.edu/resources/yellowstone/software/modules-intel-dependent has the modules impi/4.0.3.008 This module loads the Intel MPI Library. See http://software.intel.com/en-us/intel-mpi-library/ for details. impi/4.1.0.030 This module loads the Intel MPI Library. See http://software.intel.com/en-us/intel-mpi-library/ for details. Perhaps you could load those modules with the Intel compilers and avoid the IBM MPI? If that solves the problem then we know the IBM MPI is too blame. We are interested in working with you to determine the problem. Barry On Jun 23, 2013, at 9:14 PM, Fande Kong wrote: > Thanks Barry, > Thanks Jed, > > The computer I am using is Yellowstone http://en.wikipedia.org/wiki/Yellowstone_(supercomputer), or http://www2.cisl.ucar.edu/resources/yellowstone. The compiler is intel compiler. The mpi is IBM mpi which is a part of IBM PE. > > With less unknowns (about 5 \times 10^7), the code can correctly run. With unknowns (4 \times 10^8), the code produced the error messages. But with so large unknowns (4 \times 10^8), the code can also run with less cores. This is very strange. > > When I switch to gnu compiler, I can not install petsc, I got the following errors: > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Downloaded exodusii could not be used. Please check install in /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu > ******************************************************************************* > File "./config/configure.py", line 293, in petsc_configure > framework.configure(out = sys.stdout) > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/framework.py", line 933, in configure > child.configure() > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line 556, in configure > self.executeTest(self.configureLibrary) > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/base.py", line 115, in executeTest > ret = apply(test, args,kargs) > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/packages/exodusii.py", line 36, in configureLibrary > config.package.Package.configureLibrary(self) > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line 484, in configureLibrary > for location, directory, lib, incl in self.generateGuesses(): > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line 238, in generateGuesses > raise RuntimeError('Downloaded '+self.package+' could not be used. Please check install in '+d+'\n') > > > The configure.log is attached. > > Regards, > On Mon, Jun 24, 2013 at 1:03 AM, Jed Brown wrote: > Barry Smith writes: > > > What kind of computer system are you running? What MPI does it use? These values are nonsense MPI_SOURCE=-32766 MPI_TAG=-32766 > > From configure.log, this is Intel MPI. Can you ask their support what > this error condition is supposed to mean? It's not clear to me that > MPI_SOURCE or MPI_TAG contain any meaningful information (though it > could be indicative of an internal overflow), but this value of > MPI_ERROR should mean something. > > > Is it possible to run the code with valgrind? > > > > Any chance of running the code with a different compiler? > > > > Barry > > > > > > > > On Jun 23, 2013, at 4:12 AM, Fande Kong wrote: > > > >> Thanks Jed, > >> > >> I added your code into the petsc. I run my code with 10240 cores. I got the following error messages: > >> > >> [6724]PETSC ERROR: --------------------- Error Message ------------------------------------ > >> [6724]PETSC ERROR: Petsc has generated inconsistent data! > >> [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892! > >> [6724]PETSC ERROR: ------------------------------------------------------------------------ > >> [6724]PETSC ERROR: Petsc Release Version 3.4.1, unknown > >> [6724]PETSC ERROR: See docs/changes/index.html for recent updates. > >> [6724]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >> [6724]PETSC ERROR: See docs/index.html for manual pages. > >> [6724]PETSC ERROR: ------------------------------------------------------------------------ > >> [6724]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug named ys4350 by fandek Sun Jun 23 02:58:23 2013 > >> [6724]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib > >> [6724]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 > >> [6724]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --d > >> ownload-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=yes > >> [6724]PETSC ERROR: ------------------------------------------------------------------------ > >> [6724]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /src/mat/utilsmatstash.c > >> [6724]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c > >> [6724]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c > >> [6724]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp > >> [6724]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp > >> [6724]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > >> [6724]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp > >> [6724]PETSC ERROR: main() line 71 in linearElasticity3d.cpp > >> Abort(77) on node 6724 (rank 6724 in comm 1140850688): application called MPI_Abort(MPI_COMM_WORLD, 77) - process 6724 > >> [2921]PETSC ERROR: --------------------- Error Message ------------------------------------ > >> [2921]PETSC ERROR: Petsc has generated inconsistent data! > >> [2921]PETSC ERROR: Negative MPI source: stash->nrecvs=15 i=3 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=3825270! > >> [2921]PETSC ERROR: ------------------------------------------------------------------------ > >> [2921]PETSC ERROR: Petsc Release Version 3.4.1, unknown > >> [2921]PETSC ERROR: See docs/changes/index.html for recent updates. > >> [2921]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >> [2921]PETSC ERROR: See docs/index.html for manual pages. > >> [2921]PETSC ERROR: ------------------------------------------------------------------------ > >> [2921]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug named ys0270 by fandek Sun Jun 23 02:58:23 2013 > >> [2921]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib > >> [2921]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 > >> [2921]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=yes > >> [2921]PETSC ERROR: ------------------------------------------------------------------------ > >> [2921]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /src/mat/utilsmatstash.c > >> [2921]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c > >> [2921]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c > >> [2921]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp > >> [2921]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp > >> [2921]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > >> [2921]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp > >> [2921]PETSC ERROR: main() line 71 in linearElasticity3d.cpp > >> : > >> > >> On Fri, Jun 21, 2013 at 4:33 AM, Jed Brown wrote: > >> Fande Kong writes: > >> > >> > The code works well with less cores. And It also works well with > >> > petsc-3.3-p7. But it does not work with petsc-3.4.1. Thus, If you can check > >> > the differences between petsc-3.3-p7 and petsc-3.4.1, you can figure out > >> > the reason. > >> > >> That is one way to start debugging, but there are no changes to the core > >> MatStash code, and many, many changes to PETSc in total. The relevant > >> snippet of code is here: > >> > >> if (stash->reproduce) { > >> i = stash->reproduce_count++; > >> ierr = MPI_Wait(stash->recv_waits+i,&recv_status);CHKERRQ(ierr); > >> } else { > >> ierr = MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);CHKERRQ(ierr); > >> } > >> if (recv_status.MPI_SOURCE < 0) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source!"); > >> > >> So MPI returns correctly (stash->reproduce will be FALSE unless you > >> changed it). You could change the line above to the following: > >> > >> if (recv_status.MPI_SOURCE < 0) SETERRQ5(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source: stash->nrecvs=%D i=%d MPI_SOURCE=%d MPI_TAG=%d MPI_ERROR=%d", > >> stash->nrecvs,i,recv_status.MPI_SOURCE,recv_status.MPI_TAG,recv_status.MPI_ERROR); > >> > >> > >> It would help to debug --with-debugging=1, so that more checks for > >> corrupt data are performed. You can still make the compiler optimize if > >> it takes a long time to reach the error condition. > >> > >> > >> > >> -- > >> Fande Kong > >> ShenZhen Institutes of Advanced Technology > >> Chinese Academy of Sciences > > > > -- > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences > From fd.kong at siat.ac.cn Mon Jun 24 00:45:02 2013 From: fd.kong at siat.ac.cn (Fande Kong) Date: Mon, 24 Jun 2013 13:45:02 +0800 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> Message-ID: Thanks Barry? I will try impi. I have another question. In the previous email, you said If I can change to use another compiler. Why I need to change the compiler? On Mon, Jun 24, 2013 at 12:27 PM, Barry Smith wrote: > > Fande, > > We've seen trouble before with IBM on large intel systems at scale. > > From the previous configure.log you sent I see > > sh: mpicc -show > Executing: mpicc -show > sh: /ncar/opt/intel/12.1.0.233/composer_xe_2011_sp1.11.339/bin/intel64/icc -I/glade/apps/el6/include -I/glade/apps/el6/usr/include > -I/glade/apps/opt/netcdf/4.2/intel/default/include > -Wl,-rpath,/ncar/opt/intel/ > 12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64 -Wl,-rpath,/ncar/opt/intel/ > 12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/ia32 -L/glade/apps/el6/usr/lib -L/glade/apps/el6/usr/lib64 > -Wl,-rpath,/glade/apps/el6/usr/lib -Wl,-rpath,/glade/apps/el6/usr/lib64 > -L/glade/apps/opt/netcdf/4.2/intel/default/lib -lnetcdf_c++4 -lnetcdff > -lnetcdf -Wl,-rpath,/glade/apps/opt/netcdf/4.2/intel/default/lib -m64 > -D__64BIT__ -Wl,--allow-shlib-undefined -Wl,--enable-new-dtags > -Wl,-rpath,/opt/ibmhpc/pe1209/mpich2/intel/lib64 -Wl,-rpath,/ncar/opt/intel/ > 12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64-I/opt/ibmhpc/pe1209/mpich2/intel/include64 > -I/opt/ibmhpc/pe1209/base/include -L/opt/ibmhpc/pe1209/mpich2/intel/lib64 > -lmpi -ldl -L/ncar/opt/intel/ > 12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64 -lirc > -lpthread -lrt > > Note the -I/opt/ibmhpc/pe1209/base/include > -L/opt/ibmhpc/pe1209/mpich2/intel/lib64 -lmpi which is probably some IBM > hack job of some ancient mpich2 > > Now the page > http://www2.cisl.ucar.edu/resources/yellowstone/software/modules-intel-dependent has the modules > > > impi/4.0.3.008 This module loads the Intel MPI Library. See > http://software.intel.com/en-us/intel-mpi-library/ for details. > impi/4.1.0.030 This module loads the Intel MPI Library. See > http://software.intel.com/en-us/intel-mpi-library/ for details. > > Perhaps you could load those modules with the Intel compilers and avoid > the IBM MPI? If that solves the problem then we know the IBM MPI is too > blame. We are interested in working with you to determine the problem. > > Barry > > > > > On Jun 23, 2013, at 9:14 PM, Fande Kong wrote: > > > Thanks Barry, > > Thanks Jed, > > > > The computer I am using is Yellowstone > http://en.wikipedia.org/wiki/Yellowstone_(supercomputer), or > http://www2.cisl.ucar.edu/resources/yellowstone. The compiler is intel > compiler. The mpi is IBM mpi which is a part of IBM PE. > > > > With less unknowns (about 5 \times 10^7), the code can correctly run. > With unknowns (4 \times 10^8), the code produced the error messages. But > with so large unknowns (4 \times 10^8), the code can also run with less > cores. This is very strange. > > > > When I switch to gnu compiler, I can not install petsc, I got the > following errors: > > > > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > > > ------------------------------------------------------------------------------- > > Downloaded exodusii could not be used. Please check install in > /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu > > > ******************************************************************************* > > File "./config/configure.py", line 293, in petsc_configure > > framework.configure(out = sys.stdout) > > File > "/glade/p/work/fandek/petsc/config/BuildSystem/config/framework.py", line > 933, in configure > > child.configure() > > File > "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line > 556, in configure > > self.executeTest(self.configureLibrary) > > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/base.py", > line 115, in executeTest > > ret = apply(test, args,kargs) > > File > "/glade/p/work/fandek/petsc/config/BuildSystem/config/packages/exodusii.py", > line 36, in configureLibrary > > config.package.Package.configureLibrary(self) > > File > "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line > 484, in configureLibrary > > for location, directory, lib, incl in self.generateGuesses(): > > File > "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line > 238, in generateGuesses > > raise RuntimeError('Downloaded '+self.package+' could not be used. > Please check install in '+d+'\n') > > > > > > The configure.log is attached. > > > > Regards, > > On Mon, Jun 24, 2013 at 1:03 AM, Jed Brown wrote: > > Barry Smith writes: > > > > > What kind of computer system are you running? What MPI does it use? > These values are nonsense MPI_SOURCE=-32766 MPI_TAG=-32766 > > > > From configure.log, this is Intel MPI. Can you ask their support what > > this error condition is supposed to mean? It's not clear to me that > > MPI_SOURCE or MPI_TAG contain any meaningful information (though it > > could be indicative of an internal overflow), but this value of > > MPI_ERROR should mean something. > > > > > Is it possible to run the code with valgrind? > > > > > > Any chance of running the code with a different compiler? > > > > > > Barry > > > > > > > > > > > > On Jun 23, 2013, at 4:12 AM, Fande Kong wrote: > > > > > >> Thanks Jed, > > >> > > >> I added your code into the petsc. I run my code with 10240 cores. I > got the following error messages: > > >> > > >> [6724]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > >> [6724]PETSC ERROR: Petsc has generated inconsistent data! > > >> [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11 > MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892! > > >> [6724]PETSC ERROR: > ------------------------------------------------------------------------ > > >> [6724]PETSC ERROR: Petsc Release Version 3.4.1, unknown > > >> [6724]PETSC ERROR: See docs/changes/index.html for recent updates. > > >> [6724]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > >> [6724]PETSC ERROR: See docs/index.html for manual pages. > > >> [6724]PETSC ERROR: > ------------------------------------------------------------------------ > > >> [6724]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug > named ys4350 by fandek Sun Jun 23 02:58:23 2013 > > >> [6724]PETSC ERROR: Libraries linked from > /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib > > >> [6724]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 > > >> [6724]PETSC ERROR: Configure options --with-valgrind=1 > --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 > --download-f-blas-lapack=1 --with-mpi=1 --d > > >> ownload-parmetis=1 --download-metis=1 --with-64-bit-indices=1 > --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 > --download-hdf5=1 --with-debugging=yes > > >> [6724]PETSC ERROR: > ------------------------------------------------------------------------ > > >> [6724]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in > /src/mat/utilsmatstash.c > > >> [6724]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in > /src/mat/impls/aij/mpimpiaij.c > > >> [6724]PETSC ERROR: MatAssemblyEnd() line 4939 in > /src/mat/interfacematrix.c > > >> [6724]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in > meshreorder.cpp > > >> [6724]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in > meshreorder.cpp > > >> [6724]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > > >> [6724]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp > > >> [6724]PETSC ERROR: main() line 71 in linearElasticity3d.cpp > > >> Abort(77) on node 6724 (rank 6724 in comm 1140850688): application > called MPI_Abort(MPI_COMM_WORLD, 77) - process 6724 > > >> [2921]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > >> [2921]PETSC ERROR: Petsc has generated inconsistent data! > > >> [2921]PETSC ERROR: Negative MPI source: stash->nrecvs=15 i=3 > MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=3825270! > > >> [2921]PETSC ERROR: > ------------------------------------------------------------------------ > > >> [2921]PETSC ERROR: Petsc Release Version 3.4.1, unknown > > >> [2921]PETSC ERROR: See docs/changes/index.html for recent updates. > > >> [2921]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > >> [2921]PETSC ERROR: See docs/index.html for manual pages. > > >> [2921]PETSC ERROR: > ------------------------------------------------------------------------ > > >> [2921]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug > named ys0270 by fandek Sun Jun 23 02:58:23 2013 > > >> [2921]PETSC ERROR: Libraries linked from > /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib > > >> [2921]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 > > >> [2921]PETSC ERROR: Configure options --with-valgrind=1 > --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 > --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 > --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 > --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 > --with-debugging=yes > > >> [2921]PETSC ERROR: > ------------------------------------------------------------------------ > > >> [2921]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in > /src/mat/utilsmatstash.c > > >> [2921]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in > /src/mat/impls/aij/mpimpiaij.c > > >> [2921]PETSC ERROR: MatAssemblyEnd() line 4939 in > /src/mat/interfacematrix.c > > >> [2921]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in > meshreorder.cpp > > >> [2921]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in > meshreorder.cpp > > >> [2921]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > > >> [2921]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp > > >> [2921]PETSC ERROR: main() line 71 in linearElasticity3d.cpp > > >> : > > >> > > >> On Fri, Jun 21, 2013 at 4:33 AM, Jed Brown > wrote: > > >> Fande Kong writes: > > >> > > >> > The code works well with less cores. And It also works well with > > >> > petsc-3.3-p7. But it does not work with petsc-3.4.1. Thus, If you > can check > > >> > the differences between petsc-3.3-p7 and petsc-3.4.1, you can > figure out > > >> > the reason. > > >> > > >> That is one way to start debugging, but there are no changes to the > core > > >> MatStash code, and many, many changes to PETSc in total. The relevant > > >> snippet of code is here: > > >> > > >> if (stash->reproduce) { > > >> i = stash->reproduce_count++; > > >> ierr = MPI_Wait(stash->recv_waits+i,&recv_status);CHKERRQ(ierr); > > >> } else { > > >> ierr = > MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);CHKERRQ(ierr); > > >> } > > >> if (recv_status.MPI_SOURCE < 0) > SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source!"); > > >> > > >> So MPI returns correctly (stash->reproduce will be FALSE unless you > > >> changed it). You could change the line above to the following: > > >> > > >> if (recv_status.MPI_SOURCE < 0) > SETERRQ5(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source: > stash->nrecvs=%D i=%d MPI_SOURCE=%d MPI_TAG=%d MPI_ERROR=%d", > > >> > stash->nrecvs,i,recv_status.MPI_SOURCE,recv_status.MPI_TAG,recv_status.MPI_ERROR); > > >> > > >> > > >> It would help to debug --with-debugging=1, so that more checks for > > >> corrupt data are performed. You can still make the compiler optimize > if > > >> it takes a long time to reach the error condition. > > >> > > >> > > >> > > >> -- > > >> Fande Kong > > >> ShenZhen Institutes of Advanced Technology > > >> Chinese Academy of Sciences > > > > > > > > -- > > Fande Kong > > ShenZhen Institutes of Advanced Technology > > Chinese Academy of Sciences > > > > > -- Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From fd.kong at siat.ac.cn Mon Jun 24 02:53:58 2013 From: fd.kong at siat.ac.cn (Fande Kong) Date: Mon, 24 Jun 2013 01:53:58 -0600 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> Message-ID: Hi Barry, I switched to gnu compiler. I also got the similar results: [330]PETSC ERROR: --------------------- Error Message ------------------------------------ [330]PETSC ERROR: Petsc has generated inconsistent data! [330]PETSC ERROR: Negative MPI source: stash->nrecvs=27 i=33 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=5243744! [330]PETSC ERROR: ------------------------------------------------------------------------ [330]PETSC ERROR: Petsc Release Version 3.4.1, unknown [330]PETSC ERROR: See docs/changes/index.html for recent updates. [330]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [330]PETSC ERROR: See docs/index.html for manual pages. [330]PETSC ERROR: ------------------------------------------------------------------------ [330]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-opt_gnu named ys0554 by fandek Mon Jun 24 01:42:37 2013 [330]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu/lib [330]PETSC ERROR: Configure run at Mon Jun 24 00:34:40 2013 [330]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=no [330]PETSC ERROR: ------------------------------------------------------------------------ [330]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /src/mat/utilsmatstash.c [330]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c [330]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c [330]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp [330]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp [330]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp [330]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp [330]PETSC ERROR: main() line 71 in linearElasticity3d.cpp Thus, I think that it has nothing to do with the compiler. On Sun, Jun 23, 2013 at 11:45 PM, Fande Kong wrote: > Thanks Barry? > > I will try impi. > > I have another question. In the previous email, you said If I can change > to use another compiler. Why I need to change the compiler? > > > On Mon, Jun 24, 2013 at 12:27 PM, Barry Smith wrote: > >> >> Fande, >> >> We've seen trouble before with IBM on large intel systems at scale. >> >> From the previous configure.log you sent I see >> >> sh: mpicc -show >> Executing: mpicc -show >> sh: /ncar/opt/intel/ >> 12.1.0.233/composer_xe_2011_sp1.11.339/bin/intel64/icc >> -I/glade/apps/el6/include -I/glade/apps/el6/usr/include >> -I/glade/apps/opt/netcdf/4.2/intel/default/include >> -Wl,-rpath,/ncar/opt/intel/ >> 12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64 -Wl,-rpath,/ncar/opt/intel/ >> 12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/ia32 -L/glade/apps/el6/usr/lib -L/glade/apps/el6/usr/lib64 >> -Wl,-rpath,/glade/apps/el6/usr/lib -Wl,-rpath,/glade/apps/el6/usr/lib64 >> -L/glade/apps/opt/netcdf/4.2/intel/default/lib -lnetcdf_c++4 -lnetcdff >> -lnetcdf -Wl,-rpath,/glade/apps/opt/netcdf/4.2/intel/default/lib -m64 >> -D__64BIT__ -Wl,--allow-shlib-undefined -Wl,--enable-new-dtags >> -Wl,-rpath,/opt/ibmhpc/pe1209/mpich2/intel/lib64 -Wl,-rpath,/ncar/opt/intel/ >> 12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64-I/opt/ibmhpc/pe1209/mpich2/intel/include64 >> -I/opt/ibmhpc/pe1209/base/include -L/opt/ibmhpc/pe1209/mpich2/intel/lib64 >> -lmpi -ldl -L/ncar/opt/intel/ >> 12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64 -lirc >> -lpthread -lrt >> >> Note the -I/opt/ibmhpc/pe1209/base/include >> -L/opt/ibmhpc/pe1209/mpich2/intel/lib64 -lmpi which is probably some IBM >> hack job of some ancient mpich2 >> >> Now the page >> http://www2.cisl.ucar.edu/resources/yellowstone/software/modules-intel-dependent has the modules >> >> >> impi/4.0.3.008 This module loads the Intel MPI Library. See >> http://software.intel.com/en-us/intel-mpi-library/ for details. >> impi/4.1.0.030 This module loads the Intel MPI Library. See >> http://software.intel.com/en-us/intel-mpi-library/ for details. >> >> Perhaps you could load those modules with the Intel compilers and avoid >> the IBM MPI? If that solves the problem then we know the IBM MPI is too >> blame. We are interested in working with you to determine the problem. >> >> Barry >> >> >> >> >> On Jun 23, 2013, at 9:14 PM, Fande Kong wrote: >> >> > Thanks Barry, >> > Thanks Jed, >> > >> > The computer I am using is Yellowstone >> http://en.wikipedia.org/wiki/Yellowstone_(supercomputer), or >> http://www2.cisl.ucar.edu/resources/yellowstone. The compiler is >> intel compiler. The mpi is IBM mpi which is a part of IBM PE. >> > >> > With less unknowns (about 5 \times 10^7), the code can correctly run. >> With unknowns (4 \times 10^8), the code produced the error messages. But >> with so large unknowns (4 \times 10^8), the code can also run with less >> cores. This is very strange. >> > >> > When I switch to gnu compiler, I can not install petsc, I got the >> following errors: >> > >> > >> ******************************************************************************* >> > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >> for details): >> > >> ------------------------------------------------------------------------------- >> > Downloaded exodusii could not be used. Please check install in >> /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu >> > >> ******************************************************************************* >> > File "./config/configure.py", line 293, in petsc_configure >> > framework.configure(out = sys.stdout) >> > File >> "/glade/p/work/fandek/petsc/config/BuildSystem/config/framework.py", line >> 933, in configure >> > child.configure() >> > File >> "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line >> 556, in configure >> > self.executeTest(self.configureLibrary) >> > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/base.py", >> line 115, in executeTest >> > ret = apply(test, args,kargs) >> > File >> "/glade/p/work/fandek/petsc/config/BuildSystem/config/packages/exodusii.py", >> line 36, in configureLibrary >> > config.package.Package.configureLibrary(self) >> > File >> "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line >> 484, in configureLibrary >> > for location, directory, lib, incl in self.generateGuesses(): >> > File >> "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line >> 238, in generateGuesses >> > raise RuntimeError('Downloaded '+self.package+' could not be used. >> Please check install in '+d+'\n') >> > >> > >> > The configure.log is attached. >> > >> > Regards, >> > On Mon, Jun 24, 2013 at 1:03 AM, Jed Brown >> wrote: >> > Barry Smith writes: >> > >> > > What kind of computer system are you running? What MPI does it >> use? These values are nonsense MPI_SOURCE=-32766 MPI_TAG=-32766 >> > >> > From configure.log, this is Intel MPI. Can you ask their support what >> > this error condition is supposed to mean? It's not clear to me that >> > MPI_SOURCE or MPI_TAG contain any meaningful information (though it >> > could be indicative of an internal overflow), but this value of >> > MPI_ERROR should mean something. >> > >> > > Is it possible to run the code with valgrind? >> > > >> > > Any chance of running the code with a different compiler? >> > > >> > > Barry >> > > >> > > >> > > >> > > On Jun 23, 2013, at 4:12 AM, Fande Kong wrote: >> > > >> > >> Thanks Jed, >> > >> >> > >> I added your code into the petsc. I run my code with 10240 cores. I >> got the following error messages: >> > >> >> > >> [6724]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> > >> [6724]PETSC ERROR: Petsc has generated inconsistent data! >> > >> [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11 >> MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892! >> > >> [6724]PETSC ERROR: >> ------------------------------------------------------------------------ >> > >> [6724]PETSC ERROR: Petsc Release Version 3.4.1, unknown >> > >> [6724]PETSC ERROR: See docs/changes/index.html for recent updates. >> > >> [6724]PETSC ERROR: See docs/faq.html for hints about trouble >> shooting. >> > >> [6724]PETSC ERROR: See docs/index.html for manual pages. >> > >> [6724]PETSC ERROR: >> ------------------------------------------------------------------------ >> > >> [6724]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug >> named ys4350 by fandek Sun Jun 23 02:58:23 2013 >> > >> [6724]PETSC ERROR: Libraries linked from >> /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib >> > >> [6724]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 >> > >> [6724]PETSC ERROR: Configure options --with-valgrind=1 >> --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 >> --download-f-blas-lapack=1 --with-mpi=1 --d >> > >> ownload-parmetis=1 --download-metis=1 --with-64-bit-indices=1 >> --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 >> --download-hdf5=1 --with-debugging=yes >> > >> [6724]PETSC ERROR: >> ------------------------------------------------------------------------ >> > >> [6724]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in >> /src/mat/utilsmatstash.c >> > >> [6724]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in >> /src/mat/impls/aij/mpimpiaij.c >> > >> [6724]PETSC ERROR: MatAssemblyEnd() line 4939 in >> /src/mat/interfacematrix.c >> > >> [6724]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in >> meshreorder.cpp >> > >> [6724]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in >> meshreorder.cpp >> > >> [6724]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp >> > >> [6724]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp >> > >> [6724]PETSC ERROR: main() line 71 in linearElasticity3d.cpp >> > >> Abort(77) on node 6724 (rank 6724 in comm 1140850688): application >> called MPI_Abort(MPI_COMM_WORLD, 77) - process 6724 >> > >> [2921]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> > >> [2921]PETSC ERROR: Petsc has generated inconsistent data! >> > >> [2921]PETSC ERROR: Negative MPI source: stash->nrecvs=15 i=3 >> MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=3825270! >> > >> [2921]PETSC ERROR: >> ------------------------------------------------------------------------ >> > >> [2921]PETSC ERROR: Petsc Release Version 3.4.1, unknown >> > >> [2921]PETSC ERROR: See docs/changes/index.html for recent updates. >> > >> [2921]PETSC ERROR: See docs/faq.html for hints about trouble >> shooting. >> > >> [2921]PETSC ERROR: See docs/index.html for manual pages. >> > >> [2921]PETSC ERROR: >> ------------------------------------------------------------------------ >> > >> [2921]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug >> named ys0270 by fandek Sun Jun 23 02:58:23 2013 >> > >> [2921]PETSC ERROR: Libraries linked from >> /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib >> > >> [2921]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 >> > >> [2921]PETSC ERROR: Configure options --with-valgrind=1 >> --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 >> --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 >> --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 >> --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 >> --with-debugging=yes >> > >> [2921]PETSC ERROR: >> ------------------------------------------------------------------------ >> > >> [2921]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in >> /src/mat/utilsmatstash.c >> > >> [2921]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in >> /src/mat/impls/aij/mpimpiaij.c >> > >> [2921]PETSC ERROR: MatAssemblyEnd() line 4939 in >> /src/mat/interfacematrix.c >> > >> [2921]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in >> meshreorder.cpp >> > >> [2921]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in >> meshreorder.cpp >> > >> [2921]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp >> > >> [2921]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp >> > >> [2921]PETSC ERROR: main() line 71 in linearElasticity3d.cpp >> > >> : >> > >> >> > >> On Fri, Jun 21, 2013 at 4:33 AM, Jed Brown >> wrote: >> > >> Fande Kong writes: >> > >> >> > >> > The code works well with less cores. And It also works well with >> > >> > petsc-3.3-p7. But it does not work with petsc-3.4.1. Thus, If you >> can check >> > >> > the differences between petsc-3.3-p7 and petsc-3.4.1, you can >> figure out >> > >> > the reason. >> > >> >> > >> That is one way to start debugging, but there are no changes to the >> core >> > >> MatStash code, and many, many changes to PETSc in total. The >> relevant >> > >> snippet of code is here: >> > >> >> > >> if (stash->reproduce) { >> > >> i = stash->reproduce_count++; >> > >> ierr = >> MPI_Wait(stash->recv_waits+i,&recv_status);CHKERRQ(ierr); >> > >> } else { >> > >> ierr = >> MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);CHKERRQ(ierr); >> > >> } >> > >> if (recv_status.MPI_SOURCE < 0) >> SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source!"); >> > >> >> > >> So MPI returns correctly (stash->reproduce will be FALSE unless you >> > >> changed it). You could change the line above to the following: >> > >> >> > >> if (recv_status.MPI_SOURCE < 0) >> SETERRQ5(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source: >> stash->nrecvs=%D i=%d MPI_SOURCE=%d MPI_TAG=%d MPI_ERROR=%d", >> > >> >> stash->nrecvs,i,recv_status.MPI_SOURCE,recv_status.MPI_TAG,recv_status.MPI_ERROR); >> > >> >> > >> >> > >> It would help to debug --with-debugging=1, so that more checks for >> > >> corrupt data are performed. You can still make the compiler >> optimize if >> > >> it takes a long time to reach the error condition. >> > >> >> > >> >> > >> >> > >> -- >> > >> Fande Kong >> > >> ShenZhen Institutes of Advanced Technology >> > >> Chinese Academy of Sciences >> > >> > >> > >> > -- >> > Fande Kong >> > ShenZhen Institutes of Advanced Technology >> > Chinese Academy of Sciences >> > >> >> >> > > > -- > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences > -- Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.lichtner at gmail.com Mon Jun 24 08:33:53 2013 From: peter.lichtner at gmail.com (Peter Lichtner) Date: Mon, 24 Jun 2013 07:33:53 -0600 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> Message-ID: Just in case this helps I use Yellowstone for running PFLOTRAN with both gcc and intel compilers using the developer version of PETSc. My configuration script reads for intel: ./config/configure.py --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpicxx --with-clanguage=c --with-blas-lapack-dir=$BLAS_LAPACK_LIB_DIR --with-shared-libraries=0 --with-debugging=0 --download-hdf5=yes --download-parmetis=yes --download-metis=yes echo $BLAS_LAPACK_LIB_DIR /ncar/opt/intel/12.1.0.233/composer_xe_2013.1.117/mkl module load cmake/2.8.10.2 Intel was a little faster compared to gcc. ...Peter On Jun 24, 2013, at 1:53 AM, Fande Kong wrote: > Hi Barry, > > I switched to gnu compiler. I also got the similar results: > > > [330]PETSC ERROR: --------------------- Error Message ------------------------------------ > [330]PETSC ERROR: Petsc has generated inconsistent data! > [330]PETSC ERROR: Negative MPI source: stash->nrecvs=27 i=33 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=5243744! > [330]PETSC ERROR: ------------------------------------------------------------------------ > [330]PETSC ERROR: Petsc Release Version 3.4.1, unknown > [330]PETSC ERROR: See docs/changes/index.html for recent updates. > [330]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [330]PETSC ERROR: See docs/index.html for manual pages. > [330]PETSC ERROR: ------------------------------------------------------------------------ > [330]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-opt_gnu named ys0554 by fandek Mon Jun 24 01:42:37 2013 > [330]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu/lib > [330]PETSC ERROR: Configure run at Mon Jun 24 00:34:40 2013 > [330]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=no > [330]PETSC ERROR: ------------------------------------------------------------------------ > [330]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /src/mat/utilsmatstash.c > [330]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c > [330]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c > [330]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp > [330]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp > [330]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > [330]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp > [330]PETSC ERROR: main() line 71 in linearElasticity3d.cpp > > > > Thus, I think that it has nothing to do with the compiler. > > > On Sun, Jun 23, 2013 at 11:45 PM, Fande Kong wrote: > Thanks Barry? > > I will try impi. > > I have another question. In the previous email, you said If I can change to use another compiler. Why I need to change the compiler? > > > On Mon, Jun 24, 2013 at 12:27 PM, Barry Smith wrote: > > Fande, > > We've seen trouble before with IBM on large intel systems at scale. > > From the previous configure.log you sent I see > > sh: mpicc -show > Executing: mpicc -show > sh: /ncar/opt/intel/12.1.0.233/composer_xe_2011_sp1.11.339/bin/intel64/icc -I/glade/apps/el6/include -I/glade/apps/el6/usr/include -I/glade/apps/opt/netcdf/4.2/intel/default/include -Wl,-rpath,/ncar/opt/intel/12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64 -Wl,-rpath,/ncar/opt/intel/12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/ia32 -L/glade/apps/el6/usr/lib -L/glade/apps/el6/usr/lib64 -Wl,-rpath,/glade/apps/el6/usr/lib -Wl,-rpath,/glade/apps/el6/usr/lib64 -L/glade/apps/opt/netcdf/4.2/intel/default/lib -lnetcdf_c++4 -lnetcdff -lnetcdf -Wl,-rpath,/glade/apps/opt/netcdf/4.2/intel/default/lib -m64 -D__64BIT__ -Wl,--allow-shlib-undefined -Wl,--enable-new-dtags -Wl,-rpath,/opt/ibmhpc/pe1209/mpich2/intel/lib64 -Wl,-rpath,/ncar/opt/intel/12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64 -I/opt/ibmhpc/pe1209/mpich2/intel/include64 -I/opt/ibmhpc/pe1209/base/include -L/opt/ibmhpc/pe1209/mpich2/intel/lib64 -lmpi -ldl -L/ncar/opt/intel/12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64 -lirc -lpthread -lrt > > Note the -I/opt/ibmhpc/pe1209/base/include -L/opt/ibmhpc/pe1209/mpich2/intel/lib64 -lmpi which is probably some IBM hack job of some ancient mpich2 > > Now the page http://www2.cisl.ucar.edu/resources/yellowstone/software/modules-intel-dependent has the modules > > > impi/4.0.3.008 This module loads the Intel MPI Library. See http://software.intel.com/en-us/intel-mpi-library/ for details. > impi/4.1.0.030 This module loads the Intel MPI Library. See http://software.intel.com/en-us/intel-mpi-library/ for details. > > Perhaps you could load those modules with the Intel compilers and avoid the IBM MPI? If that solves the problem then we know the IBM MPI is too blame. We are interested in working with you to determine the problem. > > Barry > > > > > On Jun 23, 2013, at 9:14 PM, Fande Kong wrote: > > > Thanks Barry, > > Thanks Jed, > > > > The computer I am using is Yellowstone http://en.wikipedia.org/wiki/Yellowstone_(supercomputer), or http://www2.cisl.ucar.edu/resources/yellowstone. The compiler is intel compiler. The mpi is IBM mpi which is a part of IBM PE. > > > > With less unknowns (about 5 \times 10^7), the code can correctly run. With unknowns (4 \times 10^8), the code produced the error messages. But with so large unknowns (4 \times 10^8), the code can also run with less cores. This is very strange. > > > > When I switch to gnu compiler, I can not install petsc, I got the following errors: > > > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > > ------------------------------------------------------------------------------- > > Downloaded exodusii could not be used. Please check install in /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu > > ******************************************************************************* > > File "./config/configure.py", line 293, in petsc_configure > > framework.configure(out = sys.stdout) > > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/framework.py", line 933, in configure > > child.configure() > > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line 556, in configure > > self.executeTest(self.configureLibrary) > > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/base.py", line 115, in executeTest > > ret = apply(test, args,kargs) > > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/packages/exodusii.py", line 36, in configureLibrary > > config.package.Package.configureLibrary(self) > > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line 484, in configureLibrary > > for location, directory, lib, incl in self.generateGuesses(): > > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line 238, in generateGuesses > > raise RuntimeError('Downloaded '+self.package+' could not be used. Please check install in '+d+'\n') > > > > > > The configure.log is attached. > > > > Regards, > > On Mon, Jun 24, 2013 at 1:03 AM, Jed Brown wrote: > > Barry Smith writes: > > > > > What kind of computer system are you running? What MPI does it use? These values are nonsense MPI_SOURCE=-32766 MPI_TAG=-32766 > > > > From configure.log, this is Intel MPI. Can you ask their support what > > this error condition is supposed to mean? It's not clear to me that > > MPI_SOURCE or MPI_TAG contain any meaningful information (though it > > could be indicative of an internal overflow), but this value of > > MPI_ERROR should mean something. > > > > > Is it possible to run the code with valgrind? > > > > > > Any chance of running the code with a different compiler? > > > > > > Barry > > > > > > > > > > > > On Jun 23, 2013, at 4:12 AM, Fande Kong wrote: > > > > > >> Thanks Jed, > > >> > > >> I added your code into the petsc. I run my code with 10240 cores. I got the following error messages: > > >> > > >> [6724]PETSC ERROR: --------------------- Error Message ------------------------------------ > > >> [6724]PETSC ERROR: Petsc has generated inconsistent data! > > >> [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892! > > >> [6724]PETSC ERROR: ------------------------------------------------------------------------ > > >> [6724]PETSC ERROR: Petsc Release Version 3.4.1, unknown > > >> [6724]PETSC ERROR: See docs/changes/index.html for recent updates. > > >> [6724]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > >> [6724]PETSC ERROR: See docs/index.html for manual pages. > > >> [6724]PETSC ERROR: ------------------------------------------------------------------------ > > >> [6724]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug named ys4350 by fandek Sun Jun 23 02:58:23 2013 > > >> [6724]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib > > >> [6724]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 > > >> [6724]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --d > > >> ownload-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=yes > > >> [6724]PETSC ERROR: ------------------------------------------------------------------------ > > >> [6724]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /src/mat/utilsmatstash.c > > >> [6724]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c > > >> [6724]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c > > >> [6724]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp > > >> [6724]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp > > >> [6724]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > > >> [6724]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp > > >> [6724]PETSC ERROR: main() line 71 in linearElasticity3d.cpp > > >> Abort(77) on node 6724 (rank 6724 in comm 1140850688): application called MPI_Abort(MPI_COMM_WORLD, 77) - process 6724 > > >> [2921]PETSC ERROR: --------------------- Error Message ------------------------------------ > > >> [2921]PETSC ERROR: Petsc has generated inconsistent data! > > >> [2921]PETSC ERROR: Negative MPI source: stash->nrecvs=15 i=3 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=3825270! > > >> [2921]PETSC ERROR: ------------------------------------------------------------------------ > > >> [2921]PETSC ERROR: Petsc Release Version 3.4.1, unknown > > >> [2921]PETSC ERROR: See docs/changes/index.html for recent updates. > > >> [2921]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > >> [2921]PETSC ERROR: See docs/index.html for manual pages. > > >> [2921]PETSC ERROR: ------------------------------------------------------------------------ > > >> [2921]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug named ys0270 by fandek Sun Jun 23 02:58:23 2013 > > >> [2921]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib > > >> [2921]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 > > >> [2921]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=yes > > >> [2921]PETSC ERROR: ------------------------------------------------------------------------ > > >> [2921]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /src/mat/utilsmatstash.c > > >> [2921]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /src/mat/impls/aij/mpimpiaij.c > > >> [2921]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c > > >> [2921]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp > > >> [2921]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp > > >> [2921]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > > >> [2921]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp > > >> [2921]PETSC ERROR: main() line 71 in linearElasticity3d.cpp > > >> : > > >> > > >> On Fri, Jun 21, 2013 at 4:33 AM, Jed Brown wrote: > > >> Fande Kong writes: > > >> > > >> > The code works well with less cores. And It also works well with > > >> > petsc-3.3-p7. But it does not work with petsc-3.4.1. Thus, If you can check > > >> > the differences between petsc-3.3-p7 and petsc-3.4.1, you can figure out > > >> > the reason. > > >> > > >> That is one way to start debugging, but there are no changes to the core > > >> MatStash code, and many, many changes to PETSc in total. The relevant > > >> snippet of code is here: > > >> > > >> if (stash->reproduce) { > > >> i = stash->reproduce_count++; > > >> ierr = MPI_Wait(stash->recv_waits+i,&recv_status);CHKERRQ(ierr); > > >> } else { > > >> ierr = MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);CHKERRQ(ierr); > > >> } > > >> if (recv_status.MPI_SOURCE < 0) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source!"); > > >> > > >> So MPI returns correctly (stash->reproduce will be FALSE unless you > > >> changed it). You could change the line above to the following: > > >> > > >> if (recv_status.MPI_SOURCE < 0) SETERRQ5(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source: stash->nrecvs=%D i=%d MPI_SOURCE=%d MPI_TAG=%d MPI_ERROR=%d", > > >> stash->nrecvs,i,recv_status.MPI_SOURCE,recv_status.MPI_TAG,recv_status.MPI_ERROR); > > >> > > >> > > >> It would help to debug --with-debugging=1, so that more checks for > > >> corrupt data are performed. You can still make the compiler optimize if > > >> it takes a long time to reach the error condition. > > >> > > >> > > >> > > >> -- > > >> Fande Kong > > >> ShenZhen Institutes of Advanced Technology > > >> Chinese Academy of Sciences > > > > > > > > -- > > Fande Kong > > ShenZhen Institutes of Advanced Technology > > Chinese Academy of Sciences > > > > > > > > -- > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences > > > > -- > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences ________________ Peter Lichtner Santa Fe, NM 87507 (505) 692-4029 (c) OFM Research/LANL Guest Scientist -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpovolot at purdue.edu Mon Jun 24 17:01:38 2013 From: mpovolot at purdue.edu (Michael Povolotskyi) Date: Mon, 24 Jun 2013 18:01:38 -0400 Subject: [petsc-users] question about extracting numbers from a matrix Message-ID: <51C8C1C2.6090703@purdue.edu> Hello everybody, I have a sparse matrix defined on 1 MPI process. I want to extract all entries from it (including zeros) and put them into an array to use as an argument for some lapack functions. What is the most optimal way of doing this? Thank you, Michael. -- Michael Povolotskyi, PhD Research Assistant Professor Network for Computational Nanotechnology 207 S Martin Jischke Drive Purdue University, DLR, room 441-10 West Lafayette, Indiana 47907 phone: +1-765-494-9396 fax: +1-765-496-6026 From knepley at gmail.com Mon Jun 24 17:06:38 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Jun 2013 00:06:38 +0200 Subject: [petsc-users] question about extracting numbers from a matrix In-Reply-To: <51C8C1C2.6090703@purdue.edu> References: <51C8C1C2.6090703@purdue.edu> Message-ID: On Tue, Jun 25, 2013 at 12:01 AM, Michael Povolotskyi wrote: > Hello everybody, > I have a sparse matrix defined on 1 MPI process. > I want to extract all entries from it (including zeros) and put them into > an array to use as an argument for some lapack functions. > What is the most optimal way of doing this? > Maybe MatConvert to MATSEQDENSE, then do what you want. Matt > Thank you, > Michael. > > > -- > Michael Povolotskyi, PhD > Research Assistant Professor > Network for Computational Nanotechnology > 207 S Martin Jischke Drive > Purdue University, DLR, room 441-10 > West Lafayette, Indiana 47907 > > phone: +1-765-494-9396 > fax: +1-765-496-6026 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpovolot at purdue.edu Mon Jun 24 17:08:24 2013 From: mpovolot at purdue.edu (Michael Povolotskyi) Date: Mon, 24 Jun 2013 18:08:24 -0400 Subject: [petsc-users] question about extracting numbers from a matrix In-Reply-To: References: <51C8C1C2.6090703@purdue.edu> Message-ID: <51C8C358.4040105@purdue.edu> thank you, Michael. On 06/24/2013 06:06 PM, Matthew Knepley wrote: > On Tue, Jun 25, 2013 at 12:01 AM, Michael Povolotskyi > > wrote: > > Hello everybody, > I have a sparse matrix defined on 1 MPI process. > I want to extract all entries from it (including zeros) and put > them into an array to use as an argument for some lapack functions. > What is the most optimal way of doing this? > > > Maybe MatConvert to MATSEQDENSE, then do what you want. > > Matt > > Thank you, > Michael. > > > -- > Michael Povolotskyi, PhD > Research Assistant Professor > Network for Computational Nanotechnology > 207 S Martin Jischke Drive > Purdue University, DLR, room 441-10 > West Lafayette, Indiana 47907 > > phone: +1-765-494-9396 > fax: +1-765-496-6026 > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -- Michael Povolotskyi, PhD Research Assistant Professor Network for Computational Nanotechnology 207 S Martin Jischke Drive Purdue University, DLR, room 441-10 West Lafayette, Indiana 47907 phone: +1-765-494-9396 fax: +1-765-496-6026 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Jun 24 17:07:57 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 24 Jun 2013 16:07:57 -0600 Subject: [petsc-users] question about extracting numbers from a matrix In-Reply-To: <51C8C1C2.6090703@purdue.edu> References: <51C8C1C2.6090703@purdue.edu> Message-ID: <87obavtb5u.fsf@mcs.anl.gov> Michael Povolotskyi writes: > Hello everybody, > I have a sparse matrix defined on 1 MPI process. > I want to extract all entries from it (including zeros) and put them > into an array to use as an argument for some lapack functions. > What is the most optimal way of doing this? Mat A_dense; PetscScalar a; MatConvert(A,MATDENSE,MAT_INITIAL_MATRIX,&A_dense); MatDenseGetArray(A_dense,&a); -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From fd.kong at siat.ac.cn Mon Jun 24 21:43:10 2013 From: fd.kong at siat.ac.cn (Fande Kong) Date: Tue, 25 Jun 2013 10:43:10 +0800 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> Message-ID: Hi Barry, How to use valgrind to debug parallel program on the supercomputer with many cores? If we follow the instruction "mpiexec -n NPROC valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p PETSCPROGRAMNAME -malloc off PROGRAMOPTIONS", for 10000 cores, 10000 files will be printed. Maybe we need to put all information into a single file. How to do this? On Mon, Jun 24, 2013 at 9:33 PM, Peter Lichtner wrote: > Just in case this helps I use Yellowstone for running PFLOTRAN with both > gcc and intel compilers using the developer version of PETSc. My > configuration script reads for intel: > > ./config/configure.py --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpicxx > --with-clanguage=c --with-blas-lapack-dir=$BLAS_LAPACK_LIB_DIR > --with-shared-libraries=0 --with-debugging=0 --download-hdf5=yes > --download-parmetis=yes --download-metis=yes > > echo $BLAS_LAPACK_LIB_DIR > /ncar/opt/intel/12.1.0.233/composer_xe_2013.1.117/mkl > > module load cmake/2.8.10.2 > > Intel was a little faster compared to gcc. > > ...Peter > > On Jun 24, 2013, at 1:53 AM, Fande Kong wrote: > > Hi Barry, > > I switched to gnu compiler. I also got the similar results: > > > [330]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [330]PETSC ERROR: Petsc has generated inconsistent data! > [330]PETSC ERROR: Negative MPI source: stash->nrecvs=27 i=33 > MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=5243744! > [330]PETSC ERROR: > ------------------------------------------------------------------------ > [330]PETSC ERROR: Petsc Release Version 3.4.1, unknown > [330]PETSC ERROR: See docs/changes/index.html for recent updates. > [330]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [330]PETSC ERROR: See docs/index.html for manual pages. > [330]PETSC ERROR: > ------------------------------------------------------------------------ > [330]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-opt_gnu named > ys0554 by fandek Mon Jun 24 01:42:37 2013 > [330]PETSC ERROR: Libraries linked from > /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu/lib > [330]PETSC ERROR: Configure run at Mon Jun 24 00:34:40 2013 > [330]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx > --with-shared-libraries=1 --with-dynamic-loading=1 > --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 > --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 > --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 > --with-debugging=no > [330]PETSC ERROR: > ------------------------------------------------------------------------ > [330]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in > /src/mat/utilsmatstash.c > [330]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in > /src/mat/impls/aij/mpimpiaij.c > [330]PETSC ERROR: MatAssemblyEnd() line 4939 in /src/mat/interfacematrix.c > [330]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp > [330]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in > meshreorder.cpp > [330]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp > [330]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp > [330]PETSC ERROR: main() line 71 in linearElasticity3d.cpp > > > > Thus, I think that it has nothing to do with the compiler. > > > On Sun, Jun 23, 2013 at 11:45 PM, Fande Kong wrote: > >> Thanks Barry? >> >> I will try impi. >> >> I have another question. In the previous email, you said If I can change >> to use another compiler. Why I need to change the compiler? >> >> >> On Mon, Jun 24, 2013 at 12:27 PM, Barry Smith wrote: >> >>> >>> Fande, >>> >>> We've seen trouble before with IBM on large intel systems at scale. >>> >>> From the previous configure.log you sent I see >>> >>> sh: mpicc -show >>> Executing: mpicc -show >>> sh: /ncar/opt/intel/ >>> 12.1.0.233/composer_xe_2011_sp1.11.339/bin/intel64/icc >>> -I/glade/apps/el6/include -I/glade/apps/el6/usr/include >>> -I/glade/apps/opt/netcdf/4.2/intel/default/include >>> -Wl,-rpath,/ncar/opt/intel/ >>> 12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64 -Wl,-rpath,/ncar/opt/intel/ >>> 12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/ia32 -L/glade/apps/el6/usr/lib -L/glade/apps/el6/usr/lib64 >>> -Wl,-rpath,/glade/apps/el6/usr/lib -Wl,-rpath,/glade/apps/el6/usr/lib64 >>> -L/glade/apps/opt/netcdf/4.2/intel/default/lib -lnetcdf_c++4 -lnetcdff >>> -lnetcdf -Wl,-rpath,/glade/apps/opt/netcdf/4.2/intel/default/lib -m64 >>> -D__64BIT__ -Wl,--allow-shlib-undefined -Wl,--enable-new-dtags >>> -Wl,-rpath,/opt/ibmhpc/pe1209/mpich2/intel/lib64 -Wl,-rpath,/ncar/opt/intel/ >>> 12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64-I/opt/ibmhpc/pe1209/mpich2/intel/include64 >>> -I/opt/ibmhpc/pe1209/base/include -L/opt/ibmhpc/pe1209/mpich2/intel/lib64 >>> -lmpi -ldl -L/ncar/opt/intel/ >>> 12.1.0.233/composer_xe_2011_sp1.11.339/compiler/lib/intel64 -lirc >>> -lpthread -lrt >>> >>> Note the -I/opt/ibmhpc/pe1209/base/include >>> -L/opt/ibmhpc/pe1209/mpich2/intel/lib64 -lmpi which is probably some IBM >>> hack job of some ancient mpich2 >>> >>> Now the page >>> http://www2.cisl.ucar.edu/resources/yellowstone/software/modules-intel-dependent has the modules >>> >>> >>> impi/4.0.3.008 This module loads the Intel MPI Library. See >>> http://software.intel.com/en-us/intel-mpi-library/ for details. >>> impi/4.1.0.030 This module loads the Intel MPI Library. See >>> http://software.intel.com/en-us/intel-mpi-library/ for details. >>> >>> Perhaps you could load those modules with the Intel compilers and >>> avoid the IBM MPI? If that solves the problem then we know the IBM MPI is >>> too blame. We are interested in working with you to determine the problem. >>> >>> Barry >>> >>> >>> >>> >>> On Jun 23, 2013, at 9:14 PM, Fande Kong wrote: >>> >>> > Thanks Barry, >>> > Thanks Jed, >>> > >>> > The computer I am using is Yellowstone >>> http://en.wikipedia.org/wiki/Yellowstone_(supercomputer), or >>> http://www2.cisl.ucar.edu/resources/yellowstone. The compiler is >>> intel compiler. The mpi is IBM mpi which is a part of IBM PE. >>> > >>> > With less unknowns (about 5 \times 10^7), the code can correctly run. >>> With unknowns (4 \times 10^8), the code produced the error messages. But >>> with so large unknowns (4 \times 10^8), the code can also run with less >>> cores. This is very strange. >>> > >>> > When I switch to gnu compiler, I can not install petsc, I got the >>> following errors: >>> > >>> > >>> ******************************************************************************* >>> > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >>> for details): >>> > >>> ------------------------------------------------------------------------------- >>> > Downloaded exodusii could not be used. Please check install in >>> /glade/p/work/fandek/petsc/arch-linux2-cxx-opt_gnu >>> > >>> ******************************************************************************* >>> > File "./config/configure.py", line 293, in petsc_configure >>> > framework.configure(out = sys.stdout) >>> > File >>> "/glade/p/work/fandek/petsc/config/BuildSystem/config/framework.py", line >>> 933, in configure >>> > child.configure() >>> > File >>> "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line >>> 556, in configure >>> > self.executeTest(self.configureLibrary) >>> > File "/glade/p/work/fandek/petsc/config/BuildSystem/config/base.py", >>> line 115, in executeTest >>> > ret = apply(test, args,kargs) >>> > File >>> "/glade/p/work/fandek/petsc/config/BuildSystem/config/packages/exodusii.py", >>> line 36, in configureLibrary >>> > config.package.Package.configureLibrary(self) >>> > File >>> "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line >>> 484, in configureLibrary >>> > for location, directory, lib, incl in self.generateGuesses(): >>> > File >>> "/glade/p/work/fandek/petsc/config/BuildSystem/config/package.py", line >>> 238, in generateGuesses >>> > raise RuntimeError('Downloaded '+self.package+' could not be used. >>> Please check install in '+d+'\n') >>> > >>> > >>> > The configure.log is attached. >>> > >>> > Regards, >>> > On Mon, Jun 24, 2013 at 1:03 AM, Jed Brown >>> wrote: >>> > Barry Smith writes: >>> > >>> > > What kind of computer system are you running? What MPI does it >>> use? These values are nonsense MPI_SOURCE=-32766 MPI_TAG=-32766 >>> > >>> > From configure.log, this is Intel MPI. Can you ask their support what >>> > this error condition is supposed to mean? It's not clear to me that >>> > MPI_SOURCE or MPI_TAG contain any meaningful information (though it >>> > could be indicative of an internal overflow), but this value of >>> > MPI_ERROR should mean something. >>> > >>> > > Is it possible to run the code with valgrind? >>> > > >>> > > Any chance of running the code with a different compiler? >>> > > >>> > > Barry >>> > > >>> > > >>> > > >>> > > On Jun 23, 2013, at 4:12 AM, Fande Kong wrote: >>> > > >>> > >> Thanks Jed, >>> > >> >>> > >> I added your code into the petsc. I run my code with 10240 cores. I >>> got the following error messages: >>> > >> >>> > >> [6724]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> > >> [6724]PETSC ERROR: Petsc has generated inconsistent data! >>> > >> [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11 >>> MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892! >>> > >> [6724]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> > >> [6724]PETSC ERROR: Petsc Release Version 3.4.1, unknown >>> > >> [6724]PETSC ERROR: See docs/changes/index.html for recent updates. >>> > >> [6724]PETSC ERROR: See docs/faq.html for hints about trouble >>> shooting. >>> > >> [6724]PETSC ERROR: See docs/index.html for manual pages. >>> > >> [6724]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> > >> [6724]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug >>> named ys4350 by fandek Sun Jun 23 02:58:23 2013 >>> > >> [6724]PETSC ERROR: Libraries linked from >>> /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib >>> > >> [6724]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 >>> > >> [6724]PETSC ERROR: Configure options --with-valgrind=1 >>> --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 >>> --download-f-blas-lapack=1 --with-mpi=1 --d >>> > >> ownload-parmetis=1 --download-metis=1 --with-64-bit-indices=1 >>> --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 >>> --download-hdf5=1 --with-debugging=yes >>> > >> [6724]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> > >> [6724]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in >>> /src/mat/utilsmatstash.c >>> > >> [6724]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in >>> /src/mat/impls/aij/mpimpiaij.c >>> > >> [6724]PETSC ERROR: MatAssemblyEnd() line 4939 in >>> /src/mat/interfacematrix.c >>> > >> [6724]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in >>> meshreorder.cpp >>> > >> [6724]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in >>> meshreorder.cpp >>> > >> [6724]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp >>> > >> [6724]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp >>> > >> [6724]PETSC ERROR: main() line 71 in linearElasticity3d.cpp >>> > >> Abort(77) on node 6724 (rank 6724 in comm 1140850688): application >>> called MPI_Abort(MPI_COMM_WORLD, 77) - process 6724 >>> > >> [2921]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> > >> [2921]PETSC ERROR: Petsc has generated inconsistent data! >>> > >> [2921]PETSC ERROR: Negative MPI source: stash->nrecvs=15 i=3 >>> MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=3825270! >>> > >> [2921]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> > >> [2921]PETSC ERROR: Petsc Release Version 3.4.1, unknown >>> > >> [2921]PETSC ERROR: See docs/changes/index.html for recent updates. >>> > >> [2921]PETSC ERROR: See docs/faq.html for hints about trouble >>> shooting. >>> > >> [2921]PETSC ERROR: See docs/index.html for manual pages. >>> > >> [2921]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> > >> [2921]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-debug >>> named ys0270 by fandek Sun Jun 23 02:58:23 2013 >>> > >> [2921]PETSC ERROR: Libraries linked from >>> /glade/p/work/fandek/petsc/arch-linux2-cxx-debug/lib >>> > >> [2921]PETSC ERROR: Configure run at Sun Jun 23 00:46:05 2013 >>> > >> [2921]PETSC ERROR: Configure options --with-valgrind=1 >>> --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 >>> --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 >>> --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 >>> --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 >>> --with-debugging=yes >>> > >> [2921]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> > >> [2921]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in >>> /src/mat/utilsmatstash.c >>> > >> [2921]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in >>> /src/mat/impls/aij/mpimpiaij.c >>> > >> [2921]PETSC ERROR: MatAssemblyEnd() line 4939 in >>> /src/mat/interfacematrix.c >>> > >> [2921]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in >>> meshreorder.cpp >>> > >> [2921]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in >>> meshreorder.cpp >>> > >> [2921]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp >>> > >> [2921]PETSC ERROR: DMmeshInitialize() line 78 in mgInitialize.cpp >>> > >> [2921]PETSC ERROR: main() line 71 in linearElasticity3d.cpp >>> > >> : >>> > >> >>> > >> On Fri, Jun 21, 2013 at 4:33 AM, Jed Brown >>> wrote: >>> > >> Fande Kong writes: >>> > >> >>> > >> > The code works well with less cores. And It also works well with >>> > >> > petsc-3.3-p7. But it does not work with petsc-3.4.1. Thus, If you >>> can check >>> > >> > the differences between petsc-3.3-p7 and petsc-3.4.1, you can >>> figure out >>> > >> > the reason. >>> > >> >>> > >> That is one way to start debugging, but there are no changes to the >>> core >>> > >> MatStash code, and many, many changes to PETSc in total. The >>> relevant >>> > >> snippet of code is here: >>> > >> >>> > >> if (stash->reproduce) { >>> > >> i = stash->reproduce_count++; >>> > >> ierr = >>> MPI_Wait(stash->recv_waits+i,&recv_status);CHKERRQ(ierr); >>> > >> } else { >>> > >> ierr = >>> MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);CHKERRQ(ierr); >>> > >> } >>> > >> if (recv_status.MPI_SOURCE < 0) >>> SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source!"); >>> > >> >>> > >> So MPI returns correctly (stash->reproduce will be FALSE unless you >>> > >> changed it). You could change the line above to the following: >>> > >> >>> > >> if (recv_status.MPI_SOURCE < 0) >>> SETERRQ5(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Negative MPI source: >>> stash->nrecvs=%D i=%d MPI_SOURCE=%d MPI_TAG=%d MPI_ERROR=%d", >>> > >> >>> stash->nrecvs,i,recv_status.MPI_SOURCE,recv_status.MPI_TAG,recv_status.MPI_ERROR); >>> > >> >>> > >> >>> > >> It would help to debug --with-debugging=1, so that more checks for >>> > >> corrupt data are performed. You can still make the compiler >>> optimize if >>> > >> it takes a long time to reach the error condition. >>> > >> >>> > >> >>> > >> >>> > >> -- >>> > >> Fande Kong >>> > >> ShenZhen Institutes of Advanced Technology >>> > >> Chinese Academy of Sciences >>> > >>> > >>> > >>> > -- >>> > Fande Kong >>> > ShenZhen Institutes of Advanced Technology >>> > Chinese Academy of Sciences >>> > >>> >>> >>> >> >> >> -- >> Fande Kong >> ShenZhen Institutes of Advanced Technology >> Chinese Academy of Sciences >> > > > > -- > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences > > > ________________ > Peter Lichtner > Santa Fe, NM 87507 > (505) 692-4029 (c) > OFM Research/LANL Guest Scientist > > -- Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Jun 24 22:08:57 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 24 Jun 2013 22:08:57 -0500 (CDT) Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> Message-ID: On Tue, 25 Jun 2013, Fande Kong wrote: > Hi Barry, > > How to use valgrind to debug parallel program on the supercomputer with > many cores? If we follow the instruction "mpiexec -n NPROC valgrind > --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p > PETSCPROGRAMNAME -malloc off PROGRAMOPTIONS", for 10000 cores, 10000 files > will be printed. Maybe we need to put all information into a single file. > How to do this? For this many cores - the PIDs across nodes won't be unique. It might map over to say 1000 files - so I suggest [assuming $HOSTNAME is set on each host] --log-file=valgrind.log.%q{HOSTNAME}.%p You don't want to be mixing output from all the cores - then it would be unreadable. But if your filesystem cannot handle these many files - you could try consolidating output per node as: --log-file=valgrind.log.%q{HOSTNAME} [or perhaps create a subdir per node or something - and stash files in these dirs] for each hostname: mkdir -p ${HOSTNAME} --log-file=%q{HOSTNAME}/valgrind.log.%p Satish From fd.kong at siat.ac.cn Tue Jun 25 05:09:04 2013 From: fd.kong at siat.ac.cn (Fande Kong) Date: Tue, 25 Jun 2013 18:09:04 +0800 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> Message-ID: Hi Barry, If I use the intel mpi, my code can correctly run and can produce some correct results. Yes, you are right. The IBM MPI has some bugs. Thank you for your help. Regards, On Tue, Jun 25, 2013 at 11:08 AM, Satish Balay wrote: > On Tue, 25 Jun 2013, Fande Kong wrote: > > > Hi Barry, > > > > How to use valgrind to debug parallel program on the supercomputer with > > many cores? If we follow the instruction "mpiexec -n NPROC valgrind > > --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p > > PETSCPROGRAMNAME -malloc off PROGRAMOPTIONS", for 10000 cores, 10000 > files > > will be printed. Maybe we need to put all information into a single file. > > How to do this? > > For this many cores - the PIDs across nodes won't be unique. It might > map over to say 1000 files - so I suggest [assuming $HOSTNAME is set > on each host] > > --log-file=valgrind.log.%q{HOSTNAME}.%p > > You don't want to be mixing output from all the cores - then it would > be unreadable. > > But if your filesystem cannot handle these many files - you could try > consolidating output per node as: > > --log-file=valgrind.log.%q{HOSTNAME} > > [or perhaps create a subdir per node or something - and stash files in > these dirs] > > for each hostname: mkdir -p ${HOSTNAME} > > --log-file=%q{HOSTNAME}/valgrind.log.%p > > > Satish > > -- Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From tsariysk at craft-tech.com Tue Jun 25 09:56:30 2013 From: tsariysk at craft-tech.com (Ted Sariyski) Date: Tue, 25 Jun 2013 10:56:30 -0400 Subject: [petsc-users] how to translate a FEM problem to PETSc In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> Message-ID: <51C9AF9E.8060902@craft-tech.com> Hi, I wrote an unstructured vector finite element electromagnetic solver, which among other modules, includes domain decomposition and (complex) GMRES solver. Now I am working on improvements and I would like to migrate the code to PETSc. I am new with PETSc. I am still working through the examples, trying to figure out a 'proper' entry point for integration. My current understanding is that domain decomposition and the assembly will remain outside PETSs. I already have the global-to-local/local-to-global data structures built. My difficulty is to figure out how to translate them to PETSc compliant format. I'll highly appreciate if somebody points me to examples illustrating how to translate a FEM problem to PETSc. Thanks in advance, --Ted Sariyski Combustion Research and Flow Technology, Inc. From mpovolot at purdue.edu Tue Jun 25 10:22:20 2013 From: mpovolot at purdue.edu (Michael Povolotskyi) Date: Tue, 25 Jun 2013 11:22:20 -0400 Subject: [petsc-users] question about extracting numbers from a matrix In-Reply-To: <87obavtb5u.fsf@mcs.anl.gov> References: <51C8C1C2.6090703@purdue.edu> <87obavtb5u.fsf@mcs.anl.gov> Message-ID: <51C9B5AC.1000104@purdue.edu> On 06/24/2013 06:07 PM, Jed Brown wrote: > Michael Povolotskyi writes: > >> Hello everybody, >> I have a sparse matrix defined on 1 MPI process. >> I want to extract all entries from it (including zeros) and put them >> into an array to use as an argument for some lapack functions. >> What is the most optimal way of doing this? > Mat A_dense; > PetscScalar a; > MatConvert(A,MATDENSE,MAT_INITIAL_MATRIX,&A_dense); > MatDenseGetArray(A_dense,&a); What is the difference (if any) between MatDenseGetArray and MatGetArray? -- Michael Povolotskyi, PhD Research Assistant Professor Network for Computational Nanotechnology 207 S Martin Jischke Drive Purdue University, DLR, room 441-10 West Lafayette, Indiana 47907 phone: +1-765-494-9396 fax: +1-765-496-6026 From jefonseca at gmail.com Tue Jun 25 10:36:42 2013 From: jefonseca at gmail.com (Jim Fonseca) Date: Tue, 25 Jun 2013 11:36:42 -0400 Subject: [petsc-users] question about extracting numbers from a matrix In-Reply-To: <51C9B5AC.1000104@purdue.edu> References: <51C8C1C2.6090703@purdue.edu> <87obavtb5u.fsf@mcs.anl.gov> <51C9B5AC.1000104@purdue.edu> Message-ID: Hi Michael, It's a change in the API for PETSc 3.4 http://www.mcs.anl.gov/petsc/documentation/changes/34.html - MatGetArray()/MatRestoreArray() have been remove d. Use MatDenseGetArray()/MatDenseRestoreArray() or MatSeqAIJGetArray()/MatSeqAIJRestoreArray() Thanks, Jim On Tue, Jun 25, 2013 at 11:22 AM, Michael Povolotskyi wrote: > On 06/24/2013 06:07 PM, Jed Brown wrote: > >> Michael Povolotskyi writes: >> >> Hello everybody, >>> I have a sparse matrix defined on 1 MPI process. >>> I want to extract all entries from it (including zeros) and put them >>> into an array to use as an argument for some lapack functions. >>> What is the most optimal way of doing this? >>> >> Mat A_dense; >> PetscScalar a; >> MatConvert(A,MATDENSE,MAT_**INITIAL_MATRIX,&A_dense); >> MatDenseGetArray(A_dense,&a); >> > What is the difference (if any) between MatDenseGetArray and MatGetArray? > > > -- > Michael Povolotskyi, PhD > Research Assistant Professor > Network for Computational Nanotechnology > 207 S Martin Jischke Drive > Purdue University, DLR, room 441-10 > West Lafayette, Indiana 47907 > > phone: +1-765-494-9396 > fax: +1-765-496-6026 > > -- Jim Fonseca, PhD Research Scientist Network for Computational Nanotechnology Purdue University 765-496-6495 www.jimfonseca.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 25 10:59:18 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Jun 2013 17:59:18 +0200 Subject: [petsc-users] how to translate a FEM problem to PETSc In-Reply-To: <51C9AF9E.8060902@craft-tech.com> References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> <51C9AF9E.8060902@craft-tech.com> Message-ID: On Tue, Jun 25, 2013 at 4:56 PM, Ted Sariyski wrote: > Hi, > > I wrote an unstructured vector finite element electromagnetic solver, > which among other modules, includes domain decomposition and (complex) > GMRES solver. Now I am working on improvements and I would like to migrate > the code to PETSc. I am new with PETSc. I am still working through the > examples, trying to figure out a 'proper' entry point for integration. My > current understanding is that domain decomposition and the assembly will > remain outside PETSs. I already have the global-to-local/local-to-**global > data structures built. My difficulty is to figure out how to translate them > to PETSc compliant format. I'll highly appreciate if somebody points me to > examples illustrating how to translate a FEM problem to PETSc. > Let me outline some options, and give you my recommendations: a) Use PETSc linear algebra This means using PETSc vectors, which is very easy, and using PETSc matrices, which is just a little harder. There is a description of this in previous tutorials in the "PETSc Integration" section. Once you use PETSc linear algebra, you have access to all Krylov solvers, and some preconditioners like ILU or Chebychev. This is a fine way to proceed, but does not integrate well with any multilevel solvers except AMG. b) Create your own DMShell The DM class is used to organize assembly and hierarchy. Our solvers take a DM (KSPSetDM, SNESSetDM, TSSetDM), and can use it to form the residual vector and Jacobian matrix. You can provide your own local-to-global mappings to the DM, so that more of the assembly organization is absorbed into the PETSc. This may simpligy your code, and also allow multilevel features in the future. Also, if you decide later to manage the grid using PETSc, this is the form it would take. Thanks, Matt > Thanks in advance, > --Ted Sariyski > Combustion Research and Flow Technology, Inc. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 25 15:15:18 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 25 Jun 2013 15:15:18 -0500 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> Message-ID: <36C63944-7E4C-4F02-A373-C37D9630AE67@mcs.anl.gov> On Jun 25, 2013, at 5:09 AM, Fande Kong wrote: > Hi Barry, > > If I use the intel mpi, my code can correctly run and can produce some correct results. Yes, you are right. The IBM MPI has some bugs. > Thanks for letting us know. Barry > Thank you for your help. > > Regards, > > On Tue, Jun 25, 2013 at 11:08 AM, Satish Balay wrote: > On Tue, 25 Jun 2013, Fande Kong wrote: > > > Hi Barry, > > > > How to use valgrind to debug parallel program on the supercomputer with > > many cores? If we follow the instruction "mpiexec -n NPROC valgrind > > --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p > > PETSCPROGRAMNAME -malloc off PROGRAMOPTIONS", for 10000 cores, 10000 files > > will be printed. Maybe we need to put all information into a single file. > > How to do this? > > For this many cores - the PIDs across nodes won't be unique. It might > map over to say 1000 files - so I suggest [assuming $HOSTNAME is set > on each host] > > --log-file=valgrind.log.%q{HOSTNAME}.%p > > You don't want to be mixing output from all the cores - then it would > be unreadable. > > But if your filesystem cannot handle these many files - you could try > consolidating output per node as: > > --log-file=valgrind.log.%q{HOSTNAME} > > [or perhaps create a subdir per node or something - and stash files in > these dirs] > > for each hostname: mkdir -p ${HOSTNAME} > > --log-file=%q{HOSTNAME}/valgrind.log.%p > > > Satish > > > > > -- > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences From suifengls at gmail.com Tue Jun 25 17:54:09 2013 From: suifengls at gmail.com (Longxiang Chen) Date: Tue, 25 Jun 2013 15:54:09 -0700 Subject: [petsc-users] Segmentation fault in PetscInitialize() Message-ID: Dear all, I try to use ksp to solve Ax=b. ALL programs are in fortran, except petscsolve() in c. Because petscsolve() is a sub-function that I don't have argc and argv. So I use following statement to set the options. PetscInitialize(NULL, NULL, NULL, help); PetscOptionsSetValue("-ksp_type", "bcgs"); PetscOptionsSetValue("-pc_type", "jacobi"); PetscOptionsSetValue("-ksp_max_it", "100"); PetscOptionsInsert(NULL, NULL, NULL); The program works like this: main() Init() LOOP: call MULTI (calculate Jacob A) -> call petscsolve( solve Ax=b) -> check solution Goto LOOP The first iteration works well that I can calculate the solution x from petscsolve(). But when it enters the second iteration of LOOP, it has Segmentation fault: Program received signal SIGSEGV, Segmentation fault. 0x000000000d8e11f0 in ?? () (gdb) bt #0 0x000000000d8e11f0 in ?? () #1 0x0000000001a32625 in PMPI_Barrier () #2 0x0000000000bdd2b7 in PetscLogBegin_Private () at /home/lchen/ft-petsc/src/sys/logging/plog.c:234 #3 0x0000000000bdd876 in PetscLogBegin () at /home/lchen/ft-petsc/src/sys/logging/plog.c:278 #4 0x00000000004c1f6c in PetscOptionsCheckInitial_Private () at /home/lchen/ft-petsc/src/sys/objects/init.c:494 *#5 0x00000000004cbe6d in PetscInitialize (argc=0x0, args=0x0, file=0x0, * * help=0x20a9f00 "Replace with PETSC_solve") at /home/lchen/ft-petsc/src/sys/objects/pinit.c:779* #6 0x00000000004991e3 in petscsolve_ (N=0x20c7d28, RHS=0x9813500, WKAREA=0x4902c80, NZ=0xc0629e0, IA=0xb8b9ac0, JA=0x754e520, CO=0x9871ac0, TOL=0x22c0528, ITMAX=0x71ef048, ITER=0x7fffffffd3a8, ERR=0x7fffffffd3b0, IERR=0x7fffffffd3b8, IUNIT=0x1bcb310, RWORK=0x57d2000, LENW=0x49f4cc0, IWORK=0x3e32860, LENIW=0x49f4cc8) at petscsolve.c:184 #7 0x000000000044ed15 in lineq () at t2cg22.f:1698 #8 0x000000000046a24c in cycit () at t2f.f:2055 #9 0x00000000004483d3 in tough2 () at t2cg22.f:625 #10 0x000000000040bfcc in main () Are there any problems with PetscInitialize(NULL, NULL, NULL, help)? Thank you. Best regards, Longxiang Chen -------------- next part -------------- An HTML attachment was scrubbed... URL: From ling.zou at inl.gov Tue Jun 25 18:47:50 2013 From: ling.zou at inl.gov (Zou (Non-US), Ling) Date: Tue, 25 Jun 2013 17:47:50 -0600 Subject: [petsc-users] Segmentation fault in PetscInitialize() In-Reply-To: References: Message-ID: I have petsc 3.3p5 on my Mac. PetscInitialize(NULL, NULL, NULL, NULL); It seems working fine here. Ling On Tue, Jun 25, 2013 at 4:54 PM, Longxiang Chen wrote: > Dear all, > > I try to use ksp to solve Ax=b. > > ALL programs are in fortran, except petscsolve() in c. > > Because petscsolve() is a sub-function that I don't have argc and argv. > So I use following statement to set the options. > > PetscInitialize(NULL, NULL, NULL, help); > PetscOptionsSetValue("-ksp_type", "bcgs"); > PetscOptionsSetValue("-pc_type", "jacobi"); > PetscOptionsSetValue("-ksp_max_it", "100"); > PetscOptionsInsert(NULL, NULL, NULL); > > The program works like this: > > main() > Init() > LOOP: > call MULTI (calculate Jacob A) > -> call petscsolve( solve Ax=b) > -> check solution > Goto LOOP > > The first iteration works well that I can calculate the solution x from > petscsolve(). > But when it enters the second iteration of LOOP, it has Segmentation fault: > > Program received signal SIGSEGV, Segmentation fault. > 0x000000000d8e11f0 in ?? () > (gdb) bt > #0 0x000000000d8e11f0 in ?? () > #1 0x0000000001a32625 in PMPI_Barrier () > #2 0x0000000000bdd2b7 in PetscLogBegin_Private () at > /home/lchen/ft-petsc/src/sys/logging/plog.c:234 > #3 0x0000000000bdd876 in PetscLogBegin () at > /home/lchen/ft-petsc/src/sys/logging/plog.c:278 > #4 0x00000000004c1f6c in PetscOptionsCheckInitial_Private () > at /home/lchen/ft-petsc/src/sys/objects/init.c:494 > *#5 0x00000000004cbe6d in PetscInitialize (argc=0x0, args=0x0, file=0x0, > * > * help=0x20a9f00 "Replace with PETSC_solve") at > /home/lchen/ft-petsc/src/sys/objects/pinit.c:779* > #6 0x00000000004991e3 in petscsolve_ (N=0x20c7d28, RHS=0x9813500, > WKAREA=0x4902c80, NZ=0xc0629e0, > IA=0xb8b9ac0, JA=0x754e520, CO=0x9871ac0, TOL=0x22c0528, > ITMAX=0x71ef048, ITER=0x7fffffffd3a8, > ERR=0x7fffffffd3b0, IERR=0x7fffffffd3b8, IUNIT=0x1bcb310, > RWORK=0x57d2000, LENW=0x49f4cc0, > IWORK=0x3e32860, LENIW=0x49f4cc8) at petscsolve.c:184 > #7 0x000000000044ed15 in lineq () at t2cg22.f:1698 > #8 0x000000000046a24c in cycit () at t2f.f:2055 > #9 0x00000000004483d3 in tough2 () at t2cg22.f:625 > #10 0x000000000040bfcc in main () > > Are there any problems with PetscInitialize(NULL, NULL, NULL, help)? > Thank you. > > Best regards, > Longxiang Chen > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ling.zou at inl.gov Tue Jun 25 18:50:29 2013 From: ling.zou at inl.gov (Zou (Non-US), Ling) Date: Tue, 25 Jun 2013 17:50:29 -0600 Subject: [petsc-users] Segmentation fault in PetscInitialize() In-Reply-To: References: Message-ID: Hmmm... I guess you cannot do PetscInitialize multiple times. Ling On Tue, Jun 25, 2013 at 5:47 PM, Zou (Non-US), Ling wrote: > I have petsc 3.3p5 on my Mac. > > PetscInitialize(NULL, NULL, NULL, NULL); > > > It seems working fine here. > > > Ling > > > On Tue, Jun 25, 2013 at 4:54 PM, Longxiang Chen wrote: > >> Dear all, >> >> I try to use ksp to solve Ax=b. >> >> ALL programs are in fortran, except petscsolve() in c. >> >> Because petscsolve() is a sub-function that I don't have argc and argv. >> So I use following statement to set the options. >> >> PetscInitialize(NULL, NULL, NULL, help); >> PetscOptionsSetValue("-ksp_type", "bcgs"); >> PetscOptionsSetValue("-pc_type", "jacobi"); >> PetscOptionsSetValue("-ksp_max_it", "100"); >> PetscOptionsInsert(NULL, NULL, NULL); >> >> The program works like this: >> >> main() >> Init() >> LOOP: >> call MULTI (calculate Jacob A) >> -> call petscsolve( solve Ax=b) >> -> check solution >> Goto LOOP >> >> The first iteration works well that I can calculate the solution x from >> petscsolve(). >> But when it enters the second iteration of LOOP, it has Segmentation >> fault: >> >> Program received signal SIGSEGV, Segmentation fault. >> 0x000000000d8e11f0 in ?? () >> (gdb) bt >> #0 0x000000000d8e11f0 in ?? () >> #1 0x0000000001a32625 in PMPI_Barrier () >> #2 0x0000000000bdd2b7 in PetscLogBegin_Private () at >> /home/lchen/ft-petsc/src/sys/logging/plog.c:234 >> #3 0x0000000000bdd876 in PetscLogBegin () at >> /home/lchen/ft-petsc/src/sys/logging/plog.c:278 >> #4 0x00000000004c1f6c in PetscOptionsCheckInitial_Private () >> at /home/lchen/ft-petsc/src/sys/objects/init.c:494 >> *#5 0x00000000004cbe6d in PetscInitialize (argc=0x0, args=0x0, >> file=0x0, * >> * help=0x20a9f00 "Replace with PETSC_solve") at >> /home/lchen/ft-petsc/src/sys/objects/pinit.c:779* >> #6 0x00000000004991e3 in petscsolve_ (N=0x20c7d28, RHS=0x9813500, >> WKAREA=0x4902c80, NZ=0xc0629e0, >> IA=0xb8b9ac0, JA=0x754e520, CO=0x9871ac0, TOL=0x22c0528, >> ITMAX=0x71ef048, ITER=0x7fffffffd3a8, >> ERR=0x7fffffffd3b0, IERR=0x7fffffffd3b8, IUNIT=0x1bcb310, >> RWORK=0x57d2000, LENW=0x49f4cc0, >> IWORK=0x3e32860, LENIW=0x49f4cc8) at petscsolve.c:184 >> #7 0x000000000044ed15 in lineq () at t2cg22.f:1698 >> #8 0x000000000046a24c in cycit () at t2f.f:2055 >> #9 0x00000000004483d3 in tough2 () at t2cg22.f:625 >> #10 0x000000000040bfcc in main () >> >> Are there any problems with PetscInitialize(NULL, NULL, NULL, help)? >> Thank you. >> >> Best regards, >> Longxiang Chen >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 25 18:58:15 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 25 Jun 2013 18:58:15 -0500 Subject: [petsc-users] Segmentation fault in PetscInitialize() In-Reply-To: References: Message-ID: <95A56877-05D2-4F7A-BB37-6B756FFC22A5@mcs.anl.gov> On Jun 25, 2013, at 6:50 PM, "Zou (Non-US), Ling" wrote: > Hmmm... I guess you cannot do PetscInitialize multiple times. You must call PetscFinalize() each time before calling PetscInitialize() again. In addition you must call MPI_Init() your self ONCE at the beginning of the application code. Barry > > Ling > > > On Tue, Jun 25, 2013 at 5:47 PM, Zou (Non-US), Ling wrote: > I have petsc 3.3p5 on my Mac. > > PetscInitialize(NULL, NULL, NULL, NULL); > > It seems working fine here. > > Ling > > > On Tue, Jun 25, 2013 at 4:54 PM, Longxiang Chen wrote: > Dear all, > > I try to use ksp to solve Ax=b. > > ALL programs are in fortran, except petscsolve() in c. > > Because petscsolve() is a sub-function that I don't have argc and argv. > So I use following statement to set the options. > > PetscInitialize(NULL, NULL, NULL, help); > PetscOptionsSetValue("-ksp_type", "bcgs"); > PetscOptionsSetValue("-pc_type", "jacobi"); > PetscOptionsSetValue("-ksp_max_it", "100"); > PetscOptionsInsert(NULL, NULL, NULL); > > The program works like this: > > main() > Init() > LOOP: > call MULTI (calculate Jacob A) > -> call petscsolve( solve Ax=b) > -> check solution > Goto LOOP > > The first iteration works well that I can calculate the solution x from petscsolve(). > But when it enters the second iteration of LOOP, it has Segmentation fault: > > Program received signal SIGSEGV, Segmentation fault. > 0x000000000d8e11f0 in ?? () > (gdb) bt > #0 0x000000000d8e11f0 in ?? () > #1 0x0000000001a32625 in PMPI_Barrier () > #2 0x0000000000bdd2b7 in PetscLogBegin_Private () at /home/lchen/ft-petsc/src/sys/logging/plog.c:234 > #3 0x0000000000bdd876 in PetscLogBegin () at /home/lchen/ft-petsc/src/sys/logging/plog.c:278 > #4 0x00000000004c1f6c in PetscOptionsCheckInitial_Private () > at /home/lchen/ft-petsc/src/sys/objects/init.c:494 > #5 0x00000000004cbe6d in PetscInitialize (argc=0x0, args=0x0, file=0x0, > help=0x20a9f00 "Replace with PETSC_solve") at /home/lchen/ft-petsc/src/sys/objects/pinit.c:779 > #6 0x00000000004991e3 in petscsolve_ (N=0x20c7d28, RHS=0x9813500, WKAREA=0x4902c80, NZ=0xc0629e0, > IA=0xb8b9ac0, JA=0x754e520, CO=0x9871ac0, TOL=0x22c0528, ITMAX=0x71ef048, ITER=0x7fffffffd3a8, > ERR=0x7fffffffd3b0, IERR=0x7fffffffd3b8, IUNIT=0x1bcb310, RWORK=0x57d2000, LENW=0x49f4cc0, > IWORK=0x3e32860, LENIW=0x49f4cc8) at petscsolve.c:184 > #7 0x000000000044ed15 in lineq () at t2cg22.f:1698 > #8 0x000000000046a24c in cycit () at t2f.f:2055 > #9 0x00000000004483d3 in tough2 () at t2cg22.f:625 > #10 0x000000000040bfcc in main () > > Are there any problems with PetscInitialize(NULL, NULL, NULL, help)? > Thank you. > > Best regards, > Longxiang Chen > > > From suifengls at gmail.com Tue Jun 25 21:30:53 2013 From: suifengls at gmail.com (Longxiang Chen) Date: Tue, 25 Jun 2013 19:30:53 -0700 Subject: [petsc-users] Segmentation fault in PetscInitialize() In-Reply-To: <95A56877-05D2-4F7A-BB37-6B756FFC22A5@mcs.anl.gov> References: <95A56877-05D2-4F7A-BB37-6B756FFC22A5@mcs.anl.gov> Message-ID: Yes. I move MPI_Init() outsidet the LOOP. Now it works. Thanks you. Best regards, Longxiang Chen On Tue, Jun 25, 2013 at 4:58 PM, Barry Smith wrote: > > On Jun 25, 2013, at 6:50 PM, "Zou (Non-US), Ling" > wrote: > > > Hmmm... I guess you cannot do PetscInitialize multiple times. > > You must call PetscFinalize() each time before calling > PetscInitialize() again. In addition you must call MPI_Init() your self > ONCE at the beginning of the application code. > > Barry > > > > > Ling > > > > > > On Tue, Jun 25, 2013 at 5:47 PM, Zou (Non-US), Ling > wrote: > > I have petsc 3.3p5 on my Mac. > > > > PetscInitialize(NULL, NULL, NULL, NULL); > > > > It seems working fine here. > > > > Ling > > > > > > On Tue, Jun 25, 2013 at 4:54 PM, Longxiang Chen > wrote: > > Dear all, > > > > I try to use ksp to solve Ax=b. > > > > ALL programs are in fortran, except petscsolve() in c. > > > > Because petscsolve() is a sub-function that I don't have argc and argv. > > So I use following statement to set the options. > > > > PetscInitialize(NULL, NULL, NULL, help); > > PetscOptionsSetValue("-ksp_type", "bcgs"); > > PetscOptionsSetValue("-pc_type", "jacobi"); > > PetscOptionsSetValue("-ksp_max_it", "100"); > > PetscOptionsInsert(NULL, NULL, NULL); > > > > The program works like this: > > > > main() > > Init() > > LOOP: > > call MULTI (calculate Jacob A) > > -> call petscsolve( solve Ax=b) > > -> check solution > > Goto LOOP > > > > The first iteration works well that I can calculate the solution x from > petscsolve(). > > But when it enters the second iteration of LOOP, it has Segmentation > fault: > > > > Program received signal SIGSEGV, Segmentation fault. > > 0x000000000d8e11f0 in ?? () > > (gdb) bt > > #0 0x000000000d8e11f0 in ?? () > > #1 0x0000000001a32625 in PMPI_Barrier () > > #2 0x0000000000bdd2b7 in PetscLogBegin_Private () at > /home/lchen/ft-petsc/src/sys/logging/plog.c:234 > > #3 0x0000000000bdd876 in PetscLogBegin () at > /home/lchen/ft-petsc/src/sys/logging/plog.c:278 > > #4 0x00000000004c1f6c in PetscOptionsCheckInitial_Private () > > at /home/lchen/ft-petsc/src/sys/objects/init.c:494 > > #5 0x00000000004cbe6d in PetscInitialize (argc=0x0, args=0x0, file=0x0, > > help=0x20a9f00 "Replace with PETSC_solve") at > /home/lchen/ft-petsc/src/sys/objects/pinit.c:779 > > #6 0x00000000004991e3 in petscsolve_ (N=0x20c7d28, RHS=0x9813500, > WKAREA=0x4902c80, NZ=0xc0629e0, > > IA=0xb8b9ac0, JA=0x754e520, CO=0x9871ac0, TOL=0x22c0528, > ITMAX=0x71ef048, ITER=0x7fffffffd3a8, > > ERR=0x7fffffffd3b0, IERR=0x7fffffffd3b8, IUNIT=0x1bcb310, > RWORK=0x57d2000, LENW=0x49f4cc0, > > IWORK=0x3e32860, LENIW=0x49f4cc8) at petscsolve.c:184 > > #7 0x000000000044ed15 in lineq () at t2cg22.f:1698 > > #8 0x000000000046a24c in cycit () at t2f.f:2055 > > #9 0x00000000004483d3 in tough2 () at t2cg22.f:625 > > #10 0x000000000040bfcc in main () > > > > Are there any problems with PetscInitialize(NULL, NULL, NULL, help)? > > Thank you. > > > > Best regards, > > Longxiang Chen > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bisheshkh at gmail.com Wed Jun 26 08:44:45 2013 From: bisheshkh at gmail.com (Bishesh Khanal) Date: Wed, 26 Jun 2013 15:44:45 +0200 Subject: [petsc-users] PetscViewerBinaryMatlab example for petsc-3.4.1 Message-ID: Dear all, I'm just starting to learn using Petsc. I was trying the example (../src/dm/examples/tutorials/ex7.c) for petsc-3.4.1 . However it seems this file has not been updated for the 3.4.1 version since it uses PetscViewerBinaryMatlabOpen() and friends which have been removed from this newer version. Now I tried to change them but I have some questions: The lines I wanted to change are the ones that use following funcitons: 1. PetscViewerBinaryMatlabOpen(PETSC_COMM_WORLD,params->filename,&viewer) 2. PetscViewerBinaryMatlabOutputBag(viewer,"params",bag) 3. PetscViewerBinaryMatlabOutputVecDA(viewer,"da1",global,da) For 1, it seems I can change it with: PetscViewerBinaryOpen(PETSC_COMM_WORLD,params->filename,FILE_MODE_WRITE,&viewer) PetscViewerSetFormat(viewer,PETSC_VIEWER_BINARY_MATLAB) Now, for the 2 and 3 I thought I need to use: PetscViewerBinaryWrite(PetscViewer viewer,void *data,PetscInt count,PetscDataType dtype,PetscBool istemp) but I'm not sure what exactly to provide as argument for *count* and *dtype*to have the same functionality as intended in ex7.c I tried (for 2) using: ierr = PetscViewerBinaryWrite(viewer,&bag,1,PETSC_OBJECT,PETSC_FALSE); but it results in an error when running the executable. How should I count the no. of items for *count* variable in this case for the* bag* object ? And what should be the dtype ? Thanks, Bishesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Jun 26 09:21:48 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 26 Jun 2013 09:21:48 -0500 Subject: [petsc-users] How to understand these error messages In-Reply-To: References: <87bo703r3t.fsf@mcs.anl.gov> <87vc573g8x.fsf@mcs.anl.gov> <87zjugwyhp.fsf@mcs.anl.gov> Message-ID: <87ip11lzpf.fsf@mcs.anl.gov> Fande Kong writes: > Hi Barry, > > If I use the intel mpi, my code can correctly run and can produce some > correct results. Yes, you are right. The IBM MPI has some bugs. Fande, please report this issue to the IBM. Jeff, Fande has a reproducible case where when running on 10k cores and problem sizes over 100M, this MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status); returns [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892! It runs correctly for smaller problem sizes, smaller core counts, or for all sizes when using Intel MPI. This is on Yellowstone (iDataPlex, 4500 dx360 nodes). Do you know someone at IBM that should be notified? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From jhammond at alcf.anl.gov Wed Jun 26 09:56:01 2013 From: jhammond at alcf.anl.gov (Jeff Hammond) Date: Wed, 26 Jun 2013 09:56:01 -0500 (CDT) Subject: [petsc-users] How to understand these error messages In-Reply-To: <87ip11lzpf.fsf@mcs.anl.gov> Message-ID: <1488219718.8188107.1372258561902.JavaMail.root@alcf.anl.gov> This concerns IBM PE-MPI on iDataPlex, which is likely based upon the cluster implementation of PAMI, which is a completely different code base from the PAMI Blue Gene implementation. If you can reproduce it on Blue Gene/Q, I will care. As an IBM customer, NCAR is endowed with the ability to file bug reports directly with IBM related to the products they possess. There is a link to their support system on http://www2.cisl.ucar.edu/resources/yellowstone, which is the appropriate channel for users of Yellowstone that have issues with the system software installed there. Jeff ----- Original Message ----- From: "Jed Brown" To: "Fande Kong" , "petsc-users" Cc: "Jeff Hammond" Sent: Wednesday, June 26, 2013 9:21:48 AM Subject: Re: [petsc-users] How to understand these error messages Fande Kong writes: > Hi Barry, > > If I use the intel mpi, my code can correctly run and can produce some > correct results. Yes, you are right. The IBM MPI has some bugs. Fande, please report this issue to the IBM. Jeff, Fande has a reproducible case where when running on 10k cores and problem sizes over 100M, this MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status); returns [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892! It runs correctly for smaller problem sizes, smaller core counts, or for all sizes when using Intel MPI. This is on Yellowstone (iDataPlex, 4500 dx360 nodes). Do you know someone at IBM that should be notified? -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond at alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides From bclyons at princeton.edu Wed Jun 26 10:15:18 2013 From: bclyons at princeton.edu (Brendan C Lyons) Date: Wed, 26 Jun 2013 11:15:18 -0400 Subject: [petsc-users] SEGV on KSPSolve with mutliple processors In-Reply-To: <73f592c2f3c54877b69dbe8001dd6f9b@CSGHUB209W.pu.win.princeton.edu> References: <73f592c2f3c54877b69dbe8001dd6f9b@CSGHUB209W.pu.win.princeton.edu> Message-ID: Dear all, Sorry for the delayed response. It took me sometime to get a debug version available on the clusters I used. I reran my code compiled with debugging turned on and ran it with valgrind. No error was caught before the call to KSPSolve() and I've now received the error message below (I put it only for one of the four processors here, but I believe it's the same for all of them). Any further advice that could help me track down the cause for this segfault would be appreciated. Thanks, ~Brendan [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [3]PETSC ERROR: likely location of problem given in stack below [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [3]PETSC ERROR: INSTEAD the line number of the start of the function [3]PETSC ERROR: is given. [3]PETSC ERROR: [3] MatLUFactorNumeric_SuperLU_DIST line 284 src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c [3]PETSC ERROR: [3] MatLUFactorNumeric line 2791 src/mat/interface/matrix.c [3]PETSC ERROR: [3] PCSetUp_LU line 108 src/ksp/pc/impls/factor/lu/lu.c [3]PETSC ERROR: [3] PCSetUp line 810 src/ksp/pc/interface/precon.c [3]PETSC ERROR: [3] KSPSetUp line 182 src/ksp/ksp/interface/itfunc.c [3]PETSC ERROR: [3] KSPSolve line 351 src/ksp/ksp/interface/itfunc.c [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Signal received! [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 6, Mon Feb 11 12:26:34 CST 2013 [3]PETSC ERROR: See docs/changes/index.html for recent updates. [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [3]PETSC ERROR: See docs/index.html for manual pages. [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: N4D_d.exe on a path-ompi named [REDACTED] by blyons Tue Jun 25 13:50:35 2013 [3]PETSC ERROR: Libraries linked from ${PETSC_DIR}/path-ompi/lib [3]PETSC ERROR: Configure run at Mon Jun 24 13:11:36 2013 [3]PETSC ERROR: Configure options --PETSC_ARCH=path-ompi --PETSC_DIR=${PATHSCALE_DIR}/openmpi-1.6-pkgs/petsc-3.3-p6-debug --CFLAGS="-fPIC -O -mp" --CXXFLAGS="-fPIC -O -mp" --FFLAGS="-fPIC -O -mp" --with-debugging=1 --with-dynamic-loading=no --with-mpi=1 --with-mpi-dir=${PATHSCALE_DIR}/openmpi-1.6.4 --with-superlu=1 --with-superlu-dir=${PATHSCALE_DIR}/superlu-4.3 --with-blas-lapack-lib="[${PATHSCALE_DIR}/acml-5.3.0/open64_64/lib/libacml.a]--with-metis=1" --with-metis-dir=${PATHSCALE_DIR}/metis-5.0.3 --with-parmetis=1 --with-parmetis-dir=${PATHSCALE_DIR}/openmpi-1.6-pkgs/parmetis-4.0.2 --with-blas-lapack-lib="${PATHSCALE_DIR}/acml-5.3.0/open64_64/lib/libacml.a]" --with-blacs=1 --with-blacs-lib="[${PATHSCALE_DIR}/openmpi-1.6-pkgs/blacs-1.1p3/lib/blacsCinit_MPI-LINUX-0.a,${PATHSCALE_DIR}/openmpi-1.6-pkgs/blacs-1.1p3/lib/blacsF77init_MPI-LINUX-0.a,${PATHSCALE_DIR}/openmpi-1.6-pkgs/blacs-1.1p3/lib/blacs_MPI-LINUX-0.a]" --with-blacs-include=${PATHSCALE_DIR}/openmpi-1.6-pkgs/blacs-1.1p3/include --with-hypre=1 --download-hypre=1 --with-scalapack=1 --with-scalapack-dir=${PATHSCALE_DIR}/openmpi-1.6-pkgs/scalapack-2.0.2 --with-superlu_dist=1 --with-superlu_dist-dir=${PATHSCALE_DIR}/openmpi-1.6-pkgs/superlu_dist-3.2 [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- ==421== Thread 16: ==421== Invalid free() / delete / delete[] / realloc() ==421== at 0x4C2333A: free (vg_replace_malloc.c:446) ==421== by 0x9CB25AA: free_mem (in /lib64/libc-2.5.so) ==421== by 0x9CB21A1: __libc_freeres (in /lib64/libc-2.5.so) ==421== by 0x4A1E669: _vgnU_freeres (vg_preloaded.c:62) ==421== Address 0x4165e78 is not stack'd, malloc'd or (recently) free'd ==421== On Tue, Jun 18, 2013 at 4:52 PM, Barry Smith wrote: > > If possible you would also benefit from running the debug version under > valgrind http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > it is possible memory corruption has taken place before the point where the > code crashes, valgrind will help identify any memory corruption as soon as > it takes place. > > Barry > > On Jun 18, 2013, at 2:15 PM, Dave May wrote: > > > You should recompile your code using a debug build of petsc so you get > some meaningful info from the stack trace when the Segv occurs. > > > > Dave > > > > > > On Tuesday, 18 June 2013, Brendan C Lyons wrote: > > Hi everyone, > > > > I've run into a strange problem in my Fortran 90 code where it runs fine > with 1 processor, but then throws a segmentation fault on KSPSolve() when I > try to run it in parallel. I'm using PETSc 3.3 with the SuperLU direct > solver for the sequential case and SuperLU_dist for the parallel case. > I've called KSPView before and after KSPSolve. I'll put the KSPView > output for the sequential and parallel cases and the crash info for the > parallel case below (with some details of my system redacted). Any help > would be appreciated. If you need any other information, I'm happy to > provide it. > > > > Thank you, > > > > ~Brendan > > ------------------------------ > > > > KSPView() before sequential solve: > > > > KSP Object: 1 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: 1 MPI processes > > type: lu > > LU: out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > linear system matrix = precond matrix: > > Matrix Object: 1 MPI processes > > type: seqaij > > rows=11760, cols=11760 > > total: nonzeros=506586, allocated nonzeros=509061 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node routines > > > > KSPView() after sequential solve: > > > > KSP Object: 1 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using NONE norm type for convergence test > > PC Object: 1 MPI processes > > type: lu > > LU: out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > factor fill ratio given 0, needed 0 > > Factored matrix follows: > > Matrix Object: 1 MPI processes > > type: seqaij > > rows=11760, cols=11760 > > package used to perform factorization: superlu > > total: nonzeros=0, allocated nonzeros=0 > > total number of mallocs used during MatSetValues calls =0 > > SuperLU run parameters: > > Equil: NO > > ColPerm: 3 > > IterRefine: 0 > > SymmetricMode: NO > > DiagPivotThresh: 1 > > PivotGrowth: NO > > ConditionNumber: NO > > RowPerm: 0 > > ReplaceTinyPivot: NO > > PrintStat: NO > > lwork: 0 > > linear system matrix = precond matrix: > > Matrix Object: 1 MPI processes > > type: seqaij > > rows=11760, cols=11760 > > total: nonzeros=506586, allocated nonzeros=509061 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node routines > > > > > > KSPView() before parallel solve: > > > > KSP Object: 2 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: 2 MPI processes > > type: lu > > LU: out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: natural > > linear system matrix = precond matrix: > > Solving Electron Matrix Equation > > Matrix Object: 2 MPI processes > > type: mpiaij > > rows=11760, cols=11760 > > total: nonzeros=506586, allocated nonzeros=520821 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node (on process 0) routines > > > > Crash info for parallel solve: > > > > [1]PETSC ERROR: > ------------------------------------------------------------------------ > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > [1]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [1]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [1]PETSC ERROR: to get more information on the crash. > > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > [1]PETSC ERROR: Signal received! > > [1]PETSC ERROR: > ------------------------------------------------------------------------ > > [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 6, Mon Feb 11 > 12:26:34 CST 2013 > > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [1]PETSC ERROR: See docs/index.html for manual pages. > > [1]PETSC ERROR: > ------------------------------------------------------------------------ > > [1]PETSC ERROR: on a path-ompi named > > [1]PETSC ERROR: Libraries linked from > > [1]PETSC ERROR: Configure run at Thu Mar 21 14:19:42 2013 > > [1]PETSC ERROR: Configure options --PETSC_ARCH=path-ompi > --PETSC_DIR= --CFLAGS="-fPIC -O -mp" --CXXFLAGS="-fPIC -O -mp" > --FFLAGS="-fPIC -O -mp" --with-debugging=0 --with-dynamic-loadin=no > --with-mpi=1 --with-mpi-dir= --with-superlu=1 > --with-superlu-dir= --with-blas-lapack-lib="" > --with-scalapack=1 --with-scalapack-dir= --with-superlu_dist=1 > --with-superlu_dist-dir= --with-metis=1 > --with-metis-dir= --with-parmetis=1 > --with-parmetis-dir= --with-blacs-lib="" > --with-blacs-include= --with-hypre=1 --download-hypre=1 > > [1]PETSC ERROR: > ------------------------------------------------------------------------ > > [1]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Jun 26 10:38:11 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 26 Jun 2013 10:38:11 -0500 Subject: [petsc-users] SEGV on KSPSolve with mutliple processors In-Reply-To: References: <73f592c2f3c54877b69dbe8001dd6f9b@CSGHUB209W.pu.win.princeton.edu> Message-ID: Brendan : [3]PETSC ERROR: [3] MatLUFactorNumeric_SuperLU_DIST line 284 src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c line 284 src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c PetscFunctionBegin; i.e., the beginning of the subroutine, thus it does not tell us which line SEGV occurs. Your code seems still built with O-mode. It seems the crash occurs in superlu_dist. Suggest 1. update your code to petsc-3.4 with latest superlu_dist and mumps 2. use a local linux machine, build debug version of petsc, superlu_dist and mumps 3. then port to the cluster Hong > > Sorry for the delayed response. It took me sometime to get a debug > version available on the clusters I used. I reran my code compiled with > debugging turned on and ran it with valgrind. No error was caught before > the call to KSPSolve() and I've now received the error message below (I put > it only for one of the four processors here, but I believe it's the same > for all of them). Any further advice that could help me track down the > cause for this segfault would be appreciated. > > Thanks, > > ~Brendan > > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [3]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [3]PETSC ERROR: likely location of problem given in stack below > [3]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [3]PETSC ERROR: INSTEAD the line number of the start of the function > [3]PETSC ERROR: is given. > [3]PETSC ERROR: [3] MatLUFactorNumeric_SuperLU_DIST line 284 > src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c > [3]PETSC ERROR: [3] MatLUFactorNumeric line 2791 src/mat/interface/matrix.c > [3]PETSC ERROR: [3] PCSetUp_LU line 108 src/ksp/pc/impls/factor/lu/lu.c > [3]PETSC ERROR: [3] PCSetUp line 810 src/ksp/pc/interface/precon.c > [3]PETSC ERROR: [3] KSPSetUp line 182 src/ksp/ksp/interface/itfunc.c > [3]PETSC ERROR: [3] KSPSolve line 351 src/ksp/ksp/interface/itfunc.c > [3]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [3]PETSC ERROR: Signal received! > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 6, Mon Feb 11 12:26:34 > CST 2013 > [3]PETSC ERROR: See docs/changes/index.html for recent updates. > [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [3]PETSC ERROR: See docs/index.html for manual pages. > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: N4D_d.exe on a path-ompi named [REDACTED] by blyons Tue > Jun 25 13:50:35 2013 > [3]PETSC ERROR: Libraries linked from ${PETSC_DIR}/path-ompi/lib > [3]PETSC ERROR: Configure run at Mon Jun 24 13:11:36 2013 > [3]PETSC ERROR: Configure options --PETSC_ARCH=path-ompi > --PETSC_DIR=${PATHSCALE_DIR}/openmpi-1.6-pkgs/petsc-3.3-p6-debug --CFLAGS="-fPIC > -O -mp" --CXXFLAGS="-fPIC -O -mp" --FFLAGS="-fPIC -O -mp" > --with-debugging=1 --with-dynamic-loading=no --with-mpi=1 > --with-mpi-dir=${PATHSCALE_DIR}/openmpi-1.6.4 --with-superlu=1 > --with-superlu-dir=${PATHSCALE_DIR}/superlu-4.3 > --with-blas-lapack-lib="[${PATHSCALE_DIR}/acml-5.3.0/open64_64/lib/libacml.a]--with-metis=1" > --with-metis-dir=${PATHSCALE_DIR}/metis-5.0.3 --with-parmetis=1 > --with-parmetis-dir=${PATHSCALE_DIR}/openmpi-1.6-pkgs/parmetis-4.0.2 > --with-blas-lapack-lib="${PATHSCALE_DIR}/acml-5.3.0/open64_64/lib/libacml.a]" > --with-blacs=1 > --with-blacs-lib="[${PATHSCALE_DIR}/openmpi-1.6-pkgs/blacs-1.1p3/lib/blacsCinit_MPI-LINUX-0.a,${PATHSCALE_DIR}/openmpi-1.6-pkgs/blacs-1.1p3/lib/blacsF77init_MPI-LINUX-0.a,${PATHSCALE_DIR}/openmpi-1.6-pkgs/blacs-1.1p3/lib/blacs_MPI-LINUX-0.a]" > --with-blacs-include=${PATHSCALE_DIR}/openmpi-1.6-pkgs/blacs-1.1p3/include > --with-hypre=1 --download-hypre=1 --with-scalapack=1 > --with-scalapack-dir=${PATHSCALE_DIR}/openmpi-1.6-pkgs/scalapack-2.0.2 > --with-superlu_dist=1 > --with-superlu_dist-dir=${PATHSCALE_DIR}/openmpi-1.6-pkgs/superlu_dist-3.2 > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > ==421== Thread 16: > ==421== Invalid free() / delete / delete[] / realloc() > ==421== at 0x4C2333A: free (vg_replace_malloc.c:446) > ==421== by 0x9CB25AA: free_mem (in /lib64/libc-2.5.so) > ==421== by 0x9CB21A1: __libc_freeres (in /lib64/libc-2.5.so) > ==421== by 0x4A1E669: _vgnU_freeres (vg_preloaded.c:62) > ==421== Address 0x4165e78 is not stack'd, malloc'd or (recently) free'd > ==421== > > > > On Tue, Jun 18, 2013 at 4:52 PM, Barry Smith wrote: > >> >> If possible you would also benefit from running the debug version >> under valgrind >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind it is >> possible memory corruption has taken place before the point where the code >> crashes, valgrind will help identify any memory corruption as soon as it >> takes place. >> >> Barry >> >> On Jun 18, 2013, at 2:15 PM, Dave May wrote: >> >> > You should recompile your code using a debug build of petsc so you get >> some meaningful info from the stack trace when the Segv occurs. >> > >> > Dave >> > >> > >> > On Tuesday, 18 June 2013, Brendan C Lyons wrote: >> > Hi everyone, >> > >> > I've run into a strange problem in my Fortran 90 code where it runs >> fine with 1 processor, but then throws a segmentation fault on KSPSolve() >> when I try to run it in parallel. I'm using PETSc 3.3 with the SuperLU >> direct solver for the sequential case and SuperLU_dist for the parallel >> case. I've called KSPView before and after KSPSolve. I'll put the KSPView >> output for the sequential and parallel cases and the crash info for the >> parallel case below (with some details of my system redacted). Any help >> would be appreciated. If you need any other information, I'm happy to >> provide it. >> > >> > Thank you, >> > >> > ~Brendan >> > ------------------------------ >> > >> > KSPView() before sequential solve: >> > >> > KSP Object: 1 MPI processes >> > type: preonly >> > maximum iterations=10000, initial guess is zero >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> > left preconditioning >> > using DEFAULT norm type for convergence test >> > PC Object: 1 MPI processes >> > type: lu >> > LU: out-of-place factorization >> > tolerance for zero pivot 2.22045e-14 >> > matrix ordering: nd >> > linear system matrix = precond matrix: >> > Matrix Object: 1 MPI processes >> > type: seqaij >> > rows=11760, cols=11760 >> > total: nonzeros=506586, allocated nonzeros=509061 >> > total number of mallocs used during MatSetValues calls =0 >> > not using I-node routines >> > >> > KSPView() after sequential solve: >> > >> > KSP Object: 1 MPI processes >> > type: preonly >> > maximum iterations=10000, initial guess is zero >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> > left preconditioning >> > using NONE norm type for convergence test >> > PC Object: 1 MPI processes >> > type: lu >> > LU: out-of-place factorization >> > tolerance for zero pivot 2.22045e-14 >> > matrix ordering: nd >> > factor fill ratio given 0, needed 0 >> > Factored matrix follows: >> > Matrix Object: 1 MPI processes >> > type: seqaij >> > rows=11760, cols=11760 >> > package used to perform factorization: superlu >> > total: nonzeros=0, allocated nonzeros=0 >> > total number of mallocs used during MatSetValues calls =0 >> > SuperLU run parameters: >> > Equil: NO >> > ColPerm: 3 >> > IterRefine: 0 >> > SymmetricMode: NO >> > DiagPivotThresh: 1 >> > PivotGrowth: NO >> > ConditionNumber: NO >> > RowPerm: 0 >> > ReplaceTinyPivot: NO >> > PrintStat: NO >> > lwork: 0 >> > linear system matrix = precond matrix: >> > Matrix Object: 1 MPI processes >> > type: seqaij >> > rows=11760, cols=11760 >> > total: nonzeros=506586, allocated nonzeros=509061 >> > total number of mallocs used during MatSetValues calls =0 >> > not using I-node routines >> > >> > >> > KSPView() before parallel solve: >> > >> > KSP Object: 2 MPI processes >> > type: preonly >> > maximum iterations=10000, initial guess is zero >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> > left preconditioning >> > using DEFAULT norm type for convergence test >> > PC Object: 2 MPI processes >> > type: lu >> > LU: out-of-place factorization >> > tolerance for zero pivot 2.22045e-14 >> > matrix ordering: natural >> > linear system matrix = precond matrix: >> > Solving Electron Matrix Equation >> > Matrix Object: 2 MPI processes >> > type: mpiaij >> > rows=11760, cols=11760 >> > total: nonzeros=506586, allocated nonzeros=520821 >> > total number of mallocs used during MatSetValues calls =0 >> > not using I-node (on process 0) routines >> > >> > Crash info for parallel solve: >> > >> > [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > [1]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > [1]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSCERROR: or try >> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >> corruption errors >> > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >> and run >> > [1]PETSC ERROR: to get more information on the crash. >> > [1]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> > [1]PETSC ERROR: Signal received! >> > [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 6, Mon Feb 11 >> 12:26:34 CST 2013 >> > [1]PETSC ERROR: See docs/changes/index.html for recent updates. >> > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> > [1]PETSC ERROR: See docs/index.html for manual pages. >> > [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [1]PETSC ERROR: on a path-ompi named >> > [1]PETSC ERROR: Libraries linked from >> > [1]PETSC ERROR: Configure run at Thu Mar 21 14:19:42 2013 >> > [1]PETSC ERROR: Configure options --PETSC_ARCH=path-ompi >> --PETSC_DIR= --CFLAGS="-fPIC -O -mp" --CXXFLAGS="-fPIC -O -mp" >> --FFLAGS="-fPIC -O -mp" --with-debugging=0 --with-dynamic-loadin=no >> --with-mpi=1 --with-mpi-dir= --with-superlu=1 >> --with-superlu-dir= --with-blas-lapack-lib="" >> --with-scalapack=1 --with-scalapack-dir= --with-superlu_dist=1 >> --with-superlu_dist-dir= --with-metis=1 >> --with-metis-dir= --with-parmetis=1 >> --with-parmetis-dir= --with-blacs-lib="" >> --with-blacs-include= --with-hypre=1 --download-hypre=1 >> > [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [1]PETSC ERROR: User provided function() line 0 in unknown directory >> unknown file >> > >> > >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 26 10:39:05 2013 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 26 Jun 2013 17:39:05 +0200 Subject: [petsc-users] SEGV on KSPSolve with mutliple processors In-Reply-To: References: <73f592c2f3c54877b69dbe8001dd6f9b@CSGHUB209W.pu.win.princeton.edu> Message-ID: On Wed, Jun 26, 2013 at 5:15 PM, Brendan C Lyons wrote: > Dear all, > > Sorry for the delayed response. It took me sometime to get a debug > version available on the clusters I used. I reran my code compiled with > debugging turned on and ran it with valgrind. No error was caught before > the call to KSPSolve() and I've now received the error message below (I put > it only for one of the four processors here, but I believe it's the same > for all of them). Any further advice that could help me track down the > cause for this segfault would be appreciated. > I see at least three scenarios: 1) SuperLU_Dist has a bug You should try MUMPS 2) Our interface to SuperLU_Dist has a bug Can you run in the debugger and get a stack trace for this? 3) You have another bug somewhere writing over memory You have run valgrind, so this is unlikely Matt > Thanks, > > ~Brendan > > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [3]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [3]PETSC ERROR: likely location of problem given in stack below > [3]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [3]PETSC ERROR: INSTEAD the line number of the start of the function > [3]PETSC ERROR: is given. > [3]PETSC ERROR: [3] MatLUFactorNumeric_SuperLU_DIST line 284 > src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c > [3]PETSC ERROR: [3] MatLUFactorNumeric line 2791 src/mat/interface/matrix.c > [3]PETSC ERROR: [3] PCSetUp_LU line 108 src/ksp/pc/impls/factor/lu/lu.c > [3]PETSC ERROR: [3] PCSetUp line 810 src/ksp/pc/interface/precon.c > [3]PETSC ERROR: [3] KSPSetUp line 182 src/ksp/ksp/interface/itfunc.c > [3]PETSC ERROR: [3] KSPSolve line 351 src/ksp/ksp/interface/itfunc.c > [3]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [3]PETSC ERROR: Signal received! > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 6, Mon Feb 11 12:26:34 > CST 2013 > [3]PETSC ERROR: See docs/changes/index.html for recent updates. > [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [3]PETSC ERROR: See docs/index.html for manual pages. > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: N4D_d.exe on a path-ompi named [REDACTED] by blyons Tue > Jun 25 13:50:35 2013 > [3]PETSC ERROR: Libraries linked from ${PETSC_DIR}/path-ompi/lib > [3]PETSC ERROR: Configure run at Mon Jun 24 13:11:36 2013 > [3]PETSC ERROR: Configure options --PETSC_ARCH=path-ompi > --PETSC_DIR=${PATHSCALE_DIR}/openmpi-1.6-pkgs/petsc-3.3-p6-debug --CFLAGS="-fPIC > -O -mp" --CXXFLAGS="-fPIC -O -mp" --FFLAGS="-fPIC -O -mp" > --with-debugging=1 --with-dynamic-loading=no --with-mpi=1 > --with-mpi-dir=${PATHSCALE_DIR}/openmpi-1.6.4 --with-superlu=1 > --with-superlu-dir=${PATHSCALE_DIR}/superlu-4.3 > --with-blas-lapack-lib="[${PATHSCALE_DIR}/acml-5.3.0/open64_64/lib/libacml.a]--with-metis=1" > --with-metis-dir=${PATHSCALE_DIR}/metis-5.0.3 --with-parmetis=1 > --with-parmetis-dir=${PATHSCALE_DIR}/openmpi-1.6-pkgs/parmetis-4.0.2 > --with-blas-lapack-lib="${PATHSCALE_DIR}/acml-5.3.0/open64_64/lib/libacml.a]" > --with-blacs=1 > --with-blacs-lib="[${PATHSCALE_DIR}/openmpi-1.6-pkgs/blacs-1.1p3/lib/blacsCinit_MPI-LINUX-0.a,${PATHSCALE_DIR}/openmpi-1.6-pkgs/blacs-1.1p3/lib/blacsF77init_MPI-LINUX-0.a,${PATHSCALE_DIR}/openmpi-1.6-pkgs/blacs-1.1p3/lib/blacs_MPI-LINUX-0.a]" > --with-blacs-include=${PATHSCALE_DIR}/openmpi-1.6-pkgs/blacs-1.1p3/include > --with-hypre=1 --download-hypre=1 --with-scalapack=1 > --with-scalapack-dir=${PATHSCALE_DIR}/openmpi-1.6-pkgs/scalapack-2.0.2 > --with-superlu_dist=1 > --with-superlu_dist-dir=${PATHSCALE_DIR}/openmpi-1.6-pkgs/superlu_dist-3.2 > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > ==421== Thread 16: > ==421== Invalid free() / delete / delete[] / realloc() > ==421== at 0x4C2333A: free (vg_replace_malloc.c:446) > ==421== by 0x9CB25AA: free_mem (in /lib64/libc-2.5.so) > ==421== by 0x9CB21A1: __libc_freeres (in /lib64/libc-2.5.so) > ==421== by 0x4A1E669: _vgnU_freeres (vg_preloaded.c:62) > ==421== Address 0x4165e78 is not stack'd, malloc'd or (recently) free'd > ==421== > > > > On Tue, Jun 18, 2013 at 4:52 PM, Barry Smith wrote: > >> >> If possible you would also benefit from running the debug version >> under valgrind >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind it is >> possible memory corruption has taken place before the point where the code >> crashes, valgrind will help identify any memory corruption as soon as it >> takes place. >> >> Barry >> >> On Jun 18, 2013, at 2:15 PM, Dave May wrote: >> >> > You should recompile your code using a debug build of petsc so you get >> some meaningful info from the stack trace when the Segv occurs. >> > >> > Dave >> > >> > >> > On Tuesday, 18 June 2013, Brendan C Lyons wrote: >> > Hi everyone, >> > >> > I've run into a strange problem in my Fortran 90 code where it runs >> fine with 1 processor, but then throws a segmentation fault on KSPSolve() >> when I try to run it in parallel. I'm using PETSc 3.3 with the SuperLU >> direct solver for the sequential case and SuperLU_dist for the parallel >> case. I've called KSPView before and after KSPSolve. I'll put the KSPView >> output for the sequential and parallel cases and the crash info for the >> parallel case below (with some details of my system redacted). Any help >> would be appreciated. If you need any other information, I'm happy to >> provide it. >> > >> > Thank you, >> > >> > ~Brendan >> > ------------------------------ >> > >> > KSPView() before sequential solve: >> > >> > KSP Object: 1 MPI processes >> > type: preonly >> > maximum iterations=10000, initial guess is zero >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> > left preconditioning >> > using DEFAULT norm type for convergence test >> > PC Object: 1 MPI processes >> > type: lu >> > LU: out-of-place factorization >> > tolerance for zero pivot 2.22045e-14 >> > matrix ordering: nd >> > linear system matrix = precond matrix: >> > Matrix Object: 1 MPI processes >> > type: seqaij >> > rows=11760, cols=11760 >> > total: nonzeros=506586, allocated nonzeros=509061 >> > total number of mallocs used during MatSetValues calls =0 >> > not using I-node routines >> > >> > KSPView() after sequential solve: >> > >> > KSP Object: 1 MPI processes >> > type: preonly >> > maximum iterations=10000, initial guess is zero >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> > left preconditioning >> > using NONE norm type for convergence test >> > PC Object: 1 MPI processes >> > type: lu >> > LU: out-of-place factorization >> > tolerance for zero pivot 2.22045e-14 >> > matrix ordering: nd >> > factor fill ratio given 0, needed 0 >> > Factored matrix follows: >> > Matrix Object: 1 MPI processes >> > type: seqaij >> > rows=11760, cols=11760 >> > package used to perform factorization: superlu >> > total: nonzeros=0, allocated nonzeros=0 >> > total number of mallocs used during MatSetValues calls =0 >> > SuperLU run parameters: >> > Equil: NO >> > ColPerm: 3 >> > IterRefine: 0 >> > SymmetricMode: NO >> > DiagPivotThresh: 1 >> > PivotGrowth: NO >> > ConditionNumber: NO >> > RowPerm: 0 >> > ReplaceTinyPivot: NO >> > PrintStat: NO >> > lwork: 0 >> > linear system matrix = precond matrix: >> > Matrix Object: 1 MPI processes >> > type: seqaij >> > rows=11760, cols=11760 >> > total: nonzeros=506586, allocated nonzeros=509061 >> > total number of mallocs used during MatSetValues calls =0 >> > not using I-node routines >> > >> > >> > KSPView() before parallel solve: >> > >> > KSP Object: 2 MPI processes >> > type: preonly >> > maximum iterations=10000, initial guess is zero >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> > left preconditioning >> > using DEFAULT norm type for convergence test >> > PC Object: 2 MPI processes >> > type: lu >> > LU: out-of-place factorization >> > tolerance for zero pivot 2.22045e-14 >> > matrix ordering: natural >> > linear system matrix = precond matrix: >> > Solving Electron Matrix Equation >> > Matrix Object: 2 MPI processes >> > type: mpiaij >> > rows=11760, cols=11760 >> > total: nonzeros=506586, allocated nonzeros=520821 >> > total number of mallocs used during MatSetValues calls =0 >> > not using I-node (on process 0) routines >> > >> > Crash info for parallel solve: >> > >> > [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > [1]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > [1]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSCERROR: or try >> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >> corruption errors >> > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >> and run >> > [1]PETSC ERROR: to get more information on the crash. >> > [1]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> > [1]PETSC ERROR: Signal received! >> > [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 6, Mon Feb 11 >> 12:26:34 CST 2013 >> > [1]PETSC ERROR: See docs/changes/index.html for recent updates. >> > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> > [1]PETSC ERROR: See docs/index.html for manual pages. >> > [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [1]PETSC ERROR: on a path-ompi named >> > [1]PETSC ERROR: Libraries linked from >> > [1]PETSC ERROR: Configure run at Thu Mar 21 14:19:42 2013 >> > [1]PETSC ERROR: Configure options --PETSC_ARCH=path-ompi >> --PETSC_DIR= --CFLAGS="-fPIC -O -mp" --CXXFLAGS="-fPIC -O -mp" >> --FFLAGS="-fPIC -O -mp" --with-debugging=0 --with-dynamic-loadin=no >> --with-mpi=1 --with-mpi-dir= --with-superlu=1 >> --with-superlu-dir= --with-blas-lapack-lib="" >> --with-scalapack=1 --with-scalapack-dir= --with-superlu_dist=1 >> --with-superlu_dist-dir= --with-metis=1 >> --with-metis-dir= --with-parmetis=1 >> --with-parmetis-dir= --with-blacs-lib="" >> --with-blacs-include= --with-hypre=1 --download-hypre=1 >> > [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [1]PETSC ERROR: User provided function() line 0 in unknown directory >> unknown file >> > >> > >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shuangshuang.Jin at pnnl.gov Wed Jun 26 17:37:21 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Wed, 26 Jun 2013 15:37:21 -0700 Subject: [petsc-users] ELEMENTAL Matrix Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FD9EDD08A@EMAIL04.pnl.gov> Hello, can anyone tell me how does a MATELEMENTAL type matrix store its values in PETSC across different processors? For example, if I have a 4X4 ELEMENTAL matrix, when I use 1 processor, all of the element are stored on p0. p0 p0 p0 p0 p0 p0 p0 p0 p0 p0 p0 p0 p0 p0 p0 p0 What about 2 processor, is it something as below? p0 p0 p1 p1 p0 p0 p1 p1 p0 p0 p1 p1 p0 p0 p1 p1 And 4 processor? p0 p2 p0 p2 p1 p3 p1 p3 p0 p2 p0 p2 p1 p3 p1 p3 Finally, if I have 12x12 Elemental matrix with 4 processors, the data will be stored as the following? p0 p0 p0 p2 p2 p2 p0 p0 p0 p2 p2 p2 p0 p0 p0 p2 p2 p2 p0 p0 p0 p2 p2 p2 p0 p0 p0 p2 p2 p2 p0 p0 p0 p2 p2 p2 p1 p1 p1 p3 p3 p3 p1 p1 p1 p3 p3 p3 p1 p1 p1 p3 p3 p3 p1 p1 p1 p3 p3 p3 p1 p1 p1 p3 p3 p3 p1 p1 p1 p3 p3 p3 p0 p0 p0 p2 p2 p2 p0 p0 p0 p2 p2 p2 p0 p0 p0 p2 p2 p2 p0 p0 p0 p2 p2 p2 p0 p0 p0 p2 p2 p2 p0 p0 p0 p2 p2 p2 p1 p1 p1 p3 p3 p3 p1 p1 p1 p3 p3 p3 p1 p1 p1 p3 p3 p3 p1 p1 p1 p3 p3 p3 p1 p1 p1 p3 p3 p3 p1 p1 p1 p3 p3 p3 I'm trying to find out a clue to set up the values of my 12x12 Elemental Matrix correctly. Thanks, Shuangshuang -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Jun 26 17:49:34 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 26 Jun 2013 17:49:34 -0500 Subject: [petsc-users] ELEMENTAL Matrix In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FD9EDD08A@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD9EDD08A@EMAIL04.pnl.gov> Message-ID: <87ip10jxmp.fsf@mcs.anl.gov> "Jin, Shuangshuang" writes: > Hello, can anyone tell me how does a MATELEMENTAL type matrix store its values in PETSC across different processors? > > For example, if I have a 4X4 ELEMENTAL matrix, when I use 1 processor, all of the element are stored on p0. > > p0 p0 p0 p0 > p0 p0 p0 p0 > p0 p0 p0 p0 > p0 p0 p0 p0 > > What about 2 processor, is it something as below? > > p0 p0 p1 p1 > p0 p0 p1 p1 > p0 p0 p1 p1 > p0 p0 p1 p1 > > And 4 processor? > > p0 p2 p0 p2 > p1 p3 p1 p3 > p0 p2 p0 p2 > p1 p3 p1 p3 Elemental uses element-cyclic numbering internally, defined by the process Grid, which you can change using -mat_elemental_grid_height. The be compatible with PETSc vector ordering, we interpret the Elemental matrix indices via a permutation. You should use MatGetOwnershipIS to find out which rows and columns are owned locally. Note that in general, these will not have anything to do with the parts of the vector that are owned locally, but the indexing that you use when calling MatSetValues is the same as with any other PETSc matrix. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From Shuangshuang.Jin at pnnl.gov Wed Jun 26 17:55:07 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Wed, 26 Jun 2013 15:55:07 -0700 Subject: [petsc-users] ELEMENTAL Matrix In-Reply-To: <87ip10jxmp.fsf@mcs.anl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD9EDD08A@EMAIL04.pnl.gov> <87ip10jxmp.fsf@mcs.anl.gov> Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FD9EDD091@EMAIL04.pnl.gov> Hi, Jed, could you please forward your replied email to my sjinwsu at gmail.com address? I can only read a portion of it as shown below. It seems like there's an attachment in your email which cannot pass the security check on my side. Thanks, Shuangshuang -----Original Message----- From: Jed Brown [mailto:five9a2 at gmail.com] On Behalf Of Jed Brown Sent: Wednesday, June 26, 2013 3:50 PM To: Jin, Shuangshuang; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] ELEMENTAL Matrix "Jin, Shuangshuang" writes: > Hello, can anyone tell me how does a MATELEMENTAL type matrix store its values in PETSC across different processors? > > For example, if I have a 4X4 ELEMENTAL matrix, when I use 1 processor, From jedbrown at mcs.anl.gov Wed Jun 26 17:58:11 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 26 Jun 2013 17:58:11 -0500 Subject: [petsc-users] ELEMENTAL Matrix In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FD9EDD091@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FD9EDD08A@EMAIL04.pnl.gov> <87ip10jxmp.fsf@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FD9EDD091@EMAIL04.pnl.gov> Message-ID: <87bo6sjx8c.fsf@mcs.anl.gov> "Jin, Shuangshuang" writes: > Hi, Jed, could you please forward your replied email to my > sjinwsu at gmail.com address? I can only read a portion of it as shown > below. It seems like there's an attachment in your email which cannot > pass the security check on my side. That was a PGP/MIME signature. http://lists.mcs.anl.gov/pipermail/petsc-users/2013-June/017816.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From thomas.de-soza at edf.fr Thu Jun 27 04:08:27 2013 From: thomas.de-soza at edf.fr (Thomas DE-SOZA) Date: Thu, 27 Jun 2013 11:08:27 +0200 Subject: [petsc-users] Unreproducible number of iterations for consecutives solves (GCR + ML) Message-ID: Dear PETSc users, We've been using the Krylov solvers in PETSc for a while in our software package for implicit structural mechanics and recently added algebraic multigrid preconditioning through the ML and Hypre libraries provided as part of PETSc. Though we're quite happy with this new feature, we've encountered a strange behaviour when using KSPGCR in conjunction with the PCML preconditioner. Indeed, identical and consecutives solves inside the same run do not display the same number of Krylov iterations even in sequential and we were wondering why : 1st solve 723 KSP unpreconditioned resid norm 2.911385065051e-02 true resid norm 2.911385065051e-02 ||r(i)||/||b|| 9.979426047969e-09 2nd solve 787 KSP unpreconditioned resid norm 2.896035212670e-02 true resid norm 2.896035212670e-02 ||r(i)||/||b|| 9.926810982197e-09 3rd solve 721 KSP unpreconditioned resid norm 2.913123687343e-02 true resid norm 2.913123687343e-02 ||r(i)||/||b|| 9.985385566274e-09 Would you say this case requires a great number of iterations and therefore reproductibility is not ensured (indeed it is ill-conditioned and is not very suited for algebraic multigrid) ? Or is there a seed somewhere in ML that would explain this ? I searched the ML manual for that and couln't find any. We're using the uncoupled coarsening scheme in ML (call PetscOptionsSetValue('-pc_ml_CoarsenScheme', 'Uncoupled', ierr)). Final notes : several consecutive runs of the program do display identical behaviour (that is the 1st solve always require 723 iterations, the 2nd 787, etc). Moreover other cases that are well-conditioned require the same number number of iterations for each consecutive solve (though the converged residual differs a bit). Thanks for any hints, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Thu Jun 27 04:21:24 2013 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 27 Jun 2013 11:21:24 +0200 Subject: [petsc-users] Unreproducible number of iterations for consecutives solves (GCR + ML) In-Reply-To: References: Message-ID: Hi Thomas, Does this behaviour only occur when using ml, or do you see this with other preconditioners as well? On Thursday, 27 June 2013, Thomas DE-SOZA wrote: > > Dear PETSc users, > > We've been using the Krylov solvers in PETSc for a while in our software > package for implicit structural mechanics and recently added algebraic > multigrid preconditioning through the ML and Hypre libraries provided as > part of PETSc. > Though we're quite happy with this new feature, we've encountered a > strange behaviour when using KSPGCR in conjunction with the PCML > preconditioner. > Indeed, identical and consecutives solves inside the same run do not > display the same number of Krylov iterations even in sequential and we were > wondering why : > > 1st solve > 723 KSP unpreconditioned resid norm 2.911385065051e-02 true resid norm > 2.911385065051e-02 ||r(i)||/||b|| 9.979426047969e-09 > > 2nd solve > 787 KSP unpreconditioned resid norm 2.896035212670e-02 true resid norm > 2.896035212670e-02 ||r(i)||/||b|| 9.926810982197e-09 > > 3rd solve > 721 KSP unpreconditioned resid norm 2.913123687343e-02 true resid norm > 2.913123687343e-02 ||r(i)||/||b|| 9.985385566274e-09 > > > Would you say this case requires a great number of iterations and > therefore reproductibility is not ensured (indeed it is ill-conditioned and > is not very suited for algebraic multigrid) ? > Or is there a seed somewhere in ML that would explain this ? I searched > the ML manual for that and couln't find any. We're using the uncoupled > coarsening scheme in ML (call PetscOptionsSetValue('-pc_ml_CoarsenScheme', > 'Uncoupled', ierr)). > > Final notes : several consecutive runs of the program do display identical > behaviour (that is the 1st solve always require 723 iterations, the 2nd > 787, etc). Moreover other cases that are well-conditioned require the same > number number of iterations for each consecutive solve (though the > converged residual differs a bit). > > Thanks for any hints, > Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.de-soza at edf.fr Thu Jun 27 04:38:05 2013 From: thomas.de-soza at edf.fr (Thomas DE-SOZA) Date: Thu, 27 Jun 2013 11:38:05 +0200 Subject: [petsc-users] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A__Unreproducible_n?= =?iso-8859-1?q?umber_of_iterations_for_consecutives_solves_=28GCR_+_ML=29?= In-Reply-To: Message-ID: Hi Dave, Strangely, it only occurs when we use ML. Other preconditioners we have tested so far include BoomerAMG from Hypre and a homemade preconditioner based on a single precision factorization with MUMPS. Both are OK. Thomas dave.mayhem23 at gmail.com 27/06/2013 11:21 Pour : thomas.de-soza at edf.fr cc : petsc-users at mcs.anl.gov Objet : Re: [petsc-users] Unreproducible number of iterations for consecutives solves (GCR + ML) Hi Thomas, Does this behaviour only occur when using ml, or do you see this with other preconditioners as well? On Thursday, 27 June 2013, Thomas DE-SOZA wrote: Dear PETSc users, We've been using the Krylov solvers in PETSc for a while in our software package for implicit structural mechanics and recently added algebraic multigrid preconditioning through the ML and Hypre libraries provided as part of PETSc. Though we're quite happy with this new feature, we've encountered a strange behaviour when using KSPGCR in conjunction with the PCML preconditioner. Indeed, identical and consecutives solves inside the same run do not display the same number of Krylov iterations even in sequential and we were wondering why : 1st solve 723 KSP unpreconditioned resid norm 2.911385065051e-02 true resid norm 2.911385065051e-02 ||r(i)||/||b|| 9.979426047969e-09 2nd solve 787 KSP unpreconditioned resid norm 2.896035212670e-02 true resid norm 2.896035212670e-02 ||r(i)||/||b|| 9.926810982197e-09 3rd solve 721 KSP unpreconditioned resid norm 2.913123687343e-02 true resid norm 2.913123687343e-02 ||r(i)||/||b|| 9.985385566274e-09 Would you say this case requires a great number of iterations and therefore reproductibility is not ensured (indeed it is ill-conditioned and is not very suited for algebraic multigrid) ? Or is there a seed somewhere in ML that would explain this ? I searched the ML manual for that and couln't find any. We're using the uncoupled coarsening scheme in ML (call PetscOptionsSetValue('-pc_ml_CoarsenScheme', 'Uncoupled', ierr)). Final notes : several consecutive runs of the program do display identical behaviour (that is the 1st solve always require 723 iterations, the 2nd 787, etc). Moreover other cases that are well-conditioned require the same number number of iterations for each consecutive solve (though the converged residual differs a bit). Thanks for any hints, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Thu Jun 27 09:40:05 2013 From: mark.adams at columbia.edu (Mark F. Adams) Date: Thu, 27 Jun 2013 09:40:05 -0500 Subject: [petsc-users] Unreproducible number of iterations for consecutives solves (GCR + ML) In-Reply-To: References: Message-ID: If this happens in serial and does not happen with other solvers then there is probably a non-deterministic random seed (RHS for the eigen estimator...). I could imagine the documentation not going into that much detail, but I would not imagine that ML would be non-determinstic by default ... It could be a floating point issues also. On Jun 27, 2013, at 4:08 AM, Thomas DE-SOZA wrote: > > Dear PETSc users, > > We've been using the Krylov solvers in PETSc for a while in our software package for implicit structural mechanics and recently added algebraic multigrid preconditioning through the ML and Hypre libraries provided as part of PETSc. > Though we're quite happy with this new feature, we've encountered a strange behaviour when using KSPGCR in conjunction with the PCML preconditioner. > Indeed, identical and consecutives solves inside the same run do not display the same number of Krylov iterations even in sequential and we were wondering why : > > 1st solve > 723 KSP unpreconditioned resid norm 2.911385065051e-02 true resid norm 2.911385065051e-02 ||r(i)||/||b|| 9.979426047969e-09 > > 2nd solve > 787 KSP unpreconditioned resid norm 2.896035212670e-02 true resid norm 2.896035212670e-02 ||r(i)||/||b|| 9.926810982197e-09 > > 3rd solve > 721 KSP unpreconditioned resid norm 2.913123687343e-02 true resid norm 2.913123687343e-02 ||r(i)||/||b|| 9.985385566274e-09 > > > Would you say this case requires a great number of iterations and therefore reproductibility is not ensured (indeed it is ill-conditioned and is not very suited for algebraic multigrid) ? > Or is there a seed somewhere in ML that would explain this ? I searched the ML manual for that and couln't find any. We're using the uncoupled coarsening scheme in ML (call PetscOptionsSetValue('-pc_ml_CoarsenScheme', 'Uncoupled', ierr)). > > Final notes : several consecutive runs of the program do display identical behaviour (that is the 1st solve always require 723 iterations, the 2nd 787, etc). Moreover other cases that are well-conditioned require the same number number of iterations for each consecutive solve (though the converged residual differs a bit). > > Thanks for any hints, > Thomas > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. > > E-mail communication cannot be guaranteed to be timely secure, error or virus-free. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rupp at mcs.anl.gov Thu Jun 27 10:55:07 2013 From: rupp at mcs.anl.gov (Karl Rupp) Date: Thu, 27 Jun 2013 10:55:07 -0500 Subject: [petsc-users] Unreproducible number of iterations for consecutives solves (GCR + ML) In-Reply-To: References: Message-ID: <51CC605B.6040009@mcs.anl.gov> Hi, running a `find externalpackages/ml/src/ | xargs grep rand` returns quite a number of hits, for example src/Coarsen/ml_agg_uncoupled.c: ML_randomize(Nrows, randomVector); src/Coarsen/ml_aggregate.c: ML_random_vec(&dtemp,1,Amat->comm); src/Coarsen/ml_aggregate.c: ML_random_vec(&dtemp,1,Amat->comm); src/Coarsen/ml_aggregate.c: ML_random_global_subset(mat, 5.,&the_list, &the_length, num_PDE_eqns); src/Utils/ml_MultiLevelPreconditioner.cpp: // put random values src/Utils/ml_MultiLevelPreconditioner.cpp: field_of_values->R_coeff[0] = (double)rand()/RAND_MAX; src/Utils/ml_MultiLevelPreconditioner.cpp: field_of_values->R_coeff[1] = (double)rand()/RAND_MAX; src/Utils/ml_MultiLevelPreconditioner.cpp: field_of_values->R_coeff[2] = (double)rand()/RAND_MAX; src/Utils/ml_MultiLevelPreconditioner.cpp: field_of_values->P_coeff[0] = (double)rand()/RAND_MAX; src/Utils/ml_MultiLevelPreconditioner.cpp: field_of_values->P_coeff[1] = (double)rand()/RAND_MAX; src/Utils/ml_MultiLevelPreconditioner.cpp: field_of_values->P_coeff[2] = (double)rand()/RAND_MAX; Thus, it seems that there is a little share of randomness involved, which may only be significant when the solver needs to run a large number of iterations. So, the fluctuation of iterations when using ML does not seem to be something you need to be worried about. Best regards, Karli On 06/27/2013 09:40 AM, Mark F. Adams wrote: > If this happens in serial and does not happen with other solvers then > there is probably a non-deterministic random seed (RHS for the eigen > estimator...). I could imagine the documentation not going into that > much detail, but I would not imagine that ML would be non-determinstic > by default ... It could be a floating point issues also. > > On Jun 27, 2013, at 4:08 AM, Thomas DE-SOZA > wrote: > >> >> Dear PETSc users, >> >> We've been using the Krylov solvers in PETSc for a while in our >> software package for implicit structural mechanics and recently added >> algebraic multigrid preconditioning through the ML and Hypre libraries >> provided as part of PETSc. >> Though we're quite happy with this new feature, we've encountered a >> strange behaviour when using KSPGCR in conjunction with the PCML >> preconditioner. >> Indeed, identical and consecutives solves inside the same run do not >> display the same number of Krylov iterations even in sequential and we >> were wondering why : >> >> 1st solve >> 723 KSP unpreconditioned resid norm 2.911385065051e-02 true resid norm >> 2.911385065051e-02 ||r(i)||/||b|| 9.979426047969e-09 >> >> 2nd solve >> 787 KSP unpreconditioned resid norm 2.896035212670e-02 true resid norm >> 2.896035212670e-02 ||r(i)||/||b|| 9.926810982197e-09 >> >> 3rd solve >> 721 KSP unpreconditioned resid norm 2.913123687343e-02 true resid norm >> 2.913123687343e-02 ||r(i)||/||b|| 9.985385566274e-09 >> >> >> Would you say this case requires a great number of iterations and >> therefore reproductibility is not ensured (indeed it is >> ill-conditioned and is not very suited for algebraic multigrid) ? >> Or is there a seed somewhere in ML that would explain this ? I >> searched the ML manual for that and couln't find any. We're using the >> uncoupled coarsening scheme in ML (call >> PetscOptionsSetValue('-pc_ml_CoarsenScheme', 'Uncoupled', ierr)). >> >> Final notes : several consecutive runs of the program do display >> identical behaviour (that is the 1st solve always require 723 >> iterations, the 2nd 787, etc). Moreover other cases that are >> well-conditioned require the same number number of iterations for each >> consecutive solve (though the converged residual differs a bit). >> >> Thanks for any hints, >> Thomas >> >> >> Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont >> ?tablis ? l'intention exclusive des destinataires et les informations >> qui y figurent sont strictement confidentielles. Toute utilisation de >> ce Message non conforme ? sa destination, toute diffusion ou toute >> publication totale ou partielle, est interdite sauf autorisation expresse. >> >> Si vous n'?tes pas le destinataire de ce Message, il vous est interdit >> de le copier, de le faire suivre, de le divulguer ou d'en utiliser >> tout ou partie. Si vous avez re?u ce Message par erreur, merci de le >> supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en >> garder aucune trace sur quelque support que ce soit. Nous vous >> remercions ?galement d'en avertir imm?diatement l'exp?diteur par >> retour du message. >> >> Il est impossible de garantir que les communications par messagerie >> ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de >> toute erreur ou virus. >> ____________________________________________________ >> >> This message and any attachments (the 'Message') are intended solely >> for the addressees. The information contained in this Message is >> confidential. Any use of information contained in this Message not in >> accord with its purpose, any dissemination or disclosure, either whole >> or partial, is prohibited except formal approval. >> >> If you are not the addressee, you may not copy, forward, disclose or >> use any part of it. If you have received this message in error, please >> delete it and all copies from your system and notify the sender >> immediately by return message. >> >> E-mail communication cannot be guaranteed to be timely secure, error >> or virus-free. >> > From jedbrown at mcs.anl.gov Thu Jun 27 11:13:44 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 27 Jun 2013 11:13:44 -0500 Subject: [petsc-users] Unreproducible number of iterations for consecutives solves (GCR + ML) In-Reply-To: <51CC605B.6040009@mcs.anl.gov> References: <51CC605B.6040009@mcs.anl.gov> Message-ID: <87obarilaf.fsf@mcs.anl.gov> Karl Rupp writes: > Thus, it seems that there is a little share of randomness involved, > which may only be significant when the solver needs to run a large > number of iterations. So, the fluctuation of iterations when using ML > does not seem to be something you need to be worried about. Yes, ML has some randomness. It sets a deterministic seed so the first solve is reproducible, but multiple solves will vary some: diff --git i/src/ksp/ksp/examples/tutorials/ex56.c w/src/ksp/ksp/examples/tutorials/ex56.c index 91fad9d..fa8860c 100644 --- i/src/ksp/ksp/examples/tutorials/ex56.c +++ w/src/ksp/ksp/examples/tutorials/ex56.c @@ -284,7 +284,7 @@ int main(int argc,char **args) PetscReal emax, emin; ierr = MaybeLogStagePush(stage[2]);CHKERRQ(ierr); /* PC setup basically */ - ierr = MatScale(Amat, 100000.0);CHKERRQ(ierr); + //ierr = MatScale(Amat, 100000.0);CHKERRQ(ierr); ierr = KSPSetOperators(ksp, Amat, Amat, SAME_NONZERO_PATTERN);CHKERRQ(ierr); ierr = KSPSetUp(ksp);CHKERRQ(ierr); @@ -297,7 +297,7 @@ int main(int argc,char **args) ierr = MaybeLogStagePush(stage[4]);CHKERRQ(ierr); /* 3rd solve */ - ierr = MatScale(Amat, 100000.0);CHKERRQ(ierr); + //ierr = MatScale(Amat, 100000.0);CHKERRQ(ierr); ierr = KSPSetOperators(ksp, Amat, Amat, SAME_NONZERO_PATTERN);CHKERRQ(ierr); ierr = KSPSetUp(ksp);CHKERRQ(ierr); $ mpiexec -n 8 ./ex56 -ne 19 -alpha 1.e-3 -ksp_monitor -ksp_type cg -ksp_max_it 50 -pc_type ml -ksp_converged_reason -mg_levels_ksp_type chebyshev -mg_levels_ksp_chebyshev_estimate_eigenvalues 0,0.05,0,1.05 -mg_levels_pc_type jacobi -two_solves -ksp_norm_type unpreconditioned 0 KSP Residual norm 3.916254888964e+00 1 KSP Residual norm 2.881812838115e+01 2 KSP Residual norm 1.275966743250e+01 3 KSP Residual norm 4.773599170045e+00 4 KSP Residual norm 1.935268484585e+00 5 KSP Residual norm 4.209580628708e-01 6 KSP Residual norm 8.134351555855e-02 7 KSP Residual norm 2.105221695641e-02 8 KSP Residual norm 6.251570402725e-03 9 KSP Residual norm 1.693062086081e-03 10 KSP Residual norm 4.088111376292e-04 11 KSP Residual norm 8.282470497154e-05 12 KSP Residual norm 2.910468516666e-05 Linear solve converged due to CONVERGED_RTOL iterations 12 0 KSP Residual norm 3.916254888964e+00 1 KSP Residual norm 2.884447434423e+01 2 KSP Residual norm 1.279637972963e+01 3 KSP Residual norm 4.778606784425e+00 4 KSP Residual norm 1.945306501447e+00 5 KSP Residual norm 4.252458287438e-01 6 KSP Residual norm 8.200469618341e-02 7 KSP Residual norm 2.128564125382e-02 8 KSP Residual norm 6.325422255484e-03 9 KSP Residual norm 1.722774707230e-03 10 KSP Residual norm 4.155306725796e-04 11 KSP Residual norm 8.432468795266e-05 12 KSP Residual norm 2.965043662043e-05 Linear solve converged due to CONVERGED_RTOL iterations 12 0 KSP Residual norm 3.916254888964e+00 1 KSP Residual norm 2.884581541125e+01 2 KSP Residual norm 1.281469034767e+01 3 KSP Residual norm 4.787719947229e+00 4 KSP Residual norm 1.954021039200e+00 5 KSP Residual norm 4.286214813381e-01 6 KSP Residual norm 8.267288000027e-02 7 KSP Residual norm 2.150072898753e-02 8 KSP Residual norm 6.392547836067e-03 9 KSP Residual norm 1.749098960784e-03 10 KSP Residual norm 4.214004987050e-04 11 KSP Residual norm 8.561869518870e-05 12 KSP Residual norm 3.013123608130e-05 Linear solve converged due to CONVERGED_RTOL iterations 12 [0]main |b-Ax|/|b|=7.693890e-06, |b|=3.916255e+00, emax=9.962246e-01 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From thomas.de-soza at edf.fr Thu Jun 27 11:30:20 2013 From: thomas.de-soza at edf.fr (Thomas DE-SOZA) Date: Thu, 27 Jun 2013 18:30:20 +0200 Subject: [petsc-users] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A__Unreproducible_n?= =?iso-8859-1?q?umber_of_iterations_for_consecutives_solves_=28GCR_+_ML=29?= Message-ID: Dear PETSc developers, As usual your answers are concise and precise. Thanks a lot for the explanation ! Browsing the mailing list archive a bit has made me aware of the native PCGAMG, so we'll update PETSc and give it a try. Keep up the fantastic work, Thomas jedbrown at mcs.anl.gov Envoy? par : five9a2 at gmail.com 27/06/2013 18:13 Pour : rupp at mcs.anl.gov, thomas.de-soza at edf.fr cc : petsc-users at mcs.anl.gov Objet : Re: [petsc-users] Unreproducible number of iterations for consecutives solves (GCR + ML) NOTE : La(les) pi?ce(s) jointe(s) a(ont) ?t? supprim?e(s) du message initial. Le nom est conserv? pour information. Le : 27/06/2013, ? 18:18 Par : Thomas DE-SOZA Nom(s) du(des) fichiers(s) : C.DTF Karl Rupp writes: > Thus, it seems that there is a little share of randomness involved, > which may only be significant when the solver needs to run a large > number of iterations. So, the fluctuation of iterations when using ML > does not seem to be something you need to be worried about. Yes, ML has some randomness. It sets a deterministic seed so the first solve is reproducible, but multiple solves will vary some: diff --git i/src/ksp/ksp/examples/tutorials/ex56.c w/src/ksp/ksp/examples/tutorials/ex56.c index 91fad9d..fa8860c 100644 --- i/src/ksp/ksp/examples/tutorials/ex56.c +++ w/src/ksp/ksp/examples/tutorials/ex56.c @@ -284,7 +284,7 @@ int main(int argc,char **args) PetscReal emax, emin; ierr = MaybeLogStagePush(stage[2]);CHKERRQ(ierr); /* PC setup basically */ - ierr = MatScale(Amat, 100000.0);CHKERRQ(ierr); + //ierr = MatScale(Amat, 100000.0);CHKERRQ(ierr); ierr = KSPSetOperators(ksp, Amat, Amat, SAME_NONZERO_PATTERN);CHKERRQ(ierr); ierr = KSPSetUp(ksp);CHKERRQ(ierr); @@ -297,7 +297,7 @@ int main(int argc,char **args) ierr = MaybeLogStagePush(stage[4]);CHKERRQ(ierr); /* 3rd solve */ - ierr = MatScale(Amat, 100000.0);CHKERRQ(ierr); + //ierr = MatScale(Amat, 100000.0);CHKERRQ(ierr); ierr = KSPSetOperators(ksp, Amat, Amat, SAME_NONZERO_PATTERN);CHKERRQ(ierr); ierr = KSPSetUp(ksp);CHKERRQ(ierr); $ mpiexec -n 8 ./ex56 -ne 19 -alpha 1.e-3 -ksp_monitor -ksp_type cg -ksp_max_it 50 -pc_type ml -ksp_converged_reason -mg_levels_ksp_type chebyshev -mg_levels_ksp_chebyshev_estimate_eigenvalues 0,0.05,0,1.05 -mg_levels_pc_type jacobi -two_solves -ksp_norm_type unpreconditioned 0 KSP Residual norm 3.916254888964e+00 1 KSP Residual norm 2.881812838115e+01 2 KSP Residual norm 1.275966743250e+01 3 KSP Residual norm 4.773599170045e+00 4 KSP Residual norm 1.935268484585e+00 5 KSP Residual norm 4.209580628708e-01 6 KSP Residual norm 8.134351555855e-02 7 KSP Residual norm 2.105221695641e-02 8 KSP Residual norm 6.251570402725e-03 9 KSP Residual norm 1.693062086081e-03 10 KSP Residual norm 4.088111376292e-04 11 KSP Residual norm 8.282470497154e-05 12 KSP Residual norm 2.910468516666e-05 Linear solve converged due to CONVERGED_RTOL iterations 12 0 KSP Residual norm 3.916254888964e+00 1 KSP Residual norm 2.884447434423e+01 2 KSP Residual norm 1.279637972963e+01 3 KSP Residual norm 4.778606784425e+00 4 KSP Residual norm 1.945306501447e+00 5 KSP Residual norm 4.252458287438e-01 6 KSP Residual norm 8.200469618341e-02 7 KSP Residual norm 2.128564125382e-02 8 KSP Residual norm 6.325422255484e-03 9 KSP Residual norm 1.722774707230e-03 10 KSP Residual norm 4.155306725796e-04 11 KSP Residual norm 8.432468795266e-05 12 KSP Residual norm 2.965043662043e-05 Linear solve converged due to CONVERGED_RTOL iterations 12 0 KSP Residual norm 3.916254888964e+00 1 KSP Residual norm 2.884581541125e+01 2 KSP Residual norm 1.281469034767e+01 3 KSP Residual norm 4.787719947229e+00 4 KSP Residual norm 1.954021039200e+00 5 KSP Residual norm 4.286214813381e-01 6 KSP Residual norm 8.267288000027e-02 7 KSP Residual norm 2.150072898753e-02 8 KSP Residual norm 6.392547836067e-03 9 KSP Residual norm 1.749098960784e-03 10 KSP Residual norm 4.214004987050e-04 11 KSP Residual norm 8.561869518870e-05 12 KSP Residual norm 3.013123608130e-05 Linear solve converged due to CONVERGED_RTOL iterations 12 [0]main |b-Ax|/|b|=7.693890e-06, |b|=3.916255e+00, emax=9.962246e-01 Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: From suifengls at gmail.com Thu Jun 27 13:52:46 2013 From: suifengls at gmail.com (Longxiang Chen) Date: Thu, 27 Jun 2013 11:52:46 -0700 Subject: [petsc-users] no-debug-mode problem Message-ID: Dear all, I use ksp_solver to solve Ax=b, where A is from an outer loop of PDE. Under debug mode(default), it solves the problem in about 4000 iterations. And the final answer is correct (comparing to another solver). I use intel compiler. The program is in Fortran (by mpif90), except the solver is in c (by mpicc). However, when I re-configure with --with-debugging=0 (the only change), the program terminates in about 30 iterations with the wrong final solution. Thank you. Best regards, Longxiang Chen Do something everyday that gets you closer to being done. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stali at geology.wisc.edu Thu Jun 27 14:23:27 2013 From: stali at geology.wisc.edu (Tabrez Ali) Date: Thu, 27 Jun 2013 15:23:27 -0400 Subject: [petsc-users] no-debug-mode problem In-Reply-To: References: Message-ID: <1d49881d863e511ada2913536d29a259@geology.wisc.edu> Fortran can be tricky. Try to run the program in valgrind and/or recheck all your arguments. I once forgot MatAssemblyType in the call to MatAssembly and the code still ran fine on one machine but failed on other. It is better to test with at least 2-3 compilers (GNU, Solaris and Open64 Fortran/C compilers are all free on Linux). T On 27.06.2013 14:52, Longxiang Chen wrote: > Dear all, > > I use ksp_solver to solve Ax=b, where A is from an outer loop of PDE. > Under debug mode(default), it solves the problem in about 4000 > iterations. > And the final answer is correct (comparing to another solver). > > I use intel compiler. > The program is in Fortran (by mpif90), except the solver is in c (by > mpicc). > > However, when I re-configure with?--with-debugging=0?(the only > change), > the program terminates in about 30 iterations with the wrong final > solution. > > Thank you. > > Best regards, > Longxiang Chen > > Do something everyday that gets you closer to being done. From bsmith at mcs.anl.gov Thu Jun 27 20:42:49 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 27 Jun 2013 20:42:49 -0500 Subject: [petsc-users] no-debug-mode problem In-Reply-To: <1d49881d863e511ada2913536d29a259@geology.wisc.edu> References: <1d49881d863e511ada2913536d29a259@geology.wisc.edu> Message-ID: On Jun 27, 2013, at 2:23 PM, Tabrez Ali wrote: > Fortran can be tricky. > > Try to run the program in valgrind and/or recheck all your arguments. I once forgot MatAssemblyType in the call to MatAssembly and the code still ran fine on one machine but failed on other. It is better to test with at least 2-3 compilers (GNU, Solaris and Open64 Fortran/C compilers are all free on Linux). > > T You can also run both versions with -snes_monitor -ksp_monitor and see if they both start out the same way. Barry > > > On 27.06.2013 14:52, Longxiang Chen wrote: >> Dear all, >> >> I use ksp_solver to solve Ax=b, where A is from an outer loop of PDE. >> Under debug mode(default), it solves the problem in about 4000 >> iterations. >> And the final answer is correct (comparing to another solver). >> >> I use intel compiler. >> The program is in Fortran (by mpif90), except the solver is in c (by >> mpicc). >> >> However, when I re-configure with --with-debugging=0 (the only >> change), >> the program terminates in about 30 iterations with the wrong final >> solution. >> >> Thank you. >> >> Best regards, >> Longxiang Chen >> >> Do something everyday that gets you closer to being done. > From bsmith at mcs.anl.gov Thu Jun 27 23:16:48 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 27 Jun 2013 23:16:48 -0500 Subject: [petsc-users] PetscViewerBinaryMatlab example for petsc-3.4.1 In-Reply-To: References: Message-ID: Bishesh, Thanks for reporting this problem. I did not fix this example when removing PetscViewerBinaryMatlabOpen() nor did I test it well before the release. I have updated the example and made other needed changes to PETSc code in the maint branch of the PETSc repository. All the fixes will in the next patch release of PETSc 3.4 If you would like to try the fixed code now you can follow the instructions at http://www.mcs.anl.gov/petsc/download/index.html and follow the instructions that say use git clone -b maint https://bitbucket.org/petsc/petsc petsc with this copy ex7.c should work correctly and bin/matlab/PetscReadBinaryMatlab.m should read the data into Matlab. Please let us know of any difficulties. Barry On Jun 26, 2013, at 8:44 AM, Bishesh Khanal wrote: > Dear all, > I'm just starting to learn using Petsc. I was trying the example (../src/dm/examples/tutorials/ex7.c) > for petsc-3.4.1 . However it seems this file has not been updated for the 3.4.1 version since it uses > PetscViewerBinaryMatlabOpen() and friends which have been removed from this newer version. > Now I tried to change them but I have some questions: > > The lines I wanted to change are the ones that use following funcitons: > 1. PetscViewerBinaryMatlabOpen(PETSC_COMM_WORLD,params->filename,&viewer) > 2. PetscViewerBinaryMatlabOutputBag(viewer,"params",bag) > 3. PetscViewerBinaryMatlabOutputVecDA(viewer,"da1",global,da) > > For 1, it seems I can change it with: > PetscViewerBinaryOpen(PETSC_COMM_WORLD,params->filename,FILE_MODE_WRITE,&viewer) > PetscViewerSetFormat(viewer,PETSC_VIEWER_BINARY_MATLAB) > > Now, for the 2 and 3 I thought I need to use: > PetscViewerBinaryWrite(PetscViewer viewer,void *data,PetscInt count,PetscDataType dtype,PetscBool istemp) > but I'm not sure what exactly to provide as argument for count and dtype to have the same functionality as intended in ex7.c > I tried (for 2) using: > ierr = PetscViewerBinaryWrite(viewer,&bag,1,PETSC_OBJECT,PETSC_FALSE); > but it results in an error when running the executable. > How should I count the no. of items for count variable in this case for the bag object ? > And what should be the dtype ? > > Thanks, > Bishesh From bisheshkh at gmail.com Fri Jun 28 03:10:57 2013 From: bisheshkh at gmail.com (Bishesh Khanal) Date: Fri, 28 Jun 2013 10:10:57 +0200 Subject: [petsc-users] PetscViewerBinaryMatlab example for petsc-3.4.1 In-Reply-To: References: Message-ID: Yes, it works now. Thanks! On Fri, Jun 28, 2013 at 6:16 AM, Barry Smith wrote: > > Bishesh, > > Thanks for reporting this problem. I did not fix this example when > removing PetscViewerBinaryMatlabOpen() nor did I test it well before the > release. I have updated the example and made other needed changes to PETSc > code in the maint branch of the PETSc repository. All the fixes will in > the next patch release of PETSc 3.4 If you would like to try the fixed code > now you can follow the instructions at > http://www.mcs.anl.gov/petsc/download/index.html and follow the > instructions that say use > > git clone -b maint https://bitbucket.org/petsc/petsc petsc > > with this copy ex7.c should work correctly and > bin/matlab/PetscReadBinaryMatlab.m should read the data into Matlab. > > Please let us know of any difficulties. > > Barry > > > > > > On Jun 26, 2013, at 8:44 AM, Bishesh Khanal wrote: > > > Dear all, > > I'm just starting to learn using Petsc. I was trying the example > (../src/dm/examples/tutorials/ex7.c) > > for petsc-3.4.1 . However it seems this file has not been updated for > the 3.4.1 version since it uses > > PetscViewerBinaryMatlabOpen() and friends which have been removed from > this newer version. > > Now I tried to change them but I have some questions: > > > > The lines I wanted to change are the ones that use following funcitons: > > 1. PetscViewerBinaryMatlabOpen(PETSC_COMM_WORLD,params->filename,&viewer) > > 2. PetscViewerBinaryMatlabOutputBag(viewer,"params",bag) > > 3. PetscViewerBinaryMatlabOutputVecDA(viewer,"da1",global,da) > > > > For 1, it seems I can change it with: > > > PetscViewerBinaryOpen(PETSC_COMM_WORLD,params->filename,FILE_MODE_WRITE,&viewer) > > PetscViewerSetFormat(viewer,PETSC_VIEWER_BINARY_MATLAB) > > > > Now, for the 2 and 3 I thought I need to use: > > PetscViewerBinaryWrite(PetscViewer viewer,void *data,PetscInt > count,PetscDataType dtype,PetscBool istemp) > > but I'm not sure what exactly to provide as argument for count and dtype > to have the same functionality as intended in ex7.c > > I tried (for 2) using: > > ierr = PetscViewerBinaryWrite(viewer,&bag,1,PETSC_OBJECT,PETSC_FALSE); > > but it results in an error when running the executable. > > How should I count the no. of items for count variable in this case for > the bag object ? > > And what should be the dtype ? > > > > Thanks, > > Bishesh > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frtr at fysik.dtu.dk Fri Jun 28 07:52:05 2013 From: frtr at fysik.dtu.dk (Frederik Treue) Date: Fri, 28 Jun 2013 14:52:05 +0200 Subject: [petsc-users] 1d FFT of 2D vector (using fftw for preference) Message-ID: <1372423925.9414.6.camel@frtr-laptop> Hi, is there a simple way to perform a fourier transform of a 2D vector in only one dimension? Ie. transform the function f(x,y) in realspace (x,y) to f(k_x,y) in k space in the x direction and realspace in the y direction? I'd like to use FFTW for preference, since that is implemented on the cluster that I'm working with. /Frederik Treue From knepley at gmail.com Fri Jun 28 07:56:35 2013 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 28 Jun 2013 07:56:35 -0500 Subject: [petsc-users] 1d FFT of 2D vector (using fftw for preference) In-Reply-To: <1372423925.9414.6.camel@frtr-laptop> References: <1372423925.9414.6.camel@frtr-laptop> Message-ID: On Fri, Jun 28, 2013 at 7:52 AM, Frederik Treue wrote: > Hi, > > is there a simple way to perform a fourier transform of a 2D vector in > only one dimension? Ie. transform the function f(x,y) in realspace (x,y) > to f(k_x,y) in k space in the x direction and realspace in the y > direction? I'd like to use FFTW for preference, since that is > implemented on the cluster that I'm working with. I do not know of one other than repeated calls to the 1D transform. If you can find it in the FFTW API when could put it in. Thanks, Matt > > /Frederik Treue > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 28 09:10:23 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 28 Jun 2013 09:10:23 -0500 Subject: [petsc-users] 1d FFT of 2D vector (using fftw for preference) In-Reply-To: References: <1372423925.9414.6.camel@frtr-laptop> Message-ID: <87ppv6gwc0.fsf@mcs.anl.gov> Matthew Knepley writes: > I do not know of one other than repeated calls to the 1D transform. If > you can find it in the FFTW API when could put it in. I used this: c->p_forward = fftw_plan_guru_r2r(1, &(c->tdim), c->rank-1, c->dim, x, c->work, &redft00, flag | FFTW_PRESERVE_INPUT); https://github.com/jedbrown/spectral-petsc/blob/master/chebyshev.c#L127 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From suifengls at gmail.com Fri Jun 28 15:30:35 2013 From: suifengls at gmail.com (Longxiang Chen) Date: Fri, 28 Jun 2013 13:30:35 -0700 Subject: [petsc-users] no-debug-mode problem In-Reply-To: References: <1d49881d863e511ada2913536d29a259@geology.wisc.edu> Message-ID: A is formed by three arrays IA[NZ], JA[NZ] and VA[NZ] IA[i] is row index, JA[i] is column index and VA[i] is the value in (IA[i], JA[i]). For intel compiler: when I use --with-debugging=0, the VA[] is not correct. I don't know what kind of optimization it does. A[0][0] = 1.000000e-25 A[0][1] = 0.000000e+00 A[0][2] = 0.000000e+00 A[1][0] = 0.000000e+00 A[1][1] = -3.479028e+02 A[1][2] = 0.000000e+00 A[2][0] = 0.000000e+00 A[2][1] = 0.000000e+00 A[2][2] = -3.479028e+02 A[3][3] = 1.000000e-25 ... CORRECT: A[0][0] = -2.961372e-07 A[0][1] = 1.160201e+02 A[0][2] = 2.744589e+02 A[1][0] = 0.000000e+00 A[1][1] = -3.479028e+02 A[1][2] = 0.000000e+00 A[2][0] = -8.332708e-08 A[2][1] = 0.000000e+00 A[2][2] = -3.479028e+02 A[3][3] = -3.027917e-07 ... For gcc-4.7.2: when I use --with-debugging=0, the fortran main function cannot read input data before it starts the LOOP, the ksp_solver() is called inside the loop. ===> ERRONEOUS DATA INITIALIZATION STOP EXECUTION--------- Best regards, Longxiang Chen Do something everyday that gets you closer to being done. On Thu, Jun 27, 2013 at 6:42 PM, Barry Smith wrote: > > On Jun 27, 2013, at 2:23 PM, Tabrez Ali wrote: > > > Fortran can be tricky. > > > > Try to run the program in valgrind and/or recheck all your arguments. I > once forgot MatAssemblyType in the call to MatAssembly and the code still > ran fine on one machine but failed on other. It is better to test with at > least 2-3 compilers (GNU, Solaris and Open64 Fortran/C compilers are all > free on Linux). > > > > T > > You can also run both versions with -snes_monitor -ksp_monitor and see > if they both start out the same way. > > Barry > > > > > > > On 27.06.2013 14:52, Longxiang Chen wrote: > >> Dear all, > >> > >> I use ksp_solver to solve Ax=b, where A is from an outer loop of PDE. > >> Under debug mode(default), it solves the problem in about 4000 > >> iterations. > >> And the final answer is correct (comparing to another solver). > >> > >> I use intel compiler. > >> The program is in Fortran (by mpif90), except the solver is in c (by > >> mpicc). > >> > >> However, when I re-configure with --with-debugging=0 (the only > >> change), > >> the program terminates in about 30 iterations with the wrong final > >> solution. > >> > >> Thank you. > >> > >> Best regards, > >> Longxiang Chen > >> > >> Do something everyday that gets you closer to being done. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 28 17:25:41 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 28 Jun 2013 17:25:41 -0500 Subject: [petsc-users] no-debug-mode problem In-Reply-To: References: <1d49881d863e511ada2913536d29a259@geology.wisc.edu> Message-ID: <682B5D66-320D-4003-AC13-F4DE78752D25@mcs.anl.gov> Run both debugged and optimized versions with valgrind: http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Barry On Jun 28, 2013, at 3:30 PM, Longxiang Chen wrote: > A is formed by three arrays IA[NZ], JA[NZ] and VA[NZ] > IA[i] is row index, JA[i] is column index and VA[i] is the value in (IA[i], JA[i]). > > For intel compiler: > when I use --with-debugging=0, the VA[] is not correct. I don't know what kind of optimization it does. > > A[0][0] = 1.000000e-25 > A[0][1] = 0.000000e+00 > A[0][2] = 0.000000e+00 > A[1][0] = 0.000000e+00 > A[1][1] = -3.479028e+02 > A[1][2] = 0.000000e+00 > A[2][0] = 0.000000e+00 > A[2][1] = 0.000000e+00 > A[2][2] = -3.479028e+02 > A[3][3] = 1.000000e-25 > ... > > CORRECT: > A[0][0] = -2.961372e-07 > A[0][1] = 1.160201e+02 > A[0][2] = 2.744589e+02 > A[1][0] = 0.000000e+00 > A[1][1] = -3.479028e+02 > A[1][2] = 0.000000e+00 > A[2][0] = -8.332708e-08 > A[2][1] = 0.000000e+00 > A[2][2] = -3.479028e+02 > A[3][3] = -3.027917e-07 > ... > > For gcc-4.7.2: > when I use --with-debugging=0, the fortran main function cannot read input data before it starts the LOOP, the ksp_solver() is called inside the loop. > ===> > ERRONEOUS DATA INITIALIZATION STOP EXECUTION--------- > > Best regards, > Longxiang Chen > > Do something everyday that gets you closer to being done. > > > > On Thu, Jun 27, 2013 at 6:42 PM, Barry Smith wrote: > > On Jun 27, 2013, at 2:23 PM, Tabrez Ali wrote: > > > Fortran can be tricky. > > > > Try to run the program in valgrind and/or recheck all your arguments. I once forgot MatAssemblyType in the call to MatAssembly and the code still ran fine on one machine but failed on other. It is better to test with at least 2-3 compilers (GNU, Solaris and Open64 Fortran/C compilers are all free on Linux). > > > > T > > You can also run both versions with -snes_monitor -ksp_monitor and see if they both start out the same way. > > Barry > > > > > > > On 27.06.2013 14:52, Longxiang Chen wrote: > >> Dear all, > >> > >> I use ksp_solver to solve Ax=b, where A is from an outer loop of PDE. > >> Under debug mode(default), it solves the problem in about 4000 > >> iterations. > >> And the final answer is correct (comparing to another solver). > >> > >> I use intel compiler. > >> The program is in Fortran (by mpif90), except the solver is in c (by > >> mpicc). > >> > >> However, when I re-configure with --with-debugging=0 (the only > >> change), > >> the program terminates in about 30 iterations with the wrong final > >> solution. > >> > >> Thank you. > >> > >> Best regards, > >> Longxiang Chen > >> > >> Do something everyday that gets you closer to being done. > > > > From suifengls at gmail.com Sat Jun 29 00:24:41 2013 From: suifengls at gmail.com (Longxiang Chen) Date: Fri, 28 Jun 2013 22:24:41 -0700 Subject: [petsc-users] no-debug-mode problem In-Reply-To: <682B5D66-320D-4003-AC13-F4DE78752D25@mcs.anl.gov> References: <1d49881d863e511ada2913536d29a259@geology.wisc.edu> <682B5D66-320D-4003-AC13-F4DE78752D25@mcs.anl.gov> Message-ID: I find that the fortran program cannot add optimization flag (-O2 or -O3) if I use gfortran-4.7.2 or ifort to compile. When I change the compiler to gfortran-4.4, now the --with-debug=0 (adding -O3) works. Thank you all. Best regards, Longxiang Chen On Fri, Jun 28, 2013 at 3:25 PM, Barry Smith wrote: > > Run both debugged and optimized versions with valgrind: > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > Barry > > On Jun 28, 2013, at 3:30 PM, Longxiang Chen wrote: > > > A is formed by three arrays IA[NZ], JA[NZ] and VA[NZ] > > IA[i] is row index, JA[i] is column index and VA[i] is the value in > (IA[i], JA[i]). > > > > For intel compiler: > > when I use --with-debugging=0, the VA[] is not correct. I don't know > what kind of optimization it does. > > > > A[0][0] = 1.000000e-25 > > A[0][1] = 0.000000e+00 > > A[0][2] = 0.000000e+00 > > A[1][0] = 0.000000e+00 > > A[1][1] = -3.479028e+02 > > A[1][2] = 0.000000e+00 > > A[2][0] = 0.000000e+00 > > A[2][1] = 0.000000e+00 > > A[2][2] = -3.479028e+02 > > A[3][3] = 1.000000e-25 > > ... > > > > CORRECT: > > A[0][0] = -2.961372e-07 > > A[0][1] = 1.160201e+02 > > A[0][2] = 2.744589e+02 > > A[1][0] = 0.000000e+00 > > A[1][1] = -3.479028e+02 > > A[1][2] = 0.000000e+00 > > A[2][0] = -8.332708e-08 > > A[2][1] = 0.000000e+00 > > A[2][2] = -3.479028e+02 > > A[3][3] = -3.027917e-07 > > ... > > > > For gcc-4.7.2: > > when I use --with-debugging=0, the fortran main function cannot read > input data before it starts the LOOP, the ksp_solver() is called inside the > loop. > > ===> > > ERRONEOUS DATA INITIALIZATION STOP EXECUTION--------- > > > > Best regards, > > Longxiang Chen > > > > Do something everyday that gets you closer to being done. > > > > > > > > On Thu, Jun 27, 2013 at 6:42 PM, Barry Smith wrote: > > > > On Jun 27, 2013, at 2:23 PM, Tabrez Ali wrote: > > > > > Fortran can be tricky. > > > > > > Try to run the program in valgrind and/or recheck all your arguments. > I once forgot MatAssemblyType in the call to MatAssembly and the code still > ran fine on one machine but failed on other. It is better to test with at > least 2-3 compilers (GNU, Solaris and Open64 Fortran/C compilers are all > free on Linux). > > > > > > T > > > > You can also run both versions with -snes_monitor -ksp_monitor and > see if they both start out the same way. > > > > Barry > > > > > > > > > > > On 27.06.2013 14:52, Longxiang Chen wrote: > > >> Dear all, > > >> > > >> I use ksp_solver to solve Ax=b, where A is from an outer loop of PDE. > > >> Under debug mode(default), it solves the problem in about 4000 > > >> iterations. > > >> And the final answer is correct (comparing to another solver). > > >> > > >> I use intel compiler. > > >> The program is in Fortran (by mpif90), except the solver is in c (by > > >> mpicc). > > >> > > >> However, when I re-configure with --with-debugging=0 (the only > > >> change), > > >> the program terminates in about 30 iterations with the wrong final > > >> solution. > > >> > > >> Thank you. > > >> > > >> Best regards, > > >> Longxiang Chen > > >> > > >> Do something everyday that gets you closer to being done. > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sat Jun 29 00:41:46 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 29 Jun 2013 00:41:46 -0500 (CDT) Subject: [petsc-users] no-debug-mode problem In-Reply-To: References: <1d49881d863e511ada2913536d29a259@geology.wisc.edu> <682B5D66-320D-4003-AC13-F4DE78752D25@mcs.anl.gov> Message-ID: There could still be bugs in the code. Have you tried valgrind as suggested? Satish On Fri, 28 Jun 2013, Longxiang Chen wrote: > I find that the fortran program cannot add optimization flag (-O2 or -O3) > if I use gfortran-4.7.2 or ifort to compile. > When I change the compiler to gfortran-4.4, now the --with-debug=0 (adding > -O3) works. > > Thank you all. > > Best regards, > Longxiang Chen > > > > > On Fri, Jun 28, 2013 at 3:25 PM, Barry Smith wrote: > > > > > Run both debugged and optimized versions with valgrind: > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > Barry > > > > On Jun 28, 2013, at 3:30 PM, Longxiang Chen wrote: > > > > > A is formed by three arrays IA[NZ], JA[NZ] and VA[NZ] > > > IA[i] is row index, JA[i] is column index and VA[i] is the value in > > (IA[i], JA[i]). > > > > > > For intel compiler: > > > when I use --with-debugging=0, the VA[] is not correct. I don't know > > what kind of optimization it does. > > > > > > A[0][0] = 1.000000e-25 > > > A[0][1] = 0.000000e+00 > > > A[0][2] = 0.000000e+00 > > > A[1][0] = 0.000000e+00 > > > A[1][1] = -3.479028e+02 > > > A[1][2] = 0.000000e+00 > > > A[2][0] = 0.000000e+00 > > > A[2][1] = 0.000000e+00 > > > A[2][2] = -3.479028e+02 > > > A[3][3] = 1.000000e-25 > > > ... > > > > > > CORRECT: > > > A[0][0] = -2.961372e-07 > > > A[0][1] = 1.160201e+02 > > > A[0][2] = 2.744589e+02 > > > A[1][0] = 0.000000e+00 > > > A[1][1] = -3.479028e+02 > > > A[1][2] = 0.000000e+00 > > > A[2][0] = -8.332708e-08 > > > A[2][1] = 0.000000e+00 > > > A[2][2] = -3.479028e+02 > > > A[3][3] = -3.027917e-07 > > > ... > > > > > > For gcc-4.7.2: > > > when I use --with-debugging=0, the fortran main function cannot read > > input data before it starts the LOOP, the ksp_solver() is called inside the > > loop. > > > ===> > > > ERRONEOUS DATA INITIALIZATION STOP EXECUTION--------- > > > > > > Best regards, > > > Longxiang Chen > > > > > > Do something everyday that gets you closer to being done. > > > > > > > > > > > > On Thu, Jun 27, 2013 at 6:42 PM, Barry Smith wrote: > > > > > > On Jun 27, 2013, at 2:23 PM, Tabrez Ali wrote: > > > > > > > Fortran can be tricky. > > > > > > > > Try to run the program in valgrind and/or recheck all your arguments. > > I once forgot MatAssemblyType in the call to MatAssembly and the code still > > ran fine on one machine but failed on other. It is better to test with at > > least 2-3 compilers (GNU, Solaris and Open64 Fortran/C compilers are all > > free on Linux). > > > > > > > > T > > > > > > You can also run both versions with -snes_monitor -ksp_monitor and > > see if they both start out the same way. > > > > > > Barry > > > > > > > > > > > > > > > On 27.06.2013 14:52, Longxiang Chen wrote: > > > >> Dear all, > > > >> > > > >> I use ksp_solver to solve Ax=b, where A is from an outer loop of PDE. > > > >> Under debug mode(default), it solves the problem in about 4000 > > > >> iterations. > > > >> And the final answer is correct (comparing to another solver). > > > >> > > > >> I use intel compiler. > > > >> The program is in Fortran (by mpif90), except the solver is in c (by > > > >> mpicc). > > > >> > > > >> However, when I re-configure with --with-debugging=0 (the only > > > >> change), > > > >> the program terminates in about 30 iterations with the wrong final > > > >> solution. > > > >> > > > >> Thank you. > > > >> > > > >> Best regards, > > > >> Longxiang Chen > > > >> > > > >> Do something everyday that gets you closer to being done. > > > > > > > > > > > > > > > From bsmith at mcs.anl.gov Sat Jun 29 01:05:10 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 29 Jun 2013 01:05:10 -0500 Subject: [petsc-users] no-debug-mode problem In-Reply-To: References: <1d49881d863e511ada2913536d29a259@geology.wisc.edu> <682B5D66-320D-4003-AC13-F4DE78752D25@mcs.anl.gov> Message-ID: <117AF362-5A92-4FD1-B0F0-779A40AF414C@mcs.anl.gov> On Jun 29, 2013, at 12:24 AM, Longxiang Chen wrote: > I find that the fortran program cannot add optimization flag (-O2 or -O3) if I use gfortran-4.7.2 or ifort to compile. What do you mean? Do you mean to say that if use -O2 or -O3 flags with gfortran 4.7.2 or ifort it produces the wrong answer? Does it produce the same wrong answer with both compilers? If you use no optimization with both compilers does it produce the "correct" answer? > When I change the compiler to gfortran-4.4, now the --with-debug=0 (adding -O3) works. but if you use -O3 flag with gfortran 4.4 it runs with the correct answer? Since it works incorrectly with two very different fortran compilers (optimized) it is more likely there is something wrong with your code than with both compilers. You most definitely should run with valgrind and make sure there is no memory corruption in your code. Barry > > Thank you all. > > Best regards, > Longxiang Chen > > > > > On Fri, Jun 28, 2013 at 3:25 PM, Barry Smith wrote: > > Run both debugged and optimized versions with valgrind: http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > Barry > > On Jun 28, 2013, at 3:30 PM, Longxiang Chen wrote: > > > A is formed by three arrays IA[NZ], JA[NZ] and VA[NZ] > > IA[i] is row index, JA[i] is column index and VA[i] is the value in (IA[i], JA[i]). > > > > For intel compiler: > > when I use --with-debugging=0, the VA[] is not correct. I don't know what kind of optimization it does. > > > > A[0][0] = 1.000000e-25 > > A[0][1] = 0.000000e+00 > > A[0][2] = 0.000000e+00 > > A[1][0] = 0.000000e+00 > > A[1][1] = -3.479028e+02 > > A[1][2] = 0.000000e+00 > > A[2][0] = 0.000000e+00 > > A[2][1] = 0.000000e+00 > > A[2][2] = -3.479028e+02 > > A[3][3] = 1.000000e-25 > > ... > > > > CORRECT: > > A[0][0] = -2.961372e-07 > > A[0][1] = 1.160201e+02 > > A[0][2] = 2.744589e+02 > > A[1][0] = 0.000000e+00 > > A[1][1] = -3.479028e+02 > > A[1][2] = 0.000000e+00 > > A[2][0] = -8.332708e-08 > > A[2][1] = 0.000000e+00 > > A[2][2] = -3.479028e+02 > > A[3][3] = -3.027917e-07 > > ... > > > > For gcc-4.7.2: > > when I use --with-debugging=0, the fortran main function cannot read input data before it starts the LOOP, the ksp_solver() is called inside the loop. > > ===> > > ERRONEOUS DATA INITIALIZATION STOP EXECUTION--------- > > > > Best regards, > > Longxiang Chen > > > > Do something everyday that gets you closer to being done. > > > > > > > > On Thu, Jun 27, 2013 at 6:42 PM, Barry Smith wrote: > > > > On Jun 27, 2013, at 2:23 PM, Tabrez Ali wrote: > > > > > Fortran can be tricky. > > > > > > Try to run the program in valgrind and/or recheck all your arguments. I once forgot MatAssemblyType in the call to MatAssembly and the code still ran fine on one machine but failed on other. It is better to test with at least 2-3 compilers (GNU, Solaris and Open64 Fortran/C compilers are all free on Linux). > > > > > > T > > > > You can also run both versions with -snes_monitor -ksp_monitor and see if they both start out the same way. > > > > Barry > > > > > > > > > > > On 27.06.2013 14:52, Longxiang Chen wrote: > > >> Dear all, > > >> > > >> I use ksp_solver to solve Ax=b, where A is from an outer loop of PDE. > > >> Under debug mode(default), it solves the problem in about 4000 > > >> iterations. > > >> And the final answer is correct (comparing to another solver). > > >> > > >> I use intel compiler. > > >> The program is in Fortran (by mpif90), except the solver is in c (by > > >> mpicc). > > >> > > >> However, when I re-configure with --with-debugging=0 (the only > > >> change), > > >> the program terminates in about 30 iterations with the wrong final > > >> solution. > > >> > > >> Thank you. > > >> > > >> Best regards, > > >> Longxiang Chen > > >> > > >> Do something everyday that gets you closer to being done. > > > > > > > > > From suifengls at gmail.com Sat Jun 29 01:10:14 2013 From: suifengls at gmail.com (Longxiang Chen) Date: Fri, 28 Jun 2013 23:10:14 -0700 Subject: [petsc-users] no-debug-mode problem In-Reply-To: <117AF362-5A92-4FD1-B0F0-779A40AF414C@mcs.anl.gov> References: <1d49881d863e511ada2913536d29a259@geology.wisc.edu> <682B5D66-320D-4003-AC13-F4DE78752D25@mcs.anl.gov> <117AF362-5A92-4FD1-B0F0-779A40AF414C@mcs.anl.gov> Message-ID: When I don't use -O2 or -O3, both compilers produce the correct answers. When I use ifort with -O2/-O3, it produces wrong answer. When I use gfortran-4.7 with -O2/-O3, it cannot even read in the input file. When I use gofrtran-4.4 with -O2/-O3, it also produce correct answer. I think the problem is on the Fortran code. All the error information is about "used uninitialized" from valgrind: ==8397== Memcheck, a memory error detector ==8397== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==8397== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info ==8397== Command: ./eco2n ==8397== ==8397== Conditional jump or move depends on uninitialised value(s) ==8397== at 0x11E1F367: __intel_sse2_strcat (in /opt/intel/composer_xe_2013.3.163/compiler/lib/intel64/libirc.so) ==8397== by 0xC728F0: read_configuration_files (in /home/lchen/seq_v2/eco2n) ==8397== by 0xC20DE3: MPID_Init (in /home/lchen/seq_v2/eco2n) ==8397== by 0xBEAC54: MPIR_Init_thread (in /home/lchen/seq_v2/eco2n) ==8397== by 0xBEA443: PMPI_Init (in /home/lchen/seq_v2/eco2n) ==8397== by 0xBB6CAC: mpi_init (in /home/lchen/seq_v2/eco2n) ==8397== by 0x43C724: MAIN__ (t2cg22.f:319) ==8397== by 0x40C2CB: main (in /home/lchen/seq_v2/eco2n) ==8397== ==8397== Use of uninitialised value of size 8 ==8397== at 0x11E1F3DA: __intel_sse2_strcat (in /opt/intel/composer_xe_2013.3.163/compiler/lib/intel64/libirc.so) ==8397== by 0xC728F0: read_configuration_files (in /home/lchen/seq_v2/eco2n) ==8397== by 0xC20DE3: MPID_Init (in /home/lchen/seq_v2/eco2n) ==8397== by 0xBEAC54: MPIR_Init_thread (in /home/lchen/seq_v2/eco2n) ==8397== by 0xBEA443: PMPI_Init (in /home/lchen/seq_v2/eco2n) ==8397== by 0xBB6CAC: mpi_init (in /home/lchen/seq_v2/eco2n) ==8397== by 0x43C724: MAIN__ (t2cg22.f:319) ==8397== by 0x40C2CB: main (in /home/lchen/seq_v2/eco2n) ==8397== Memcheck, a memory error detector ==8397== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==8397== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info ==8397== Command: ./eco2n ==8397== ==8397== Conditional jump or move depends on uninitialised value(s) ==8397== at 0x11E1F367: __intel_sse2_strcat (in /opt/intel/composer_xe_2013.3.163/compiler/lib/intel64/libirc.so) ==8397== by 0xC728F0: read_configuration_files (in /home/lchen/seq_v2/eco2n) ==8397== by 0xC20DE3: MPID_Init (in /home/lchen/seq_v2/eco2n) ==8397== by 0xBEAC54: MPIR_Init_thread (in /home/lchen/seq_v2/eco2n) ==8397== by 0xBEA443: PMPI_Init (in /home/lchen/seq_v2/eco2n) ==8397== by 0xBB6CAC: mpi_init (in /home/lchen/seq_v2/eco2n) ==8397== by 0x43C724: MAIN__ (t2cg22.f:319) ==8397== by 0x40C2CB: main (in /home/lchen/seq_v2/eco2n) ==8397== ==8397== Use of uninitialised value of size 8 ==8397== at 0x11E1F3DA: __intel_sse2_strcat (in /opt/intel/composer_xe_2013.3.163/compiler/lib/intel64/libirc.so) ==8397== by 0xC728F0: read_configuration_files (in /home/lchen/seq_v2/eco2n) ==8397== by 0xC20DE3: MPID_Init (in /home/lchen/seq_v2/eco2n) ==8397== by 0xBEAC54: MPIR_Init_thread (in /home/lchen/seq_v2/eco2n) ==8397== by 0xBEA443: PMPI_Init (in /home/lchen/seq_v2/eco2n) ==8397== by 0xBB6CAC: mpi_init (in /home/lchen/seq_v2/eco2n) ==8397== by 0x43C724: MAIN__ (t2cg22.f:319) ==8397== by 0x40C2CB: main (in /home/lchen/seq_v2/eco2n) ... hundreds of similar error. Best regards, Longxiang Chen On Fri, Jun 28, 2013 at 11:05 PM, Barry Smith wrote: > > On Jun 29, 2013, at 12:24 AM, Longxiang Chen wrote: > > > I find that the fortran program cannot add optimization flag (-O2 or > -O3) if I use gfortran-4.7.2 or ifort to compile. > > What do you mean? Do you mean to say that if use -O2 or -O3 flags with > gfortran 4.7.2 or ifort it produces the wrong answer? Does it produce the > same wrong answer with both compilers? If you use no optimization with > both compilers does it produce the "correct" answer? > > > When I change the compiler to gfortran-4.4, now the --with-debug=0 > (adding -O3) works. > > but if you use -O3 flag with gfortran 4.4 it runs with the correct > answer? > > Since it works incorrectly with two very different fortran compilers > (optimized) it is more likely there is something wrong with your code than > with both compilers. You most definitely should run with valgrind and make > sure there is no memory corruption in your code. > > Barry > > > > > Thank you all. > > > > Best regards, > > Longxiang Chen > > > > > > > > > > On Fri, Jun 28, 2013 at 3:25 PM, Barry Smith wrote: > > > > Run both debugged and optimized versions with valgrind: > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > Barry > > > > On Jun 28, 2013, at 3:30 PM, Longxiang Chen wrote: > > > > > A is formed by three arrays IA[NZ], JA[NZ] and VA[NZ] > > > IA[i] is row index, JA[i] is column index and VA[i] is the value in > (IA[i], JA[i]). > > > > > > For intel compiler: > > > when I use --with-debugging=0, the VA[] is not correct. I don't know > what kind of optimization it does. > > > > > > A[0][0] = 1.000000e-25 > > > A[0][1] = 0.000000e+00 > > > A[0][2] = 0.000000e+00 > > > A[1][0] = 0.000000e+00 > > > A[1][1] = -3.479028e+02 > > > A[1][2] = 0.000000e+00 > > > A[2][0] = 0.000000e+00 > > > A[2][1] = 0.000000e+00 > > > A[2][2] = -3.479028e+02 > > > A[3][3] = 1.000000e-25 > > > ... > > > > > > CORRECT: > > > A[0][0] = -2.961372e-07 > > > A[0][1] = 1.160201e+02 > > > A[0][2] = 2.744589e+02 > > > A[1][0] = 0.000000e+00 > > > A[1][1] = -3.479028e+02 > > > A[1][2] = 0.000000e+00 > > > A[2][0] = -8.332708e-08 > > > A[2][1] = 0.000000e+00 > > > A[2][2] = -3.479028e+02 > > > A[3][3] = -3.027917e-07 > > > ... > > > > > > For gcc-4.7.2: > > > when I use --with-debugging=0, the fortran main function cannot read > input data before it starts the LOOP, the ksp_solver() is called inside the > loop. > > > ===> > > > ERRONEOUS DATA INITIALIZATION STOP EXECUTION--------- > > > > > > Best regards, > > > Longxiang Chen > > > > > > Do something everyday that gets you closer to being done. > > > > > > > > > > > > On Thu, Jun 27, 2013 at 6:42 PM, Barry Smith > wrote: > > > > > > On Jun 27, 2013, at 2:23 PM, Tabrez Ali > wrote: > > > > > > > Fortran can be tricky. > > > > > > > > Try to run the program in valgrind and/or recheck all your > arguments. I once forgot MatAssemblyType in the call to MatAssembly and the > code still ran fine on one machine but failed on other. It is better to > test with at least 2-3 compilers (GNU, Solaris and Open64 Fortran/C > compilers are all free on Linux). > > > > > > > > T > > > > > > You can also run both versions with -snes_monitor -ksp_monitor and > see if they both start out the same way. > > > > > > Barry > > > > > > > > > > > > > > > On 27.06.2013 14:52, Longxiang Chen wrote: > > > >> Dear all, > > > >> > > > >> I use ksp_solver to solve Ax=b, where A is from an outer loop of > PDE. > > > >> Under debug mode(default), it solves the problem in about 4000 > > > >> iterations. > > > >> And the final answer is correct (comparing to another solver). > > > >> > > > >> I use intel compiler. > > > >> The program is in Fortran (by mpif90), except the solver is in c (by > > > >> mpicc). > > > >> > > > >> However, when I re-configure with --with-debugging=0 (the only > > > >> change), > > > >> the program terminates in about 30 iterations with the wrong final > > > >> solution. > > > >> > > > >> Thank you. > > > >> > > > >> Best regards, > > > >> Longxiang Chen > > > >> > > > >> Do something everyday that gets you closer to being done. > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Jun 29 01:18:26 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 29 Jun 2013 01:18:26 -0500 Subject: [petsc-users] no-debug-mode problem In-Reply-To: References: <1d49881d863e511ada2913536d29a259@geology.wisc.edu> <682B5D66-320D-4003-AC13-F4DE78752D25@mcs.anl.gov> <117AF362-5A92-4FD1-B0F0-779A40AF414C@mcs.anl.gov> Message-ID: <8DFF9454-EC80-4447-BC53-F9467D6D4249@mcs.anl.gov> These ones are not the problem, but they are annoying since they make it hard to find real problems. Install PETSc with --download-mpich instead of using that MPI you have used. PETSc installs a valgrind clean MPI that will not have MPI false positives. Then compile your code with that version of PETSc and run with valgrind and send us the output. Barry On Jun 29, 2013, at 1:10 AM, Longxiang Chen wrote: > When I don't use -O2 or -O3, both compilers produce the correct answers. > When I use ifort with -O2/-O3, it produces wrong answer. > When I use gfortran-4.7 with -O2/-O3, it cannot even read in the input file. > When I use gofrtran-4.4 with -O2/-O3, it also produce correct answer. > I think the problem is on the Fortran code. > > All the error information is about "used uninitialized" from valgrind: > > ==8397== Memcheck, a memory error detector > ==8397== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. > ==8397== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info > ==8397== Command: ./eco2n > ==8397== > ==8397== Conditional jump or move depends on uninitialised value(s) > ==8397== at 0x11E1F367: __intel_sse2_strcat (in /opt/intel/composer_xe_2013.3.163/compiler/lib/intel64/libirc.so) > ==8397== by 0xC728F0: read_configuration_files (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xC20DE3: MPID_Init (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xBEAC54: MPIR_Init_thread (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xBEA443: PMPI_Init (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xBB6CAC: mpi_init (in /home/lchen/seq_v2/eco2n) > ==8397== by 0x43C724: MAIN__ (t2cg22.f:319) > ==8397== by 0x40C2CB: main (in /home/lchen/seq_v2/eco2n) > ==8397== > ==8397== Use of uninitialised value of size 8 > ==8397== at 0x11E1F3DA: __intel_sse2_strcat (in /opt/intel/composer_xe_2013.3.163/compiler/lib/intel64/libirc.so) > ==8397== by 0xC728F0: read_configuration_files (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xC20DE3: MPID_Init (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xBEAC54: MPIR_Init_thread (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xBEA443: PMPI_Init (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xBB6CAC: mpi_init (in /home/lchen/seq_v2/eco2n) > ==8397== by 0x43C724: MAIN__ (t2cg22.f:319) > ==8397== by 0x40C2CB: main (in /home/lchen/seq_v2/eco2n) > > > ==8397== Memcheck, a memory error detector > ==8397== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. > ==8397== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info > ==8397== Command: ./eco2n > ==8397== > ==8397== Conditional jump or move depends on uninitialised value(s) > ==8397== at 0x11E1F367: __intel_sse2_strcat (in /opt/intel/composer_xe_2013.3.163/compiler/lib/intel64/libirc.so) > ==8397== by 0xC728F0: read_configuration_files (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xC20DE3: MPID_Init (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xBEAC54: MPIR_Init_thread (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xBEA443: PMPI_Init (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xBB6CAC: mpi_init (in /home/lchen/seq_v2/eco2n) > ==8397== by 0x43C724: MAIN__ (t2cg22.f:319) > ==8397== by 0x40C2CB: main (in /home/lchen/seq_v2/eco2n) > ==8397== > ==8397== Use of uninitialised value of size 8 > ==8397== at 0x11E1F3DA: __intel_sse2_strcat (in /opt/intel/composer_xe_2013.3.163/compiler/lib/intel64/libirc.so) > ==8397== by 0xC728F0: read_configuration_files (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xC20DE3: MPID_Init (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xBEAC54: MPIR_Init_thread (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xBEA443: PMPI_Init (in /home/lchen/seq_v2/eco2n) > ==8397== by 0xBB6CAC: mpi_init (in /home/lchen/seq_v2/eco2n) > ==8397== by 0x43C724: MAIN__ (t2cg22.f:319) > ==8397== by 0x40C2CB: main (in /home/lchen/seq_v2/eco2n) > > ... > hundreds of similar error. > > Best regards, > Longxiang Chen > > > > On Fri, Jun 28, 2013 at 11:05 PM, Barry Smith wrote: > > On Jun 29, 2013, at 12:24 AM, Longxiang Chen wrote: > > > I find that the fortran program cannot add optimization flag (-O2 or -O3) if I use gfortran-4.7.2 or ifort to compile. > > What do you mean? Do you mean to say that if use -O2 or -O3 flags with gfortran 4.7.2 or ifort it produces the wrong answer? Does it produce the same wrong answer with both compilers? If you use no optimization with both compilers does it produce the "correct" answer? > > > When I change the compiler to gfortran-4.4, now the --with-debug=0 (adding -O3) works. > > but if you use -O3 flag with gfortran 4.4 it runs with the correct answer? > > Since it works incorrectly with two very different fortran compilers (optimized) it is more likely there is something wrong with your code than with both compilers. You most definitely should run with valgrind and make sure there is no memory corruption in your code. > > Barry > > > > > Thank you all. > > > > Best regards, > > Longxiang Chen > > > > > > > > > > On Fri, Jun 28, 2013 at 3:25 PM, Barry Smith wrote: > > > > Run both debugged and optimized versions with valgrind: http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > Barry > > > > On Jun 28, 2013, at 3:30 PM, Longxiang Chen wrote: > > > > > A is formed by three arrays IA[NZ], JA[NZ] and VA[NZ] > > > IA[i] is row index, JA[i] is column index and VA[i] is the value in (IA[i], JA[i]). > > > > > > For intel compiler: > > > when I use --with-debugging=0, the VA[] is not correct. I don't know what kind of optimization it does. > > > > > > A[0][0] = 1.000000e-25 > > > A[0][1] = 0.000000e+00 > > > A[0][2] = 0.000000e+00 > > > A[1][0] = 0.000000e+00 > > > A[1][1] = -3.479028e+02 > > > A[1][2] = 0.000000e+00 > > > A[2][0] = 0.000000e+00 > > > A[2][1] = 0.000000e+00 > > > A[2][2] = -3.479028e+02 > > > A[3][3] = 1.000000e-25 > > > ... > > > > > > CORRECT: > > > A[0][0] = -2.961372e-07 > > > A[0][1] = 1.160201e+02 > > > A[0][2] = 2.744589e+02 > > > A[1][0] = 0.000000e+00 > > > A[1][1] = -3.479028e+02 > > > A[1][2] = 0.000000e+00 > > > A[2][0] = -8.332708e-08 > > > A[2][1] = 0.000000e+00 > > > A[2][2] = -3.479028e+02 > > > A[3][3] = -3.027917e-07 > > > ... > > > > > > For gcc-4.7.2: > > > when I use --with-debugging=0, the fortran main function cannot read input data before it starts the LOOP, the ksp_solver() is called inside the loop. > > > ===> > > > ERRONEOUS DATA INITIALIZATION STOP EXECUTION--------- > > > > > > Best regards, > > > Longxiang Chen > > > > > > Do something everyday that gets you closer to being done. > > > > > > > > > > > > On Thu, Jun 27, 2013 at 6:42 PM, Barry Smith wrote: > > > > > > On Jun 27, 2013, at 2:23 PM, Tabrez Ali wrote: > > > > > > > Fortran can be tricky. > > > > > > > > Try to run the program in valgrind and/or recheck all your arguments. I once forgot MatAssemblyType in the call to MatAssembly and the code still ran fine on one machine but failed on other. It is better to test with at least 2-3 compilers (GNU, Solaris and Open64 Fortran/C compilers are all free on Linux). > > > > > > > > T > > > > > > You can also run both versions with -snes_monitor -ksp_monitor and see if they both start out the same way. > > > > > > Barry > > > > > > > > > > > > > > > On 27.06.2013 14:52, Longxiang Chen wrote: > > > >> Dear all, > > > >> > > > >> I use ksp_solver to solve Ax=b, where A is from an outer loop of PDE. > > > >> Under debug mode(default), it solves the problem in about 4000 > > > >> iterations. > > > >> And the final answer is correct (comparing to another solver). > > > >> > > > >> I use intel compiler. > > > >> The program is in Fortran (by mpif90), except the solver is in c (by > > > >> mpicc). > > > >> > > > >> However, when I re-configure with --with-debugging=0 (the only > > > >> change), > > > >> the program terminates in about 30 iterations with the wrong final > > > >> solution. > > > >> > > > >> Thank you. > > > >> > > > >> Best regards, > > > >> Longxiang Chen > > > >> > > > >> Do something everyday that gets you closer to being done. > > > > > > > > > > > > > > > >