From yann.jobic at univ-amu.fr Mon Apr 2 09:42:29 2018 From: yann.jobic at univ-amu.fr (yann JOBIC) Date: Mon, 2 Apr 2018 16:42:29 +0200 Subject: [petsc-users] DMForestTransferVec with -petscspace_order 0 Message-ID: <91696bcc-a34a-1142-8322-817bc497263b@univ-amu.fr> Hi, I'm using DMForestTransferVec, as in "dm/impls/forest/examples/tests/ex2.c". I would like to use it with a space approximation order at zero (-petscspace_order 0). However, in this case, it's not working (valgrind output of ex2.c from forest test) : ==8604== Conditional jump or move depends on uninitialised value(s) ==8604==??? at 0x47D74F: DMPlexVecGetClosure (plex.c:4035) ==8604==??? by 0x557612: DMPlexLocatePoint_General_3D_Internal (plexgeometry.c:153) ==8604==??? by 0x559B85: DMPlexLocatePoint_Internal (plexgeometry.c:383) ==8604==??? by 0x611EED: DMPlexComputeInjectorReferenceTree (plextree.c:3247) ==8604==??? by 0x6148DB: DMPlexReferenceTreeGetInjector (plextree.c:3454) ==8604==??? by 0x61EAD8: DMPlexTransferVecTree_Inject (plextree.c:4319) ==8604==??? by 0x620CC1: DMPlexTransferVecTree (plextree.c:4527) ==8604==??? by 0x7F23D8: DMForestTransferVec_p8est (pforest.c:4239) ==8604==??? by 0x429B48: DMForestTransferVec (forest.c:985) ==8604==??? by 0x40BB8A: main (transf_test.c:136) With the error message : [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range It's working fine in 2D (p4est). The problem arise in 3D (p8est). Is it an expected behavior ? Am i doing something wrong ? Thanks in advance, Yann From fande.kong at inl.gov Mon Apr 2 16:46:37 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Mon, 2 Apr 2018 15:46:37 -0600 Subject: [petsc-users] A bad commit affects MOOSE Message-ID: Hi All, I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its applications. I have a error message for a standard test: *preconditioners/pbp.lots_of_variables: MPI had an errorpreconditioners/pbp.lots_of_variables: ------------------------------------------------preconditioners/pbp.lots_of_variables: Other MPI error, error stack:preconditioners/pbp.lots_of_variables: PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: PMPI_Comm_dup(162)..................: preconditioners/pbp.lots_of_variables: MPIR_Comm_dup_impl(57)..............: preconditioners/pbp.lots_of_variables: MPIR_Comm_copy(739).................: preconditioners/pbp.lots_of_variables: MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 free on this process; ignore_id=0)* I did "git bisect', and the following commit introduces this issue: *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano Zampini >Date: Sat Nov 5 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE hpmat already stores two HYPRE vectors* Before I debug line-by-line, anyone has a clue on this? Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Mon Apr 2 17:02:38 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Tue, 3 Apr 2018 00:02:38 +0200 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: Message-ID: maybe this will fix ? *diff --git a/src/ksp/pc/impls/hypre/hypre.c b/src/ksp/pc/impls/hypre/hypre.c* *index 28addcf533..6a756d4c57 100644* *--- a/src/ksp/pc/impls/hypre/hypre.c* *+++ b/src/ksp/pc/impls/hypre/hypre.c* @@ -142,8 +142,7 @@ static PetscErrorCode PCSetUp_HYPRE(PC pc) ierr = PetscObjectTypeCompare((PetscObject)pc->pmat,MATHYPRE,&ishypre);CHKERRQ(ierr); if (!ishypre) { - ierr = MatDestroy(&jac->hpmat);CHKERRQ(ierr); - ierr = MatConvert(pc->pmat,MATHYPRE,MAT_INITIAL_MATRIX,&jac->hpmat);CHKERRQ(ierr); + ierr = MatConvert(pc->pmat,MATHYPRE,jac->hpmat ? MAT_REUSE_MATRIX : MAT_INITIAL_MATRIX,&jac->hpmat);CHKERRQ(ierr); } else { ierr = PetscObjectReference((PetscObject)pc->pmat);CHKERRQ(ierr); ierr = MatDestroy(&jac->hpmat);CHKERRQ(ierr); 2018-04-02 23:46 GMT+02:00 Kong, Fande : > Hi All, > > I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > applications. I have a error message for a standard test: > > > > > > > > > > *preconditioners/pbp.lots_of_variables: MPI had an > errorpreconditioners/pbp.lots_of_variables: > ------------------------------------------------preconditioners/pbp.lots_of_variables: > Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > PMPI_Comm_dup(162)..................: > preconditioners/pbp.lots_of_variables: > MPIR_Comm_dup_impl(57)..............: > preconditioners/pbp.lots_of_variables: > MPIR_Comm_copy(739).................: > preconditioners/pbp.lots_of_variables: > MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 free > on this process; ignore_id=0)* > > > I did "git bisect', and the following commit introduces this issue: > > > > > > > > > *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano Zampini > >Date: Sat Nov 5 > 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > hpmat already stores two HYPRE vectors* > > Before I debug line-by-line, anyone has a clue on this? > > > Fande, > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Mon Apr 2 17:19:33 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Mon, 2 Apr 2018 16:19:33 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: Message-ID: Nope. There is a back trace: ** thread #1: tid = 0x3b477b4, 0x00007fffb306cd42 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT * frame #0: 0x00007fffb306cd42 libsystem_kernel.dylib`__pthread_kill + 10 frame #1: 0x00007fffb315a457 libsystem_pthread.dylib`pthread_kill + 90 frame #2: 0x00007fffb2fd2420 libsystem_c.dylib`abort + 129 frame #3: 0x00000001057ff30a libpetsc.3.07.dylib`Petsc_MPI_AbortOnError(comm=, flag=) + 26 at init.c:185 [opt] frame #4: 0x0000000106bd3245 libpmpi.12.dylib`MPIR_Err_return_comm + 533 frame #5: 0x00000001068defd4 libmpi.12.dylib`MPI_Comm_create + 3492 frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, participate=, new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] frame #7: 0x000000010618f8ba libpetsc.3.07.dylib`hypre_GaussElimSetup(amg_data=0x00007fe7ff857a00, level=, relax_type=9) + 74 at par_relax.c:4209 [opt] frame #8: 0x0000000106140e93 libpetsc.3.07.dylib`hypre_BoomerAMGSetup(amg_vdata=, A=0x00007fe80842aff0, f=0x00007fe80842a980, u=0x00007fe80842a510) + 17699 at par_amg_setup.c:2108 [opt] frame #9: 0x0000000105ec773c libpetsc.3.07.dylib`PCSetUp_HYPRE(pc=) + 2540 at hypre.c:226 [opt] frame #10: 0x0000000105eea68d libpetsc.3.07.dylib`PCSetUp(pc=0x00007fe805553f50) + 797 at precon.c:968 [opt] frame #11: 0x0000000105ee9fe5 libpetsc.3.07.dylib`PCApply(pc=0x00007fe805553f50, x=0x00007fe80052d420, y=0x00007fe800522c20) + 181 at precon.c:478 [opt] frame #12: 0x00000001015cf218 libmesh_opt.0.dylib`libMesh::PetscPreconditioner::apply(libMesh::NumericVector const&, libMesh::NumericVector&) + 24 frame #13: 0x00000001009c7998 libmoose-opt.0.dylib`PhysicsBasedPreconditioner::apply(libMesh::NumericVector const&, libMesh::NumericVector&) + 520 frame #14: 0x00000001016ad701 libmesh_opt.0.dylib`libmesh_petsc_preconditioner_apply + 129 frame #15: 0x0000000105e7e715 libpetsc.3.07.dylib`PCApply_Shell(pc=0x00007fe8052623f0, x=0x00007fe806805a20, y=0x00007fe806805420) + 117 at shellpc.c:123 [opt] frame #16: 0x0000000105eea079 libpetsc.3.07.dylib`PCApply(pc=0x00007fe8052623f0, x=0x00007fe806805a20, y=0x00007fe806805420) + 329 at precon.c:482 [opt] frame #17: 0x0000000105eeb611 libpetsc.3.07.dylib`PCApplyBAorAB(pc=0x00007fe8052623f0, side=PC_RIGHT, x=0x00007fe806805a20, y=0x00007fe806806020, work=0x00007fe806805420) + 945 at precon.c:714 [opt] frame #18: 0x0000000105f31658 libpetsc.3.07.dylib`KSPGMRESCycle [inlined] KSP_PCApplyBAorAB(ksp=0x00007fe806022220, x=, y=0x00007fe806806020, w=) + 191 at kspimpl.h:295 [opt] frame #19: 0x0000000105f31599 libpetsc.3.07.dylib`KSPGMRESCycle(itcount=, ksp=) + 553 at gmres.c:156 [opt] frame #20: 0x0000000105f326bd libpetsc.3.07.dylib`KSPSolve_GMRES(ksp=) + 221 at gmres.c:240 [opt] frame #21: 0x0000000105f5f671 libpetsc.3.07.dylib`KSPSolve(ksp=0x00007fe806022220, b=0x00007fe7fd946220, x=) + 1345 at itfunc.c:677 [opt] frame #22: 0x0000000105fd0251 libpetsc.3.07.dylib`SNESSolve_NEWTONLS(snes=) + 1425 at ls.c:230 [opt] frame #23: 0x0000000105fa10ca libpetsc.3.07.dylib`SNESSolve(snes=, b=, x=0x00007fe7fd865e20) + 858 at snes.c:4128 [opt] frame #24: 0x00000001016b63c3 libmesh_opt.0.dylib`libMesh::PetscNonlinearSolver::solve(libMesh::SparseMatrix&, libMesh::NumericVector&, libMesh::NumericVector&, double, unsigned int) + 835 frame #25: 0x00000001016fc244 libmesh_opt.0.dylib`libMesh::NonlinearImplicitSystem::solve() + 324 frame #26: 0x0000000100a71dc8 libmoose-opt.0.dylib`NonlinearSystem::solve() + 472 frame #27: 0x00000001009fe815 libmoose-opt.0.dylib`FEProblemBase::solve() + 117 frame #28: 0x0000000100761fba libmoose-opt.0.dylib`Steady::execute() + 266 frame #29: 0x0000000100b78ac3 libmoose-opt.0.dylib`MooseApp::run() + 259 frame #30: 0x00000001003843aa moose_test-opt`main + 122 frame #31: 0x00007fffb2f3e235 libdyld.dylib`start + 1* Fande, On Mon, Apr 2, 2018 at 4:02 PM, Stefano Zampini wrote: > maybe this will fix ? > > > *diff --git a/src/ksp/pc/impls/hypre/hypre.c > b/src/ksp/pc/impls/hypre/hypre.c* > > *index 28addcf533..6a756d4c57 100644* > > *--- a/src/ksp/pc/impls/hypre/hypre.c* > > *+++ b/src/ksp/pc/impls/hypre/hypre.c* > > @@ -142,8 +142,7 @@ static PetscErrorCode PCSetUp_HYPRE(PC pc) > > > > ierr = PetscObjectTypeCompare((PetscObject)pc->pmat,MATHYPRE, > &ishypre);CHKERRQ(ierr); > > if (!ishypre) { > > - ierr = MatDestroy(&jac->hpmat);CHKERRQ(ierr); > > - ierr = MatConvert(pc->pmat,MATHYPRE,MAT_INITIAL_MATRIX,&jac-> > hpmat);CHKERRQ(ierr); > > + ierr = MatConvert(pc->pmat,MATHYPRE,jac->hpmat ? MAT_REUSE_MATRIX : > MAT_INITIAL_MATRIX,&jac->hpmat);CHKERRQ(ierr); > > } else { > > ierr = PetscObjectReference((PetscObject)pc->pmat);CHKERRQ(ierr); > > ierr = MatDestroy(&jac->hpmat);CHKERRQ(ierr); > > > > 2018-04-02 23:46 GMT+02:00 Kong, Fande : > >> Hi All, >> >> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its >> applications. I have a error message for a standard test: >> >> >> >> >> >> >> >> >> >> *preconditioners/pbp.lots_of_variables: MPI had an >> errorpreconditioners/pbp.lots_of_variables: >> ------------------------------------------------preconditioners/pbp.lots_of_variables: >> Other MPI error, error stack:preconditioners/pbp.lots_of_variables: >> PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, >> new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: >> PMPI_Comm_dup(162)..................: >> preconditioners/pbp.lots_of_variables: >> MPIR_Comm_dup_impl(57)..............: >> preconditioners/pbp.lots_of_variables: >> MPIR_Comm_copy(739).................: >> preconditioners/pbp.lots_of_variables: >> MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 free >> on this process; ignore_id=0)* >> >> >> I did "git bisect', and the following commit introduces this issue: >> >> >> >> >> >> >> >> >> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano Zampini >> >Date: Sat Nov 5 >> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE >> hpmat already stores two HYPRE vectors* >> >> Before I debug line-by-line, anyone has a clue on this? >> >> >> Fande, >> > > > > -- > Stefano > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Apr 2 17:23:36 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 2 Apr 2018 17:23:36 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: Message-ID: Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? If so - you could try changing to PETSC_COMM_WORLD Satish On Mon, 2 Apr 2018, Kong, Fande wrote: > Hi All, > > I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > applications. I have a error message for a standard test: > > > > > > > > > > *preconditioners/pbp.lots_of_variables: MPI had an > errorpreconditioners/pbp.lots_of_variables: > ------------------------------------------------preconditioners/pbp.lots_of_variables: > Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > PMPI_Comm_dup(162)..................: > preconditioners/pbp.lots_of_variables: > MPIR_Comm_dup_impl(57)..............: > preconditioners/pbp.lots_of_variables: > MPIR_Comm_copy(739).................: > preconditioners/pbp.lots_of_variables: > MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 free > on this process; ignore_id=0)* > > > I did "git bisect', and the following commit introduces this issue: > > > > > > > > > *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano Zampini > >Date: Sat Nov 5 > 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > hpmat already stores two HYPRE vectors* > > Before I debug line-by-line, anyone has a clue on this? > > > Fande, > From fande.kong at inl.gov Mon Apr 2 17:59:20 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Mon, 2 Apr 2018 16:59:20 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: Message-ID: On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay wrote: > Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? > > If so - you could try changing to PETSC_COMM_WORLD > I do not think we are using PETSC_COMM_WORLD when creating PETSc objects. Why we can not use MPI_COMM_WORLD? Fande, > > Satish > > > On Mon, 2 Apr 2018, Kong, Fande wrote: > > > Hi All, > > > > I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > > applications. I have a error message for a standard test: > > > > > > > > > > > > > > > > > > > > *preconditioners/pbp.lots_of_variables: MPI had an > > errorpreconditioners/pbp.lots_of_variables: > > ------------------------------------------------ > preconditioners/pbp.lots_of_variables: > > Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > > PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > > new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > > PMPI_Comm_dup(162)..................: > > preconditioners/pbp.lots_of_variables: > > MPIR_Comm_dup_impl(57)..............: > > preconditioners/pbp.lots_of_variables: > > MPIR_Comm_copy(739).................: > > preconditioners/pbp.lots_of_variables: > > MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 > free > > on this process; ignore_id=0)* > > > > > > I did "git bisect', and the following commit introduces this issue: > > > > > > > > > > > > > > > > > > *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano Zampini > > >Date: Sat Nov 5 > > 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > > hpmat already stores two HYPRE vectors* > > > > Before I debug line-by-line, anyone has a clue on this? > > > > > > Fande, > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Apr 2 18:08:33 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 2 Apr 2018 18:08:33 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: Message-ID: PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls to MPI_Comm_dup() - thus potentially avoiding such errors http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscCommDuplicate.html Satish On Mon, 2 Apr 2018, Kong, Fande wrote: > On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay wrote: > > > Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? > > > > If so - you could try changing to PETSC_COMM_WORLD > > > > > I do not think we are using PETSC_COMM_WORLD when creating PETSc objects. > Why we can not use MPI_COMM_WORLD? > > > Fande, > > > > > > Satish > > > > > > On Mon, 2 Apr 2018, Kong, Fande wrote: > > > > > Hi All, > > > > > > I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > > > applications. I have a error message for a standard test: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *preconditioners/pbp.lots_of_variables: MPI had an > > > errorpreconditioners/pbp.lots_of_variables: > > > ------------------------------------------------ > > preconditioners/pbp.lots_of_variables: > > > Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > > > PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > > > new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > > > PMPI_Comm_dup(162)..................: > > > preconditioners/pbp.lots_of_variables: > > > MPIR_Comm_dup_impl(57)..............: > > > preconditioners/pbp.lots_of_variables: > > > MPIR_Comm_copy(739).................: > > > preconditioners/pbp.lots_of_variables: > > > MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 > > free > > > on this process; ignore_id=0)* > > > > > > > > > I did "git bisect', and the following commit introduces this issue: > > > > > > > > > > > > > > > > > > > > > > > > > > > *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano Zampini > > > >Date: Sat Nov 5 > > > 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > > > hpmat already stores two HYPRE vectors* > > > > > > Before I debug line-by-line, anyone has a clue on this? > > > > > > > > > Fande, > > > > > > > > From fande.kong at inl.gov Mon Apr 2 18:44:10 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Mon, 2 Apr 2018 17:44:10 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: Message-ID: Why we do not use user-level MPI communicators directly? What are potential risks here? Fande, On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay wrote: > PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls to > MPI_Comm_dup() - thus potentially avoiding such errors > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs. > anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_ > PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi > CY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_ > zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > > Satish > > On Mon, 2 Apr 2018, Kong, Fande wrote: > > > On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay wrote: > > > > > Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? > > > > > > If so - you could try changing to PETSC_COMM_WORLD > > > > > > > > > I do not think we are using PETSC_COMM_WORLD when creating PETSc objects. > > Why we can not use MPI_COMM_WORLD? > > > > > > Fande, > > > > > > > > > > Satish > > > > > > > > > On Mon, 2 Apr 2018, Kong, Fande wrote: > > > > > > > Hi All, > > > > > > > > I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > > > > applications. I have a error message for a standard test: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *preconditioners/pbp.lots_of_variables: MPI had an > > > > errorpreconditioners/pbp.lots_of_variables: > > > > ------------------------------------------------ > > > preconditioners/pbp.lots_of_variables: > > > > Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > > > > PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > > > > new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > > > > PMPI_Comm_dup(162)..................: > > > > preconditioners/pbp.lots_of_variables: > > > > MPIR_Comm_dup_impl(57)..............: > > > > preconditioners/pbp.lots_of_variables: > > > > MPIR_Comm_copy(739).................: > > > > preconditioners/pbp.lots_of_variables: > > > > MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 > > > free > > > > on this process; ignore_id=0)* > > > > > > > > > > > > I did "git bisect', and the following commit introduces this issue: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano > Zampini > > > > >Date: Sat > Nov 5 > > > > 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > > > > hpmat already stores two HYPRE vectors* > > > > > > > > Before I debug line-by-line, anyone has a clue on this? > > > > > > > > > > > > Fande, > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedmud at gmail.com Mon Apr 2 19:07:35 2018 From: friedmud at gmail.com (Derek Gaston) Date: Tue, 03 Apr 2018 00:07:35 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: Message-ID: I?m working with Fande on this and I would like to add a bit more. There are many circumstances where we aren?t working on COMM_WORLD at all (e.g. working on a sub-communicator) but PETSc was initialized using MPI_COMM_WORLD (think multi-level solves)? and we need to create arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering duplication. Can you explain why PETSc needs to duplicate the communicator so much? Thanks for your help in tracking this down! Derek On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: > Why we do not use user-level MPI communicators directly? What are > potential risks here? > > > Fande, > > On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay wrote: > >> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls to >> MPI_Comm_dup() - thus potentially avoiding such errors >> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > > >> >> Satish >> >> On Mon, 2 Apr 2018, Kong, Fande wrote: >> >> > On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay wrote: >> > >> > > Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? >> > > >> > > If so - you could try changing to PETSC_COMM_WORLD >> > > >> > >> > >> > I do not think we are using PETSC_COMM_WORLD when creating PETSc >> objects. >> > Why we can not use MPI_COMM_WORLD? >> > >> > >> > Fande, >> > >> > >> > > >> > > Satish >> > > >> > > >> > > On Mon, 2 Apr 2018, Kong, Fande wrote: >> > > >> > > > Hi All, >> > > > >> > > > I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its >> > > > applications. I have a error message for a standard test: >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > *preconditioners/pbp.lots_of_variables: MPI had an >> > > > errorpreconditioners/pbp.lots_of_variables: >> > > > ------------------------------------------------ >> > > preconditioners/pbp.lots_of_variables: >> > > > Other MPI error, error stack:preconditioners/pbp.lots_of_variables: >> > > > PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, >> > > > new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: >> > > > PMPI_Comm_dup(162)..................: >> > > > preconditioners/pbp.lots_of_variables: >> > > > MPIR_Comm_dup_impl(57)..............: >> > > > preconditioners/pbp.lots_of_variables: >> > > > MPIR_Comm_copy(739).................: >> > > > preconditioners/pbp.lots_of_variables: >> > > > MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 >> > > free >> > > > on this process; ignore_id=0)* >> > > > >> > > > >> > > > I did "git bisect', and the following commit introduces this issue: >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano >> Zampini >> > > > >Date: Sat >> Nov 5 >> > > > 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE >> > > > hpmat already stores two HYPRE vectors* >> > > > >> > > > Before I debug line-by-line, anyone has a clue on this? >> > > > >> > > > >> > > > Fande, >> > > > >> > > >> > > >> > >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Apr 2 19:24:04 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 3 Apr 2018 00:24:04 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: Message-ID: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> Are we sure this is a PETSc comm issue and not a hypre comm duplication issue frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, participate=, new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] Looks like hypre is needed to generate subcomms, perhaps it generates too many? Barry > On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: > > I?m working with Fande on this and I would like to add a bit more. There are many circumstances where we aren?t working on COMM_WORLD at all (e.g. working on a sub-communicator) but PETSc was initialized using MPI_COMM_WORLD (think multi-level solves)? and we need to create arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering duplication. > > Can you explain why PETSc needs to duplicate the communicator so much? > > Thanks for your help in tracking this down! > > Derek > > On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: > Why we do not use user-level MPI communicators directly? What are potential risks here? > > > Fande, > > On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay wrote: > PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls to MPI_Comm_dup() - thus potentially avoiding such errors > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > > > Satish > > On Mon, 2 Apr 2018, Kong, Fande wrote: > > > On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay wrote: > > > > > Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? > > > > > > If so - you could try changing to PETSC_COMM_WORLD > > > > > > > > > I do not think we are using PETSC_COMM_WORLD when creating PETSc objects. > > Why we can not use MPI_COMM_WORLD? > > > > > > Fande, > > > > > > > > > > Satish > > > > > > > > > On Mon, 2 Apr 2018, Kong, Fande wrote: > > > > > > > Hi All, > > > > > > > > I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > > > > applications. I have a error message for a standard test: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *preconditioners/pbp.lots_of_variables: MPI had an > > > > errorpreconditioners/pbp.lots_of_variables: > > > > ------------------------------------------------ > > > preconditioners/pbp.lots_of_variables: > > > > Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > > > > PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > > > > new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > > > > PMPI_Comm_dup(162)..................: > > > > preconditioners/pbp.lots_of_variables: > > > > MPIR_Comm_dup_impl(57)..............: > > > > preconditioners/pbp.lots_of_variables: > > > > MPIR_Comm_copy(739).................: > > > > preconditioners/pbp.lots_of_variables: > > > > MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 > > > free > > > > on this process; ignore_id=0)* > > > > > > > > > > > > I did "git bisect', and the following commit introduces this issue: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano Zampini > > > > >Date: Sat Nov 5 > > > > 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > > > > hpmat already stores two HYPRE vectors* > > > > > > > > Before I debug line-by-line, anyone has a clue on this? > > > > > > > > > > > > Fande, > > > > > > > > > > > > > From balay at mcs.anl.gov Mon Apr 2 19:48:14 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 2 Apr 2018 19:48:14 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: Message-ID: Sorry - I was going down the wrong path.. Sure MPI_COMM_WORLD vs PETSC_COMM_WORLD shouldn't make a difference [except for a couple of extra mpi_comm_dup() calls.] Satish On Tue, 3 Apr 2018, Derek Gaston wrote: > I?m working with Fande on this and I would like to add a bit more. There > are many circumstances where we aren?t working on COMM_WORLD at all (e.g. > working on a sub-communicator) but PETSc was initialized using > MPI_COMM_WORLD (think multi-level solves)? and we need to create > arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We > definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering > duplication. > > Can you explain why PETSc needs to duplicate the communicator so much? > > Thanks for your help in tracking this down! > > Derek > > On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: > > > Why we do not use user-level MPI communicators directly? What are > > potential risks here? > > > > > > Fande, > > > > On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay wrote: > > > >> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls to > >> MPI_Comm_dup() - thus potentially avoiding such errors > >> > >> > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > > > > > >> > >> Satish > >> > >> On Mon, 2 Apr 2018, Kong, Fande wrote: > >> > >> > On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay wrote: > >> > > >> > > Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? > >> > > > >> > > If so - you could try changing to PETSC_COMM_WORLD > >> > > > >> > > >> > > >> > I do not think we are using PETSC_COMM_WORLD when creating PETSc > >> objects. > >> > Why we can not use MPI_COMM_WORLD? > >> > > >> > > >> > Fande, > >> > > >> > > >> > > > >> > > Satish > >> > > > >> > > > >> > > On Mon, 2 Apr 2018, Kong, Fande wrote: > >> > > > >> > > > Hi All, > >> > > > > >> > > > I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > >> > > > applications. I have a error message for a standard test: > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > *preconditioners/pbp.lots_of_variables: MPI had an > >> > > > errorpreconditioners/pbp.lots_of_variables: > >> > > > ------------------------------------------------ > >> > > preconditioners/pbp.lots_of_variables: > >> > > > Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > >> > > > PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > >> > > > new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > >> > > > PMPI_Comm_dup(162)..................: > >> > > > preconditioners/pbp.lots_of_variables: > >> > > > MPIR_Comm_dup_impl(57)..............: > >> > > > preconditioners/pbp.lots_of_variables: > >> > > > MPIR_Comm_copy(739).................: > >> > > > preconditioners/pbp.lots_of_variables: > >> > > > MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 > >> > > free > >> > > > on this process; ignore_id=0)* > >> > > > > >> > > > > >> > > > I did "git bisect', and the following commit introduces this issue: > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano > >> Zampini > >> > > > >Date: Sat > >> Nov 5 > >> > > > 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > >> > > > hpmat already stores two HYPRE vectors* > >> > > > > >> > > > Before I debug line-by-line, anyone has a clue on this? > >> > > > > >> > > > > >> > > > Fande, > >> > > > > >> > > > >> > > > >> > > >> > >> > From tisaac at cc.gatech.edu Mon Apr 2 20:33:41 2018 From: tisaac at cc.gatech.edu (Tobin Isaac) Date: Mon, 2 Apr 2018 21:33:41 -0400 Subject: [petsc-users] DMForestTransferVec with -petscspace_order 0 In-Reply-To: <91696bcc-a34a-1142-8322-817bc497263b@univ-amu.fr> References: <91696bcc-a34a-1142-8322-817bc497263b@univ-amu.fr> Message-ID: <20180403013341.6kewwzmm7evmoytf@gatech.edu> Hi Yann, Thanks for pointing this out to us. Matt and I are the two most actively developing in this area. We have been working on separate threads and this looks like an issue where we need to sync up. I think there is a simple fix, but it would be helpful to know which version petsc you're working from: I'm seeing different behavior. Could you please send along more complete output? Cheers, Toby On Mon, Apr 02, 2018 at 04:42:29PM +0200, yann JOBIC wrote: > Hi, > > I'm using DMForestTransferVec, as in "dm/impls/forest/examples/tests/ex2.c". > > I would like to use it with a space approximation order at zero > (-petscspace_order 0). However, in this case, it's not working (valgrind > output of ex2.c from forest test) : > > ==8604== Conditional jump or move depends on uninitialised value(s) > ==8604==??? at 0x47D74F: DMPlexVecGetClosure (plex.c:4035) > ==8604==??? by 0x557612: DMPlexLocatePoint_General_3D_Internal > (plexgeometry.c:153) > ==8604==??? by 0x559B85: DMPlexLocatePoint_Internal (plexgeometry.c:383) > ==8604==??? by 0x611EED: DMPlexComputeInjectorReferenceTree > (plextree.c:3247) > ==8604==??? by 0x6148DB: DMPlexReferenceTreeGetInjector (plextree.c:3454) > ==8604==??? by 0x61EAD8: DMPlexTransferVecTree_Inject (plextree.c:4319) > ==8604==??? by 0x620CC1: DMPlexTransferVecTree (plextree.c:4527) > ==8604==??? by 0x7F23D8: DMForestTransferVec_p8est (pforest.c:4239) > ==8604==??? by 0x429B48: DMForestTransferVec (forest.c:985) > ==8604==??? by 0x40BB8A: main (transf_test.c:136) > > With the error message : > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > It's working fine in 2D (p4est). The problem arise in 3D (p8est). > > Is it an expected behavior ? Am i doing something wrong ? > > Thanks in advance, > > Yann > From balay at mcs.anl.gov Mon Apr 2 20:44:56 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 2 Apr 2018 20:44:56 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> Message-ID: We do a MPI_Comm_dup() for objects related to externalpackages. Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm);CHKERRQ(ierr); src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); src/ksp/pc/impls/spai/ispai.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_spai));CHKERRQ(ierr); src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, &comm);CHKERRQ(ierr); src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); src/mat/impls/aij/mpi/mumps/mumps.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_mumps));CHKERRQ(ierr); src/mat/impls/aij/mpi/pastix/pastix.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_comm));CHKERRQ(ierr); src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_superlu));CHKERRQ(ierr); src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); src/mat/partition/impls/pmetis/pmetis.c: ierr = MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a MPI_Comm_dup() of each of these (duplicates of duplicates return the same communictor) src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr); src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr); src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out);CHKERRQ(ierr); src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm);CHKERRQ(ierr); src/ts/impls/implicit/sundials/sundials.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_sundials));CHKERRQ(ierr); Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid these MPI_Comm_dup() calls? Satish On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > Are we sure this is a PETSc comm issue and not a hypre comm duplication issue > > frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, participate=, new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] > > Looks like hypre is needed to generate subcomms, perhaps it generates too many? > > Barry > > > > On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: > > > > I?m working with Fande on this and I would like to add a bit more. There are many circumstances where we aren?t working on COMM_WORLD at all (e.g. working on a sub-communicator) but PETSc was initialized using MPI_COMM_WORLD (think multi-level solves)? and we need to create arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering duplication. > > > > Can you explain why PETSc needs to duplicate the communicator so much? > > > > Thanks for your help in tracking this down! > > > > Derek > > > > On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: > > Why we do not use user-level MPI communicators directly? What are potential risks here? > > > > > > Fande, > > > > On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay wrote: > > PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls to MPI_Comm_dup() - thus potentially avoiding such errors > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > > > > > > Satish > > > > On Mon, 2 Apr 2018, Kong, Fande wrote: > > > > > On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay wrote: > > > > > > > Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? > > > > > > > > If so - you could try changing to PETSC_COMM_WORLD > > > > > > > > > > > > > I do not think we are using PETSC_COMM_WORLD when creating PETSc objects. > > > Why we can not use MPI_COMM_WORLD? > > > > > > > > > Fande, > > > > > > > > > > > > > > Satish > > > > > > > > > > > > On Mon, 2 Apr 2018, Kong, Fande wrote: > > > > > > > > > Hi All, > > > > > > > > > > I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > > > > > applications. I have a error message for a standard test: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *preconditioners/pbp.lots_of_variables: MPI had an > > > > > errorpreconditioners/pbp.lots_of_variables: > > > > > ------------------------------------------------ > > > > preconditioners/pbp.lots_of_variables: > > > > > Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > > > > > PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > > > > > new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > > > > > PMPI_Comm_dup(162)..................: > > > > > preconditioners/pbp.lots_of_variables: > > > > > MPIR_Comm_dup_impl(57)..............: > > > > > preconditioners/pbp.lots_of_variables: > > > > > MPIR_Comm_copy(739).................: > > > > > preconditioners/pbp.lots_of_variables: > > > > > MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 > > > > free > > > > > on this process; ignore_id=0)* > > > > > > > > > > > > > > > I did "git bisect', and the following commit introduces this issue: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano Zampini > > > > > >Date: Sat Nov 5 > > > > > 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > > > > > hpmat already stores two HYPRE vectors* > > > > > > > > > > Before I debug line-by-line, anyone has a clue on this? > > > > > > > > > > > > > > > Fande, > > > > > > > > > > > > > > > > > > > > From skavou1 at lsu.edu Mon Apr 2 21:52:51 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Tue, 3 Apr 2018 02:52:51 +0000 Subject: [petsc-users] read variables from a source file in petsc Message-ID: Hi all, I wrote a petsc code, and I defined my variables in a separate file, namely common.h, and common.c as following common.h: extern int i; extern int n; common.c: #include #include "common.h PetscScalar i; // Define i and initialize PetscScalar n; // Define n and initialize i=2.2; n=3.5; Now in my prog.c code I have: #include #include static char help[]="program main"; int main (int argc, char **argv){ PetscInitialize(&argc,&argv, (char*) 0,help); PetscPrintf(PETSC_COMM_WORLD,"i=%f!\n",a); PetscPrintf(PETSC_COMM_WORLD,"n=%f!\n",b) ; PetscFinalize(); } if my it was not using petsc and it was a normal c code, I could run it using the following command: gcc prog. c common.c but now that I am using petsc, I do not know how should I include common.c in my makefile. my makefile looks like this: include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules prog: prog.o chkopts -${CLINKER} -o prog.out prog.o ${PETSC_LIB} ${RM} *.o .PHONY: distclean rune_1 test distclean: @rm -f *~ prog Thanks, Sepideh -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 3 02:17:24 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 3 Apr 2018 07:17:24 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> Message-ID: Each external package definitely needs its own duplicated communicator; cannot share between packages. The only problem with the dups below is if they are in a loop and get called many times. To debug the hypre/duplication issue in MOOSE I would run in the debugger with a break point in MPI_Comm_dup() and see who keeps calling it an unreasonable amount of times. (My guess is this is a new "feature" in hypre that they will need to fix but only debugging will tell) Barry > On Apr 2, 2018, at 7:44 PM, Balay, Satish wrote: > > We do a MPI_Comm_dup() for objects related to externalpackages. > > Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is > using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think > is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 > > src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm);CHKERRQ(ierr); > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > src/ksp/pc/impls/spai/ispai.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_spai));CHKERRQ(ierr); > src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, &comm);CHKERRQ(ierr); > src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); > src/mat/impls/aij/mpi/mumps/mumps.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_mumps));CHKERRQ(ierr); > src/mat/impls/aij/mpi/pastix/pastix.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_comm));CHKERRQ(ierr); > src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_superlu));CHKERRQ(ierr); > src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); > src/mat/partition/impls/pmetis/pmetis.c: ierr = MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); > src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a MPI_Comm_dup() of each of these (duplicates of duplicates return the same communictor) > src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) > src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr); > src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr); > src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out);CHKERRQ(ierr); > src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm);CHKERRQ(ierr); > src/ts/impls/implicit/sundials/sundials.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_sundials));CHKERRQ(ierr); > > Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid these MPI_Comm_dup() calls? > > Satish > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > >> >> Are we sure this is a PETSc comm issue and not a hypre comm duplication issue >> >> frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, participate=, new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] >> >> Looks like hypre is needed to generate subcomms, perhaps it generates too many? >> >> Barry >> >> >>> On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: >>> >>> I?m working with Fande on this and I would like to add a bit more. There are many circumstances where we aren?t working on COMM_WORLD at all (e.g. working on a sub-communicator) but PETSc was initialized using MPI_COMM_WORLD (think multi-level solves)? and we need to create arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering duplication. >>> >>> Can you explain why PETSc needs to duplicate the communicator so much? >>> >>> Thanks for your help in tracking this down! >>> >>> Derek >>> >>> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: >>> Why we do not use user-level MPI communicators directly? What are potential risks here? >>> >>> >>> Fande, >>> >>> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay wrote: >>> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls to MPI_Comm_dup() - thus potentially avoiding such errors >>> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= >>> >>> >>> Satish >>> >>> On Mon, 2 Apr 2018, Kong, Fande wrote: >>> >>>> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay wrote: >>>> >>>>> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? >>>>> >>>>> If so - you could try changing to PETSC_COMM_WORLD >>>>> >>>> >>>> >>>> I do not think we are using PETSC_COMM_WORLD when creating PETSc objects. >>>> Why we can not use MPI_COMM_WORLD? >>>> >>>> >>>> Fande, >>>> >>>> >>>>> >>>>> Satish >>>>> >>>>> >>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its >>>>>> applications. I have a error message for a standard test: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *preconditioners/pbp.lots_of_variables: MPI had an >>>>>> errorpreconditioners/pbp.lots_of_variables: >>>>>> ------------------------------------------------ >>>>> preconditioners/pbp.lots_of_variables: >>>>>> Other MPI error, error stack:preconditioners/pbp.lots_of_variables: >>>>>> PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, >>>>>> new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: >>>>>> PMPI_Comm_dup(162)..................: >>>>>> preconditioners/pbp.lots_of_variables: >>>>>> MPIR_Comm_dup_impl(57)..............: >>>>>> preconditioners/pbp.lots_of_variables: >>>>>> MPIR_Comm_copy(739).................: >>>>>> preconditioners/pbp.lots_of_variables: >>>>>> MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 >>>>> free >>>>>> on this process; ignore_id=0)* >>>>>> >>>>>> >>>>>> I did "git bisect', and the following commit introduces this issue: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano Zampini >>>>>> >Date: Sat Nov 5 >>>>>> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE >>>>>> hpmat already stores two HYPRE vectors* >>>>>> >>>>>> Before I debug line-by-line, anyone has a clue on this? >>>>>> >>>>>> >>>>>> Fande, >>>>>> >>>>> >>>>> >>>> >>> >> From knepley at gmail.com Tue Apr 3 05:53:56 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 3 Apr 2018 06:53:56 -0400 Subject: [petsc-users] read variables from a source file in petsc In-Reply-To: References: Message-ID: On Mon, Apr 2, 2018 at 10:52 PM, Sepideh Kavousi wrote: > Hi all, > > I wrote a petsc code, and I defined my variables in a separate file, > namely common.h, and common.c as following > > > common.h: > > extern int i; > extern int n; > > Notice that these declarations are inconsistent for the main file. > common.c: > > > #include > #include "common.h > PetscScalar i; // Define i and initialize > PetscScalar n; // Define n and initialize > i=2.2; > n=3.5; > > Now in my prog.c code I have: > > #include > #include > static char help[]="program main"; > int main (int argc, char **argv){ > PetscInitialize(&argc,&argv, (char*) 0,help); > PetscPrintf(PETSC_COMM_WORLD,"i=%f!\n",a); > PetscPrintf(PETSC_COMM_WORLD,"n=%f!\n",b) ; > PetscFinalize(); > } > > if my it was not using petsc and it was a normal c code, I could run it > using the following command: > gcc prog. c common.c > but now that I am using petsc, I do not know how should I include common.c > in my makefile. > my makefile looks like this: > > include ${PETSC_DIR}/lib/petsc/conf/variables > include ${PETSC_DIR}/lib/petsc/conf/rules > prog: prog.o chkopts > > -${CLINKER} -o prog.out prog.o ${PETSC_LIB} > > ${RM} *.o > > You want prog: prog.o common.o chkopts -${CLINKER} -o prog.out prog.o common.o ${PETSC_LIB} Matt > .PHONY: distclean rune_1 test > distclean: > > @rm -f *~ prog > > Thanks, > Sepideh > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Apr 3 08:45:45 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 3 Apr 2018 08:45:45 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> Message-ID: Fande claimed 49a781f5cee36db85e8d5b951eec29f10ac13593 made a difference. [so assuming same hypre version was used before and after this commit - for this bisection] So the extra MPI_Comm_dup() calls due to MATHYPRE must be pushing the total communicators over the limit. And wrt debugging - perhaps we need to check MPI_Comm_free() aswell? Presumably freed communicators can get reused so we have to look for outstanding/unfreed communicators? Per message below - MPICH[?] provides a max of 2048 communicators. And there is some discussion of this issue at: https://lists.mpich.org/pipermail/discuss/2012-December/000148.html And wrt 'sharing' - I was thining in terms of: Can one use MPI_COMM_WORLD with all hypre objects we create? If so - we could somehow attach one more inner-comm - that could be obtained and reused with multiple hypre objects [that got created off the same petsc_comm?] Satish On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > Each external package definitely needs its own duplicated communicator; cannot share between packages. > > The only problem with the dups below is if they are in a loop and get called many times. > > To debug the hypre/duplication issue in MOOSE I would run in the debugger with a break point in MPI_Comm_dup() and see > who keeps calling it an unreasonable amount of times. (My guess is this is a new "feature" in hypre that they will need to fix but only debugging will tell) > > Barry > > > > On Apr 2, 2018, at 7:44 PM, Balay, Satish wrote: > > > > We do a MPI_Comm_dup() for objects related to externalpackages. > > > > Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is > > using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think > > is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 > > > > src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm);CHKERRQ(ierr); > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > src/ksp/pc/impls/spai/ispai.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_spai));CHKERRQ(ierr); > > src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, &comm);CHKERRQ(ierr); > > src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); > > src/mat/impls/aij/mpi/mumps/mumps.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_mumps));CHKERRQ(ierr); > > src/mat/impls/aij/mpi/pastix/pastix.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_comm));CHKERRQ(ierr); > > src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_superlu));CHKERRQ(ierr); > > src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); > > src/mat/partition/impls/pmetis/pmetis.c: ierr = MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); > > src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a MPI_Comm_dup() of each of these (duplicates of duplicates return the same communictor) > > src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) > > src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr); > > src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr); > > src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out);CHKERRQ(ierr); > > src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm);CHKERRQ(ierr); > > src/ts/impls/implicit/sundials/sundials.c: ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_sundials));CHKERRQ(ierr); > > > > Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid these MPI_Comm_dup() calls? > > > > Satish > > > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > >> > >> Are we sure this is a PETSc comm issue and not a hypre comm duplication issue > >> > >> frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, participate=, new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] > >> > >> Looks like hypre is needed to generate subcomms, perhaps it generates too many? > >> > >> Barry > >> > >> > >>> On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: > >>> > >>> I?m working with Fande on this and I would like to add a bit more. There are many circumstances where we aren?t working on COMM_WORLD at all (e.g. working on a sub-communicator) but PETSc was initialized using MPI_COMM_WORLD (think multi-level solves)? and we need to create arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering duplication. > >>> > >>> Can you explain why PETSc needs to duplicate the communicator so much? > >>> > >>> Thanks for your help in tracking this down! > >>> > >>> Derek > >>> > >>> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: > >>> Why we do not use user-level MPI communicators directly? What are potential risks here? > >>> > >>> > >>> Fande, > >>> > >>> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay wrote: > >>> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls to MPI_Comm_dup() - thus potentially avoiding such errors > >>> > >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > >>> > >>> > >>> Satish > >>> > >>> On Mon, 2 Apr 2018, Kong, Fande wrote: > >>> > >>>> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay wrote: > >>>> > >>>>> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? > >>>>> > >>>>> If so - you could try changing to PETSC_COMM_WORLD > >>>>> > >>>> > >>>> > >>>> I do not think we are using PETSC_COMM_WORLD when creating PETSc objects. > >>>> Why we can not use MPI_COMM_WORLD? > >>>> > >>>> > >>>> Fande, > >>>> > >>>> > >>>>> > >>>>> Satish > >>>>> > >>>>> > >>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: > >>>>> > >>>>>> Hi All, > >>>>>> > >>>>>> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > >>>>>> applications. I have a error message for a standard test: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> *preconditioners/pbp.lots_of_variables: MPI had an > >>>>>> errorpreconditioners/pbp.lots_of_variables: > >>>>>> ------------------------------------------------ > >>>>> preconditioners/pbp.lots_of_variables: > >>>>>> Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > >>>>>> PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > >>>>>> new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > >>>>>> PMPI_Comm_dup(162)..................: > >>>>>> preconditioners/pbp.lots_of_variables: > >>>>>> MPIR_Comm_dup_impl(57)..............: > >>>>>> preconditioners/pbp.lots_of_variables: > >>>>>> MPIR_Comm_copy(739).................: > >>>>>> preconditioners/pbp.lots_of_variables: > >>>>>> MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 > >>>>> free > >>>>>> on this process; ignore_id=0)* > >>>>>> > >>>>>> > >>>>>> I did "git bisect', and the following commit introduces this issue: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano Zampini > >>>>>> >Date: Sat Nov 5 > >>>>>> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > >>>>>> hpmat already stores two HYPRE vectors* > >>>>>> > >>>>>> Before I debug line-by-line, anyone has a clue on this? > >>>>>> > >>>>>> > >>>>>> Fande, > >>>>>> > >>>>> > >>>>> > >>>> > >>> > >> > > From skavou1 at lsu.edu Tue Apr 3 08:46:07 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Tue, 3 Apr 2018 13:46:07 +0000 Subject: [petsc-users] read variables from a source file in petsc In-Reply-To: References: , Message-ID: thanks. It works Sepideh ________________________________ From: Matthew Knepley Sent: Tuesday, April 3, 2018 5:53:56 AM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] read variables from a source file in petsc On Mon, Apr 2, 2018 at 10:52 PM, Sepideh Kavousi > wrote: Hi all, I wrote a petsc code, and I defined my variables in a separate file, namely common.h, and common.c as following common.h: extern int i; extern int n; Notice that these declarations are inconsistent for the main file. common.c: #include #include "common.h PetscScalar i; // Define i and initialize PetscScalar n; // Define n and initialize i=2.2; n=3.5; Now in my prog.c code I have: #include #include static char help[]="program main"; int main (int argc, char **argv){ PetscInitialize(&argc,&argv, (char*) 0,help); PetscPrintf(PETSC_COMM_WORLD,"i=%f!\n",a); PetscPrintf(PETSC_COMM_WORLD,"n=%f!\n",b) ; PetscFinalize(); } if my it was not using petsc and it was a normal c code, I could run it using the following command: gcc prog. c common.c but now that I am using petsc, I do not know how should I include common.c in my makefile. my makefile looks like this: include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules prog: prog.o chkopts -${CLINKER} -o prog.out prog.o ${PETSC_LIB} ${RM} *.o You want prog: prog.o common.o chkopts -${CLINKER} -o prog.out prog.o common.o ${PETSC_LIB} Matt .PHONY: distclean rune_1 test distclean: @rm -f *~ prog Thanks, Sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedmud at gmail.com Tue Apr 3 09:37:55 2018 From: friedmud at gmail.com (Derek Gaston) Date: Tue, 03 Apr 2018 14:37:55 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> Message-ID: I like the idea that Hypre (as a package) would get _one_ comm (for all the solvers/matrices created) that was duped from the one given to PETSc in Vec/MatCreate(). Seems like the tricky part would be figuring out _which_ comm that is based on the incoming comm. For instance - we would definitely have the case where we are doing a Hypre solve on effectively COMM_WORLD? and then many Hypre solves on sub-communicators (and even Hypre solves on sub-communicators of those sub-communicators). The system for getting ?the? Hypre Comm would have to match up the incoming Comm in the Vec/MatCreate() call and find the correct Hypre comm to use. Derek On Tue, Apr 3, 2018 at 7:46 AM Satish Balay wrote: > Fande claimed 49a781f5cee36db85e8d5b951eec29f10ac13593 made a difference. > [so assuming same hypre version was used before and after this commit - for > this bisection] > > So the extra MPI_Comm_dup() calls due to MATHYPRE must be pushing the > total communicators over the limit. > > And wrt debugging - perhaps we need to check MPI_Comm_free() aswell? > Presumably freed communicators can get reused so we have to look for > outstanding/unfreed communicators? > > Per message below - MPICH[?] provides a max of 2048 communicators. And > there is some discussion of this issue at: > https://lists.mpich.org/pipermail/discuss/2012-December/000148.html > > And wrt 'sharing' - I was thining in terms of: Can one use MPI_COMM_WORLD > with all hypre objects we create? If so - we could somehow attach one more > inner-comm - that could be obtained and reused with multiple hypre objects > [that got created off the same petsc_comm?] > > Satish > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > > > Each external package definitely needs its own duplicated > communicator; cannot share between packages. > > > > The only problem with the dups below is if they are in a loop and get > called many times. > > > > To debug the hypre/duplication issue in MOOSE I would run in the > debugger with a break point in MPI_Comm_dup() and see > > who keeps calling it an unreasonable amount of times. (My guess is this > is a new "feature" in hypre that they will need to fix but only debugging > will tell) > > > > Barry > > > > > > > On Apr 2, 2018, at 7:44 PM, Balay, Satish wrote: > > > > > > We do a MPI_Comm_dup() for objects related to externalpackages. > > > > > > Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is > > > using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think > > > is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 > > > > > > src/dm/impls/da/hypre/mhyp.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > > src/dm/impls/da/hypre/mhyp.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > > src/dm/impls/swarm/data_ex.c: ierr = > MPI_Comm_dup(comm,&d->comm);CHKERRQ(ierr); > > > src/ksp/pc/impls/hypre/hypre.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); > > > src/ksp/pc/impls/hypre/hypre.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > > src/ksp/pc/impls/hypre/hypre.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > > src/ksp/pc/impls/spai/ispai.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_spai));CHKERRQ(ierr); > > > src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, > &comm);CHKERRQ(ierr); > > > src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); > > > src/mat/impls/aij/mpi/mumps/mumps.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_mumps));CHKERRQ(ierr); > > > src/mat/impls/aij/mpi/pastix/pastix.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_comm));CHKERRQ(ierr); > > > src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_superlu));CHKERRQ(ierr); > > > src/mat/impls/hypre/mhypre.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); > > > src/mat/partition/impls/pmetis/pmetis.c: ierr = > MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); > > > src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a > MPI_Comm_dup() of each of these (duplicates of duplicates return the same > communictor) > > > src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) > > > src/sys/objects/pinit.c: ierr = > MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr); > > > src/sys/objects/pinit.c: ierr = > MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr); > > > src/sys/objects/tagm.c: ierr = > MPI_Comm_dup(comm_in,comm_out);CHKERRQ(ierr); > > > src/sys/utils/mpiu.c: ierr = > MPI_Comm_dup(comm,&local_comm);CHKERRQ(ierr); > > > src/ts/impls/implicit/sundials/sundials.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_sundials));CHKERRQ(ierr); > > > > > > Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid > these MPI_Comm_dup() calls? > > > > > > Satish > > > > > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > > > >> > > >> Are we sure this is a PETSc comm issue and not a hypre comm > duplication issue > > >> > > >> frame #6: 0x00000001061345d9 > libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, > participate=, new_comm_ptr=) + 409 at > gen_redcs_mat.c:531 [opt] > > >> > > >> Looks like hypre is needed to generate subcomms, perhaps it generates > too many? > > >> > > >> Barry > > >> > > >> > > >>> On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: > > >>> > > >>> I?m working with Fande on this and I would like to add a bit more. > There are many circumstances where we aren?t working on COMM_WORLD at all > (e.g. working on a sub-communicator) but PETSc was initialized using > MPI_COMM_WORLD (think multi-level solves)? and we need to create > arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We > definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering > duplication. > > >>> > > >>> Can you explain why PETSc needs to duplicate the communicator so > much? > > >>> > > >>> Thanks for your help in tracking this down! > > >>> > > >>> Derek > > >>> > > >>> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande > wrote: > > >>> Why we do not use user-level MPI communicators directly? What are > potential risks here? > > >>> > > >>> > > >>> Fande, > > >>> > > >>> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay > wrote: > > >>> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize > calls to MPI_Comm_dup() - thus potentially avoiding such errors > > >>> > > >>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > > >>> > > >>> > > >>> Satish > > >>> > > >>> On Mon, 2 Apr 2018, Kong, Fande wrote: > > >>> > > >>>> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay > wrote: > > >>>> > > >>>>> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc > objects? > > >>>>> > > >>>>> If so - you could try changing to PETSC_COMM_WORLD > > >>>>> > > >>>> > > >>>> > > >>>> I do not think we are using PETSC_COMM_WORLD when creating PETSc > objects. > > >>>> Why we can not use MPI_COMM_WORLD? > > >>>> > > >>>> > > >>>> Fande, > > >>>> > > >>>> > > >>>>> > > >>>>> Satish > > >>>>> > > >>>>> > > >>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: > > >>>>> > > >>>>>> Hi All, > > >>>>>> > > >>>>>> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > > >>>>>> applications. I have a error message for a standard test: > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> *preconditioners/pbp.lots_of_variables: MPI had an > > >>>>>> errorpreconditioners/pbp.lots_of_variables: > > >>>>>> ------------------------------------------------ > > >>>>> preconditioners/pbp.lots_of_variables: > > >>>>>> Other MPI error, error > stack:preconditioners/pbp.lots_of_variables: > > >>>>>> PMPI_Comm_dup(177)..................: > MPI_Comm_dup(comm=0x84000001, > > >>>>>> new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > > >>>>>> PMPI_Comm_dup(162)..................: > > >>>>>> preconditioners/pbp.lots_of_variables: > > >>>>>> MPIR_Comm_dup_impl(57)..............: > > >>>>>> preconditioners/pbp.lots_of_variables: > > >>>>>> MPIR_Comm_copy(739).................: > > >>>>>> preconditioners/pbp.lots_of_variables: > > >>>>>> MPIR_Get_contextid_sparse_group(614): Too many communicators > (0/2048 > > >>>>> free > > >>>>>> on this process; ignore_id=0)* > > >>>>>> > > >>>>>> > > >>>>>> I did "git bisect', and the following commit introduces this > issue: > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano > Zampini > > >>>>>> >Date: > Sat Nov 5 > > >>>>>> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > > >>>>>> hpmat already stores two HYPRE vectors* > > >>>>>> > > >>>>>> Before I debug line-by-line, anyone has a clue on this? > > >>>>>> > > >>>>>> > > >>>>>> Fande, > > >>>>>> > > >>>>> > > >>>>> > > >>>> > > >>> > > >> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Tue Apr 3 09:46:41 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Tue, 3 Apr 2018 08:46:41 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> Message-ID: On Tue, Apr 3, 2018 at 1:17 AM, Smith, Barry F. wrote: > > Each external package definitely needs its own duplicated communicator; > cannot share between packages. > > The only problem with the dups below is if they are in a loop and get > called many times. > The "standard test" that has this issue actually has 1K fields. MOOSE creates its own field-split preconditioner (not based on the PETSc fieldsplit), and each filed is associated with one PC HYPRE. If PETSc duplicates communicators, we should easily reach the limit 2048. I also want to confirm what extra communicators are introduced in the bad commit. Fande, > > To debug the hypre/duplication issue in MOOSE I would run in the > debugger with a break point in MPI_Comm_dup() and see > who keeps calling it an unreasonable amount of times. (My guess is this is > a new "feature" in hypre that they will need to fix but only debugging will > tell) > > Barry > > > > On Apr 2, 2018, at 7:44 PM, Balay, Satish wrote: > > > > We do a MPI_Comm_dup() for objects related to externalpackages. > > > > Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is > > using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think > > is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 > > > > src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( > PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( > PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm); > CHKERRQ(ierr); > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > src/ksp/pc/impls/spai/ispai.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_ > spai));CHKERRQ(ierr); > > src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, > &comm);CHKERRQ(ierr); > > src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_ > cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); > > src/mat/impls/aij/mpi/mumps/mumps.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_ > mumps));CHKERRQ(ierr); > > src/mat/impls/aij/mpi/pastix/pastix.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_ > comm));CHKERRQ(ierr); > > src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_ > superlu));CHKERRQ(ierr); > > src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > PetscObject)B),&hB->comm);CHKERRQ(ierr); > > src/mat/partition/impls/pmetis/pmetis.c: ierr = > MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); > > src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a > MPI_Comm_dup() of each of these (duplicates of duplicates return the same > communictor) > > src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) > > src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& > local_comm);CHKERRQ(ierr); > > src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& > local_comm);CHKERRQ(ierr); > > src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out) > ;CHKERRQ(ierr); > > src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm) > ;CHKERRQ(ierr); > > src/ts/impls/implicit/sundials/sundials.c: ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_ > sundials));CHKERRQ(ierr); > > > > Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid > these MPI_Comm_dup() calls? > > > > Satish > > > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > >> > >> Are we sure this is a PETSc comm issue and not a hypre comm > duplication issue > >> > >> frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_ > GenerateSubComm(comm=-1006627852, participate=, > new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] > >> > >> Looks like hypre is needed to generate subcomms, perhaps it generates > too many? > >> > >> Barry > >> > >> > >>> On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: > >>> > >>> I?m working with Fande on this and I would like to add a bit more. > There are many circumstances where we aren?t working on COMM_WORLD at all > (e.g. working on a sub-communicator) but PETSc was initialized using > MPI_COMM_WORLD (think multi-level solves)? and we need to create > arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We > definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering > duplication. > >>> > >>> Can you explain why PETSc needs to duplicate the communicator so much? > >>> > >>> Thanks for your help in tracking this down! > >>> > >>> Derek > >>> > >>> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: > >>> Why we do not use user-level MPI communicators directly? What are > potential risks here? > >>> > >>> > >>> Fande, > >>> > >>> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay > wrote: > >>> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls > to MPI_Comm_dup() - thus potentially avoiding such errors > >>> > >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs. > anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_ > PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi > CY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_ > zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > >>> > >>> > >>> Satish > >>> > >>> On Mon, 2 Apr 2018, Kong, Fande wrote: > >>> > >>>> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay > wrote: > >>>> > >>>>> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? > >>>>> > >>>>> If so - you could try changing to PETSC_COMM_WORLD > >>>>> > >>>> > >>>> > >>>> I do not think we are using PETSC_COMM_WORLD when creating PETSc > objects. > >>>> Why we can not use MPI_COMM_WORLD? > >>>> > >>>> > >>>> Fande, > >>>> > >>>> > >>>>> > >>>>> Satish > >>>>> > >>>>> > >>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: > >>>>> > >>>>>> Hi All, > >>>>>> > >>>>>> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > >>>>>> applications. I have a error message for a standard test: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> *preconditioners/pbp.lots_of_variables: MPI had an > >>>>>> errorpreconditioners/pbp.lots_of_variables: > >>>>>> ------------------------------------------------ > >>>>> preconditioners/pbp.lots_of_variables: > >>>>>> Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > >>>>>> PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > >>>>>> new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > >>>>>> PMPI_Comm_dup(162)..................: > >>>>>> preconditioners/pbp.lots_of_variables: > >>>>>> MPIR_Comm_dup_impl(57)..............: > >>>>>> preconditioners/pbp.lots_of_variables: > >>>>>> MPIR_Comm_copy(739).................: > >>>>>> preconditioners/pbp.lots_of_variables: > >>>>>> MPIR_Get_contextid_sparse_group(614): Too many communicators > (0/2048 > >>>>> free > >>>>>> on this process; ignore_id=0)* > >>>>>> > >>>>>> > >>>>>> I did "git bisect', and the following commit introduces this issue: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano > Zampini > >>>>>> >Date: Sat > Nov 5 > >>>>>> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > >>>>>> hpmat already stores two HYPRE vectors* > >>>>>> > >>>>>> Before I debug line-by-line, anyone has a clue on this? > >>>>>> > >>>>>> > >>>>>> Fande, > >>>>>> > >>>>> > >>>>> > >>>> > >>> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Apr 3 09:50:17 2018 From: jed at jedbrown.org (Jed Brown) Date: Tue, 03 Apr 2018 08:50:17 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> Message-ID: <87a7uktkd2.fsf@jedbrown.org> The PETSc model is that the "outer" communicator (passed by the caller) is dup'd to create an "inner" communicator which as attached (using MPI attributes) to the outer communicator. In the future, PETSc will find the inner communicator and use that, instead of dup'ing again. Derek Gaston writes: > I like the idea that Hypre (as a package) would get _one_ comm (for all the > solvers/matrices created) that was duped from the one given to PETSc in > Vec/MatCreate(). > > Seems like the tricky part would be figuring out _which_ comm that is based > on the incoming comm. For instance - we would definitely have the case > where we are doing a Hypre solve on effectively COMM_WORLD? and then many > Hypre solves on sub-communicators (and even Hypre solves on > sub-communicators of those sub-communicators). The system for getting > ?the? Hypre Comm would have to match up the incoming Comm in the > Vec/MatCreate() call and find the correct Hypre comm to use. > > Derek > > > > On Tue, Apr 3, 2018 at 7:46 AM Satish Balay wrote: > >> Fande claimed 49a781f5cee36db85e8d5b951eec29f10ac13593 made a difference. >> [so assuming same hypre version was used before and after this commit - for >> this bisection] >> >> So the extra MPI_Comm_dup() calls due to MATHYPRE must be pushing the >> total communicators over the limit. >> >> And wrt debugging - perhaps we need to check MPI_Comm_free() aswell? >> Presumably freed communicators can get reused so we have to look for >> outstanding/unfreed communicators? >> >> Per message below - MPICH[?] provides a max of 2048 communicators. And >> there is some discussion of this issue at: >> https://lists.mpich.org/pipermail/discuss/2012-December/000148.html >> >> And wrt 'sharing' - I was thining in terms of: Can one use MPI_COMM_WORLD >> with all hypre objects we create? If so - we could somehow attach one more >> inner-comm - that could be obtained and reused with multiple hypre objects >> [that got created off the same petsc_comm?] >> >> Satish >> >> On Tue, 3 Apr 2018, Smith, Barry F. wrote: >> >> > >> > Each external package definitely needs its own duplicated >> communicator; cannot share between packages. >> > >> > The only problem with the dups below is if they are in a loop and get >> called many times. >> > >> > To debug the hypre/duplication issue in MOOSE I would run in the >> debugger with a break point in MPI_Comm_dup() and see >> > who keeps calling it an unreasonable amount of times. (My guess is this >> is a new "feature" in hypre that they will need to fix but only debugging >> will tell) >> > >> > Barry >> > >> > >> > > On Apr 2, 2018, at 7:44 PM, Balay, Satish wrote: >> > > >> > > We do a MPI_Comm_dup() for objects related to externalpackages. >> > > >> > > Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is >> > > using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think >> > > is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 >> > > >> > > src/dm/impls/da/hypre/mhyp.c: ierr = >> MPI_Comm_dup(PetscObjectComm((PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); >> > > src/dm/impls/da/hypre/mhyp.c: ierr = >> MPI_Comm_dup(PetscObjectComm((PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); >> > > src/dm/impls/swarm/data_ex.c: ierr = >> MPI_Comm_dup(comm,&d->comm);CHKERRQ(ierr); >> > > src/ksp/pc/impls/hypre/hypre.c: ierr = >> MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); >> > > src/ksp/pc/impls/hypre/hypre.c: ierr = >> MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); >> > > src/ksp/pc/impls/hypre/hypre.c: ierr = >> MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); >> > > src/ksp/pc/impls/spai/ispai.c: ierr = >> MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_spai));CHKERRQ(ierr); >> > > src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, >> &comm);CHKERRQ(ierr); >> > > src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = >> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); >> > > src/mat/impls/aij/mpi/mumps/mumps.c: ierr = >> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_mumps));CHKERRQ(ierr); >> > > src/mat/impls/aij/mpi/pastix/pastix.c: ierr = >> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_comm));CHKERRQ(ierr); >> > > src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = >> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_superlu));CHKERRQ(ierr); >> > > src/mat/impls/hypre/mhypre.c: ierr = >> MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); >> > > src/mat/partition/impls/pmetis/pmetis.c: ierr = >> MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); >> > > src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a >> MPI_Comm_dup() of each of these (duplicates of duplicates return the same >> communictor) >> > > src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) >> > > src/sys/objects/pinit.c: ierr = >> MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr); >> > > src/sys/objects/pinit.c: ierr = >> MPI_Comm_dup(MPI_COMM_WORLD,&local_comm);CHKERRQ(ierr); >> > > src/sys/objects/tagm.c: ierr = >> MPI_Comm_dup(comm_in,comm_out);CHKERRQ(ierr); >> > > src/sys/utils/mpiu.c: ierr = >> MPI_Comm_dup(comm,&local_comm);CHKERRQ(ierr); >> > > src/ts/impls/implicit/sundials/sundials.c: ierr = >> MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_sundials));CHKERRQ(ierr); >> > > >> > > Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid >> these MPI_Comm_dup() calls? >> > > >> > > Satish >> > > >> > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: >> > > >> > >> >> > >> Are we sure this is a PETSc comm issue and not a hypre comm >> duplication issue >> > >> >> > >> frame #6: 0x00000001061345d9 >> libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, >> participate=, new_comm_ptr=) + 409 at >> gen_redcs_mat.c:531 [opt] >> > >> >> > >> Looks like hypre is needed to generate subcomms, perhaps it generates >> too many? >> > >> >> > >> Barry >> > >> >> > >> >> > >>> On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: >> > >>> >> > >>> I?m working with Fande on this and I would like to add a bit more. >> There are many circumstances where we aren?t working on COMM_WORLD at all >> (e.g. working on a sub-communicator) but PETSc was initialized using >> MPI_COMM_WORLD (think multi-level solves)? and we need to create >> arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We >> definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering >> duplication. >> > >>> >> > >>> Can you explain why PETSc needs to duplicate the communicator so >> much? >> > >>> >> > >>> Thanks for your help in tracking this down! >> > >>> >> > >>> Derek >> > >>> >> > >>> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande >> wrote: >> > >>> Why we do not use user-level MPI communicators directly? What are >> potential risks here? >> > >>> >> > >>> >> > >>> Fande, >> > >>> >> > >>> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay >> wrote: >> > >>> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize >> calls to MPI_Comm_dup() - thus potentially avoiding such errors >> > >>> >> > >>> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= >> > >>> >> > >>> >> > >>> Satish >> > >>> >> > >>> On Mon, 2 Apr 2018, Kong, Fande wrote: >> > >>> >> > >>>> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay >> wrote: >> > >>>> >> > >>>>> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc >> objects? >> > >>>>> >> > >>>>> If so - you could try changing to PETSC_COMM_WORLD >> > >>>>> >> > >>>> >> > >>>> >> > >>>> I do not think we are using PETSC_COMM_WORLD when creating PETSc >> objects. >> > >>>> Why we can not use MPI_COMM_WORLD? >> > >>>> >> > >>>> >> > >>>> Fande, >> > >>>> >> > >>>> >> > >>>>> >> > >>>>> Satish >> > >>>>> >> > >>>>> >> > >>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: >> > >>>>> >> > >>>>>> Hi All, >> > >>>>>> >> > >>>>>> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its >> > >>>>>> applications. I have a error message for a standard test: >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> *preconditioners/pbp.lots_of_variables: MPI had an >> > >>>>>> errorpreconditioners/pbp.lots_of_variables: >> > >>>>>> ------------------------------------------------ >> > >>>>> preconditioners/pbp.lots_of_variables: >> > >>>>>> Other MPI error, error >> stack:preconditioners/pbp.lots_of_variables: >> > >>>>>> PMPI_Comm_dup(177)..................: >> MPI_Comm_dup(comm=0x84000001, >> > >>>>>> new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: >> > >>>>>> PMPI_Comm_dup(162)..................: >> > >>>>>> preconditioners/pbp.lots_of_variables: >> > >>>>>> MPIR_Comm_dup_impl(57)..............: >> > >>>>>> preconditioners/pbp.lots_of_variables: >> > >>>>>> MPIR_Comm_copy(739).................: >> > >>>>>> preconditioners/pbp.lots_of_variables: >> > >>>>>> MPIR_Get_contextid_sparse_group(614): Too many communicators >> (0/2048 >> > >>>>> free >> > >>>>>> on this process; ignore_id=0)* >> > >>>>>> >> > >>>>>> >> > >>>>>> I did "git bisect', and the following commit introduces this >> issue: >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano >> Zampini >> > >>>>>> >Date: >> Sat Nov 5 >> > >>>>>> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE >> > >>>>>> hpmat already stores two HYPRE vectors* >> > >>>>>> >> > >>>>>> Before I debug line-by-line, anyone has a clue on this? >> > >>>>>> >> > >>>>>> >> > >>>>>> Fande, >> > >>>>>> >> > >>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> >> > >> > >> From balay at mcs.anl.gov Tue Apr 3 09:58:00 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 3 Apr 2018 09:58:00 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> Message-ID: On Tue, 3 Apr 2018, Kong, Fande wrote: > On Tue, Apr 3, 2018 at 1:17 AM, Smith, Barry F. wrote: > > > > > Each external package definitely needs its own duplicated communicator; > > cannot share between packages. > > > > The only problem with the dups below is if they are in a loop and get > > called many times. > > > > > The "standard test" that has this issue actually has 1K fields. MOOSE > creates its own field-split preconditioner (not based on the PETSc > fieldsplit), and each filed is associated with one PC HYPRE. If PETSc > duplicates communicators, we should easily reach the limit 2048. > > I also want to confirm what extra communicators are introduced in the bad > commit. To me it looks like there is 1 extra comm created [for MATHYPRE] for each PCHYPRE that is created [which also creates one comm for this object]. But you might want to verify [by linking with mpi trace library?] There are some debugging hints at https://lists.mpich.org/pipermail/discuss/2012-December/000148.html [wrt mpich] - which I haven't checked.. Satish > > > Fande, > > > > > > > To debug the hypre/duplication issue in MOOSE I would run in the > > debugger with a break point in MPI_Comm_dup() and see > > who keeps calling it an unreasonable amount of times. (My guess is this is > > a new "feature" in hypre that they will need to fix but only debugging will > > tell) > > > > Barry > > > > > > > On Apr 2, 2018, at 7:44 PM, Balay, Satish wrote: > > > > > > We do a MPI_Comm_dup() for objects related to externalpackages. > > > > > > Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is > > > using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think > > > is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 > > > > > > src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > > src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > > src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm); > > CHKERRQ(ierr); > > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); > > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > > src/ksp/pc/impls/spai/ispai.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_ > > spai));CHKERRQ(ierr); > > > src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, > > &comm);CHKERRQ(ierr); > > > src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_ > > cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); > > > src/mat/impls/aij/mpi/mumps/mumps.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_ > > mumps));CHKERRQ(ierr); > > > src/mat/impls/aij/mpi/pastix/pastix.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_ > > comm));CHKERRQ(ierr); > > > src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_ > > superlu));CHKERRQ(ierr); > > > src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)B),&hB->comm);CHKERRQ(ierr); > > > src/mat/partition/impls/pmetis/pmetis.c: ierr = > > MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); > > > src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a > > MPI_Comm_dup() of each of these (duplicates of duplicates return the same > > communictor) > > > src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) > > > src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& > > local_comm);CHKERRQ(ierr); > > > src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& > > local_comm);CHKERRQ(ierr); > > > src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out) > > ;CHKERRQ(ierr); > > > src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm) > > ;CHKERRQ(ierr); > > > src/ts/impls/implicit/sundials/sundials.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_ > > sundials));CHKERRQ(ierr); > > > > > > Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid > > these MPI_Comm_dup() calls? > > > > > > Satish > > > > > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > > > >> > > >> Are we sure this is a PETSc comm issue and not a hypre comm > > duplication issue > > >> > > >> frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_ > > GenerateSubComm(comm=-1006627852, participate=, > > new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] > > >> > > >> Looks like hypre is needed to generate subcomms, perhaps it generates > > too many? > > >> > > >> Barry > > >> > > >> > > >>> On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: > > >>> > > >>> I?m working with Fande on this and I would like to add a bit more. > > There are many circumstances where we aren?t working on COMM_WORLD at all > > (e.g. working on a sub-communicator) but PETSc was initialized using > > MPI_COMM_WORLD (think multi-level solves)? and we need to create > > arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We > > definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering > > duplication. > > >>> > > >>> Can you explain why PETSc needs to duplicate the communicator so much? > > >>> > > >>> Thanks for your help in tracking this down! > > >>> > > >>> Derek > > >>> > > >>> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: > > >>> Why we do not use user-level MPI communicators directly? What are > > potential risks here? > > >>> > > >>> > > >>> Fande, > > >>> > > >>> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay > > wrote: > > >>> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls > > to MPI_Comm_dup() - thus potentially avoiding such errors > > >>> > > >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs. > > anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_ > > PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > > _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi > > CY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_ > > zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > > >>> > > >>> > > >>> Satish > > >>> > > >>> On Mon, 2 Apr 2018, Kong, Fande wrote: > > >>> > > >>>> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay > > wrote: > > >>>> > > >>>>> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? > > >>>>> > > >>>>> If so - you could try changing to PETSC_COMM_WORLD > > >>>>> > > >>>> > > >>>> > > >>>> I do not think we are using PETSC_COMM_WORLD when creating PETSc > > objects. > > >>>> Why we can not use MPI_COMM_WORLD? > > >>>> > > >>>> > > >>>> Fande, > > >>>> > > >>>> > > >>>>> > > >>>>> Satish > > >>>>> > > >>>>> > > >>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: > > >>>>> > > >>>>>> Hi All, > > >>>>>> > > >>>>>> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > > >>>>>> applications. I have a error message for a standard test: > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> *preconditioners/pbp.lots_of_variables: MPI had an > > >>>>>> errorpreconditioners/pbp.lots_of_variables: > > >>>>>> ------------------------------------------------ > > >>>>> preconditioners/pbp.lots_of_variables: > > >>>>>> Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > > >>>>>> PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > > >>>>>> new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > > >>>>>> PMPI_Comm_dup(162)..................: > > >>>>>> preconditioners/pbp.lots_of_variables: > > >>>>>> MPIR_Comm_dup_impl(57)..............: > > >>>>>> preconditioners/pbp.lots_of_variables: > > >>>>>> MPIR_Comm_copy(739).................: > > >>>>>> preconditioners/pbp.lots_of_variables: > > >>>>>> MPIR_Get_contextid_sparse_group(614): Too many communicators > > (0/2048 > > >>>>> free > > >>>>>> on this process; ignore_id=0)* > > >>>>>> > > >>>>>> > > >>>>>> I did "git bisect', and the following commit introduces this issue: > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano > > Zampini > > >>>>>> >Date: Sat > > Nov 5 > > >>>>>> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > > >>>>>> hpmat already stores two HYPRE vectors* > > >>>>>> > > >>>>>> Before I debug line-by-line, anyone has a clue on this? > > >>>>>> > > >>>>>> > > >>>>>> Fande, > > >>>>>> > > >>>>> > > >>>>> > > >>>> > > >>> > > >> > > > > > From stefano.zampini at gmail.com Tue Apr 3 10:12:04 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Tue, 3 Apr 2018 17:12:04 +0200 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> Message-ID: <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> > On Apr 3, 2018, at 4:58 PM, Satish Balay wrote: > > On Tue, 3 Apr 2018, Kong, Fande wrote: > >> On Tue, Apr 3, 2018 at 1:17 AM, Smith, Barry F. wrote: >> >>> >>> Each external package definitely needs its own duplicated communicator; >>> cannot share between packages. >>> >>> The only problem with the dups below is if they are in a loop and get >>> called many times. >>> >> >> >> The "standard test" that has this issue actually has 1K fields. MOOSE >> creates its own field-split preconditioner (not based on the PETSc >> fieldsplit), and each filed is associated with one PC HYPRE. If PETSc >> duplicates communicators, we should easily reach the limit 2048. >> >> I also want to confirm what extra communicators are introduced in the bad >> commit. > > To me it looks like there is 1 extra comm created [for MATHYPRE] for each PCHYPRE that is created [which also creates one comm for this object]. > You?re right; however, it was the same before the commit. I don?t understand how this specific commit is related with this issue, being the error not in the MPI_Comm_Dup which is inside MatCreate_MATHYPRE. Actually, the error comes from MPI_Comm_create frame #5: 0x00000001068defd4 libmpi.12.dylib`MPI_Comm_create + 3492 frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, participate=, new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] frame #7: 0x000000010618f8ba libpetsc.3.07.dylib`hypre_GaussElimSetup(amg_data=0x00007fe7ff857a00, level=, relax_type=9) + 74 at par_relax.c:4209 [opt] frame #8: 0x0000000106140e93 libpetsc.3.07.dylib`hypre_BoomerAMGSetup(amg_vdata=, A=0x00007fe80842aff0, f=0x00007fe80842a980, u=0x00007fe80842a510) + 17699 at par_amg_setup.c:2108 [opt] frame #9: 0x0000000105ec773c libpetsc.3.07.dylib`PCSetUp_HYPRE(pc=) + 2540 at hypre.c:226 [opt How did you perform the bisection? make clean + make all ? Which version of HYPRE are you using? > But you might want to verify [by linking with mpi trace library?] > > > There are some debugging hints at https://lists.mpich.org/pipermail/discuss/2012-December/000148.html [wrt mpich] - which I haven't checked.. > > Satish > >> >> >> Fande, >> >> >> >>> >>> To debug the hypre/duplication issue in MOOSE I would run in the >>> debugger with a break point in MPI_Comm_dup() and see >>> who keeps calling it an unreasonable amount of times. (My guess is this is >>> a new "feature" in hypre that they will need to fix but only debugging will >>> tell) >>> >>> Barry >>> >>> >>>> On Apr 2, 2018, at 7:44 PM, Balay, Satish wrote: >>>> >>>> We do a MPI_Comm_dup() for objects related to externalpackages. >>>> >>>> Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is >>>> using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think >>>> is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 >>>> >>>> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>> PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); >>>> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>> PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); >>>> src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm); >>> CHKERRQ(ierr); >>>> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>> PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); >>>> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>> PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); >>>> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>> PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); >>>> src/ksp/pc/impls/spai/ispai.c: ierr = >>> MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_ >>> spai));CHKERRQ(ierr); >>>> src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, >>> &comm);CHKERRQ(ierr); >>>> src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = >>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_ >>> cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); >>>> src/mat/impls/aij/mpi/mumps/mumps.c: ierr = >>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_ >>> mumps));CHKERRQ(ierr); >>>> src/mat/impls/aij/mpi/pastix/pastix.c: ierr = >>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_ >>> comm));CHKERRQ(ierr); >>>> src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = >>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_ >>> superlu));CHKERRQ(ierr); >>>> src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>> PetscObject)B),&hB->comm);CHKERRQ(ierr); >>>> src/mat/partition/impls/pmetis/pmetis.c: ierr = >>> MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); >>>> src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a >>> MPI_Comm_dup() of each of these (duplicates of duplicates return the same >>> communictor) >>>> src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) >>>> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& >>> local_comm);CHKERRQ(ierr); >>>> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& >>> local_comm);CHKERRQ(ierr); >>>> src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out) >>> ;CHKERRQ(ierr); >>>> src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm) >>> ;CHKERRQ(ierr); >>>> src/ts/impls/implicit/sundials/sundials.c: ierr = >>> MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_ >>> sundials));CHKERRQ(ierr); >>>> >>>> Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid >>> these MPI_Comm_dup() calls? >>>> >>>> Satish >>>> >>>> On Tue, 3 Apr 2018, Smith, Barry F. wrote: >>>> >>>>> >>>>> Are we sure this is a PETSc comm issue and not a hypre comm >>> duplication issue >>>>> >>>>> frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_ >>> GenerateSubComm(comm=-1006627852, participate=, >>> new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] >>>>> >>>>> Looks like hypre is needed to generate subcomms, perhaps it generates >>> too many? >>>>> >>>>> Barry >>>>> >>>>> >>>>>> On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: >>>>>> >>>>>> I?m working with Fande on this and I would like to add a bit more. >>> There are many circumstances where we aren?t working on COMM_WORLD at all >>> (e.g. working on a sub-communicator) but PETSc was initialized using >>> MPI_COMM_WORLD (think multi-level solves)? and we need to create >>> arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We >>> definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering >>> duplication. >>>>>> >>>>>> Can you explain why PETSc needs to duplicate the communicator so much? >>>>>> >>>>>> Thanks for your help in tracking this down! >>>>>> >>>>>> Derek >>>>>> >>>>>> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: >>>>>> Why we do not use user-level MPI communicators directly? What are >>> potential risks here? >>>>>> >>>>>> >>>>>> Fande, >>>>>> >>>>>> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay >>> wrote: >>>>>> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls >>> to MPI_Comm_dup() - thus potentially avoiding such errors >>>>>> >>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs. >>> anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_ >>> PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ >>> _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi >>> CY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_ >>> zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= >>>>>> >>>>>> >>>>>> Satish >>>>>> >>>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: >>>>>> >>>>>>> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay >>> wrote: >>>>>>> >>>>>>>> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? >>>>>>>> >>>>>>>> If so - you could try changing to PETSC_COMM_WORLD >>>>>>>> >>>>>>> >>>>>>> >>>>>>> I do not think we are using PETSC_COMM_WORLD when creating PETSc >>> objects. >>>>>>> Why we can not use MPI_COMM_WORLD? >>>>>>> >>>>>>> >>>>>>> Fande, >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Satish >>>>>>>> >>>>>>>> >>>>>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its >>>>>>>>> applications. I have a error message for a standard test: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *preconditioners/pbp.lots_of_variables: MPI had an >>>>>>>>> errorpreconditioners/pbp.lots_of_variables: >>>>>>>>> ------------------------------------------------ >>>>>>>> preconditioners/pbp.lots_of_variables: >>>>>>>>> Other MPI error, error stack:preconditioners/pbp.lots_of_variables: >>>>>>>>> PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, >>>>>>>>> new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: >>>>>>>>> PMPI_Comm_dup(162)..................: >>>>>>>>> preconditioners/pbp.lots_of_variables: >>>>>>>>> MPIR_Comm_dup_impl(57)..............: >>>>>>>>> preconditioners/pbp.lots_of_variables: >>>>>>>>> MPIR_Comm_copy(739).................: >>>>>>>>> preconditioners/pbp.lots_of_variables: >>>>>>>>> MPIR_Get_contextid_sparse_group(614): Too many communicators >>> (0/2048 >>>>>>>> free >>>>>>>>> on this process; ignore_id=0)* >>>>>>>>> >>>>>>>>> >>>>>>>>> I did "git bisect', and the following commit introduces this issue: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano >>> Zampini >>>>>>>>> >Date: Sat >>> Nov 5 >>>>>>>>> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE >>>>>>>>> hpmat already stores two HYPRE vectors* >>>>>>>>> >>>>>>>>> Before I debug line-by-line, anyone has a clue on this? >>>>>>>>> >>>>>>>>> >>>>>>>>> Fande, >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From yann.jobic at univ-amu.fr Tue Apr 3 10:29:39 2018 From: yann.jobic at univ-amu.fr (Yann Jobic) Date: Tue, 3 Apr 2018 17:29:39 +0200 Subject: [petsc-users] DMForestTransferVec with -petscspace_order 0 In-Reply-To: <20180403013341.6kewwzmm7evmoytf@gatech.edu> References: <91696bcc-a34a-1142-8322-817bc497263b@univ-amu.fr> <20180403013341.6kewwzmm7evmoytf@gatech.edu> Message-ID: Hi, Thanks for the fast answer ! And sorry for my late one... As a test, i'm using ex2.c in test/forest directory. I'm using 2 git depot (master), one from this week end, and one of today. I see some different behaviors. For the git from the week end : v3.8.4-1-g756c7f9 In 2D, everything is ok (petscspace_order = 0,1,2) : mpiexec -n 1 ./ex2 -petscspace_poly_tensor -petscspace_order 2 -dim 2 In 3D, only -petscspace_order 2 works. For the other ones, i put the full message in a log file. For the git from today : v3.8.4-2420-g8f4cb0b In 2D, petscspace_order is ok for 2 and 1, but do not work for 0. In 3D, same as 2D, it's working for petscspace_order 2 and 1, but not for 0. (see log file) Many thanks for the help ! Regards, Yann Le 03/04/2018 ? 03:33, Tobin Isaac a ?crit?: > Hi Yann, > > Thanks for pointing this out to us. Matt and I are the two most > actively developing in this area. We have been working on separate > threads and this looks like an issue where we need to sync up. I > think there is a simple fix, but it would be helpful to know which > version petsc you're working from: I'm seeing different > behavior. Could you please send along more complete output? > > Cheers, > Toby > > On Mon, Apr 02, 2018 at 04:42:29PM +0200, yann JOBIC wrote: >> Hi, >> >> I'm using DMForestTransferVec, as in "dm/impls/forest/examples/tests/ex2.c". >> >> I would like to use it with a space approximation order at zero >> (-petscspace_order 0). However, in this case, it's not working (valgrind >> output of ex2.c from forest test) : >> >> ==8604== Conditional jump or move depends on uninitialised value(s) >> ==8604==??? at 0x47D74F: DMPlexVecGetClosure (plex.c:4035) >> ==8604==??? by 0x557612: DMPlexLocatePoint_General_3D_Internal >> (plexgeometry.c:153) >> ==8604==??? by 0x559B85: DMPlexLocatePoint_Internal (plexgeometry.c:383) >> ==8604==??? by 0x611EED: DMPlexComputeInjectorReferenceTree >> (plextree.c:3247) >> ==8604==??? by 0x6148DB: DMPlexReferenceTreeGetInjector (plextree.c:3454) >> ==8604==??? by 0x61EAD8: DMPlexTransferVecTree_Inject (plextree.c:4319) >> ==8604==??? by 0x620CC1: DMPlexTransferVecTree (plextree.c:4527) >> ==8604==??? by 0x7F23D8: DMForestTransferVec_p8est (pforest.c:4239) >> ==8604==??? by 0x429B48: DMForestTransferVec (forest.c:985) >> ==8604==??? by 0x40BB8A: main (transf_test.c:136) >> >> With the error message : >> >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> >> It's working fine in 2D (p4est). The problem arise in 3D (p8est). >> >> Is it an expected behavior ? Am i doing something wrong ? >> >> Thanks in advance, >> >> Yann >> -- ___________________________ Yann JOBIC HPC engineer IUSTI-CNRS UMR 7343 - Polytech Marseille Technop?le de Ch?teau Gombert 5 rue Enrico Fermi 13453 Marseille cedex 13 Tel : (33) 4 91 10 69 43 Fax : (33) 4 91 10 69 69 --- L'absence de virus dans ce courrier ?lectronique a ?t? v?rifi?e par le logiciel antivirus Avast. https://www.avast.com/antivirus -------------- next part -------------- yann at crabe:~/projet/AMR/moulinette$ mpiexec -n 1 ./ex2 -petscspace_poly_tensor -petscspace_order 0 -dim 3 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] DMPlexVecGetClosure line 4020 /home/yann/src/petsc-git/src/dm/impls/plex/plex.c [0]PETSC ERROR: [0] DMPlexLocatePoint_General_3D_Internal line 150 /home/yann/src/petsc-git/src/dm/impls/plex/plexgeometry.c [0]PETSC ERROR: [0] DMPlexLocatePoint_Internal line 361 /home/yann/src/petsc-git/src/dm/impls/plex/plexgeometry.c [0]PETSC ERROR: [0] DMPlexComputeInjectorReferenceTree line 3008 /home/yann/src/petsc-git/src/dm/impls/plex/plextree.c [0]PETSC ERROR: [0] DMPlexReferenceTreeGetInjector line 3449 /home/yann/src/petsc-git/src/dm/impls/plex/plextree.c [0]PETSC ERROR: [0] DMPlexTransferVecTree_Inject line 4309 /home/yann/src/petsc-git/src/dm/impls/plex/plextree.c [0]PETSC ERROR: [0] DMPlexTransferVecTree line 4486 /home/yann/src/petsc-git/src/dm/impls/plex/plextree.c [0]PETSC ERROR: [0] DMForestTransferVec_p8est line 4213 /home/yann/src/petsc-git/src/dm/impls/forest/p4est/pforest.c [0]PETSC ERROR: [0] DMForestTransferVec line 978 /home/yann/src/petsc-git/src/dm/impls/forest/forest.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.8.4, unknown [0]PETSC ERROR: ./ex2 on a depot-git named crabe by yann Tue Apr 3 17:18:02 2018 [0]PETSC ERROR: Configure options --prefix=/local/lib/petsc/depot-git/gcc/openmpi_gcc_all --with-single-library=0 --with-debugging=1 --download-scalapack=1 --download-metis=1 --download-parmetis=1 --download-ptscotch=1 --download-mumps=1 --download-hypre=1 --download-superlu=1 --download-superlu_dist=1 --download-fblaslapack=1 --download-metis=1 --download-cmake=1 --download-ml=1 --download-p4est=1 --download-netcdf=1 --download-pragmatic=1 --download-eigen=1 --download-parms=1 --download-triangle=1 --download-hdf5=1 --with-zlib=1 --download-szlib=1 --download-chaco=1 --download-spai=1 --download-suitesparse=1 --with-shared-libraries=0 PETSC_ARCH=depot-git [0]PETSC ERROR: #1 User provided function() line 0 in unknown file -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- ------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code.. Per user-direction, the job has been aborted. ------------------------------------------------------- -------------------------------------------------------------------------- mpiexec detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[36533,1],0] Exit code: 59 -------------------------------------------------------------------------- -------------- next part -------------- jobic at stargate:~/projet/AMR/moulinette$ mpirun -np 1 ./ex2 -petscspace_poly_tensor -dim 3 -petscspace_order 0 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 2 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2420-g8f4cb0b GIT Date: 2018-04-02 16:00:52 -0500 [0]PETSC ERROR: ./ex2 on a named stargate by jobic Tue Apr 3 17:22:04 2018 [0]PETSC ERROR: Configure options --download-chaco=1 --download-cmake=1 --download-eigen=1 --download-fblaslapack=1 --download-hdf5=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mumps=1 --download-netcdf=1 --download-p4est=1 --download-parmetis=1 --download-parms=1 --download-pragmatic=1 --download-ptscotch=1 --download-scalapack=1 --download-spai=1 --download-suitesparse=1 --download-superlu=1 --download-superlu_dist=1 --download-szlib=1 --download-triangle=1 --prefix=/local/lib/petsc/git/gcc/openmpi_gcc_all --with-debugging=1 --with-shared-libraries=0 --with-single-library=0 --with-zlib=1 PETSC_ARCH=openmpi_gcc_all [0]PETSC ERROR: #1 PetscFESetDualSpace() line 3615 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtfe.c [0]PETSC ERROR: #2 PetscFEGetHeightSubspace() line 6630 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtfe.c [0]PETSC ERROR: #3 PetscDSGetHeightSubspace() line 2880 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtds.c [0]PETSC ERROR: #4 DMProjectLocal_Generic_Plex() line 325 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/plex/plexproject.c [0]PETSC ERROR: #5 DMProjectFunctionLocal_Plex() line 428 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/plex/plexproject.c [0]PETSC ERROR: #6 DMProjectFunctionLocal() line 6265 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #7 DMProjectFunctionLocal_p8est() line 4385 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/forest/p4est/pforest.c [0]PETSC ERROR: #8 DMProjectFunctionLocal() line 6265 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #9 DMProjectFunction() line 6250 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #10 main() line 161 in /home/jobic/projet/AMR/moulinette/ex2.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -dim 3 [0]PETSC ERROR: -petscspace_order 0 [0]PETSC ERROR: -petscspace_poly_tensor [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 85. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- With valgrind : jobic at stargate:~/projet/AMR/moulinette$ valgrind mpirun -np 1 ./ex2 -petscspace_poly_tensor -dim 3 -petscspace_order 0 ==9730== Memcheck, a memory error detector ==9730== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==9730== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info ==9730== Command: mpirun -np 1 ./ex2 -petscspace_poly_tensor -dim 3 -petscspace_order 0 ==9730== ==10212== Warning: invalid file descriptor 1024 in syscall close() ==10212== Warning: invalid file descriptor 1025 in syscall close() ==10212== Warning: invalid file descriptor 1026 in syscall close() ==10212== Warning: invalid file descriptor 1027 in syscall close() ==10212== Use --log-fd= to select an alternative log fd. ==10212== Warning: invalid file descriptor 1028 in syscall close() ==10212== Warning: invalid file descriptor 1029 in syscall close() [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 2 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2420-g8f4cb0b GIT Date: 2018-04-02 16:00:52 -0500 [0]PETSC ERROR: ./ex2 on a named stargate by jobic Tue Apr 3 17:27:58 2018 [0]PETSC ERROR: Configure options --download-chaco=1 --download-cmake=1 --download-eigen=1 --download-fblaslapack=1 --download-hdf5=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mumps=1 --download-netcdf=1 --download-p4est=1 --download-parmetis=1 --download-parms=1 --download-pragmatic=1 --download-ptscotch=1 --download-scalapack=1 --download-spai=1 --download-suitesparse=1 --download-superlu=1 --download-superlu_dist=1 --download-szlib=1 --download-triangle=1 --prefix=/local/lib/petsc/git/gcc/openmpi_gcc_all --with-debugging=1 --with-shared-libraries=0 --with-single-library=0 --with-zlib=1 PETSC_ARCH=openmpi_gcc_all [0]PETSC ERROR: #1 PetscFESetDualSpace() line 3615 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtfe.c [0]PETSC ERROR: #2 PetscFEGetHeightSubspace() line 6630 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtfe.c [0]PETSC ERROR: #3 PetscDSGetHeightSubspace() line 2880 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtds.c [0]PETSC ERROR: #4 DMProjectLocal_Generic_Plex() line 325 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/plex/plexproject.c [0]PETSC ERROR: #5 DMProjectFunctionLocal_Plex() line 428 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/plex/plexproject.c [0]PETSC ERROR: #6 DMProjectFunctionLocal() line 6265 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #7 DMProjectFunctionLocal_p8est() line 4385 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/forest/p4est/pforest.c [0]PETSC ERROR: #8 DMProjectFunctionLocal() line 6265 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #9 DMProjectFunction() line 6250 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #10 main() line 161 in /home/jobic/projet/AMR/moulinette/ex2.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -dim 3 [0]PETSC ERROR: -petscspace_order 0 [0]PETSC ERROR: -petscspace_poly_tensor [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 85. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- ==9730== ==9730== HEAP SUMMARY: ==9730== in use at exit: 55,755 bytes in 116 blocks ==9730== total heap usage: 19,779 allocs, 19,663 frees, 6,633,193 bytes allocated ==9730== ==9730== LEAK SUMMARY: ==9730== definitely lost: 2,675 bytes in 11 blocks ==9730== indirectly lost: 48,765 bytes in 75 blocks ==9730== possibly lost: 200 bytes in 1 blocks ==9730== still reachable: 4,115 bytes in 29 blocks ==9730== suppressed: 0 bytes in 0 blocks ==9730== Rerun with --leak-check=full to see details of leaked memory ==9730== ==9730== For counts of detected and suppressed errors, rerun with: -v ==9730== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) From balay at mcs.anl.gov Tue Apr 3 10:32:50 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 3 Apr 2018 10:32:50 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: On Tue, 3 Apr 2018, Stefano Zampini wrote: > > > On Apr 3, 2018, at 4:58 PM, Satish Balay wrote: > > > > On Tue, 3 Apr 2018, Kong, Fande wrote: > > > >> On Tue, Apr 3, 2018 at 1:17 AM, Smith, Barry F. wrote: > >> > >>> > >>> Each external package definitely needs its own duplicated communicator; > >>> cannot share between packages. > >>> > >>> The only problem with the dups below is if they are in a loop and get > >>> called many times. > >>> > >> > >> > >> The "standard test" that has this issue actually has 1K fields. MOOSE > >> creates its own field-split preconditioner (not based on the PETSc > >> fieldsplit), and each filed is associated with one PC HYPRE. If PETSc > >> duplicates communicators, we should easily reach the limit 2048. > >> > >> I also want to confirm what extra communicators are introduced in the bad > >> commit. > > > > To me it looks like there is 1 extra comm created [for MATHYPRE] for each PCHYPRE that is created [which also creates one comm for this object]. > > > > You?re right; however, it was the same before the commit. https://bitbucket.org/petsc/petsc/commits/49a781f5cee36db85e8d5b951eec29f10ac13593 Before the commit - PCHYPRE was not calling MatConvert(MATHYPRE) [this results in an additional call to MPI_Comm_dup() for hypre calls] PCHYPRE was calling MatHYPRE_IJMatrixCreate() directly [which I presume reusing the comm created by the call to MPI_Comm_dup() in PCHYPRE - for hypre calls] > I don?t understand how this specific commit is related with this issue, being the error not in the MPI_Comm_Dup which is inside MatCreate_MATHYPRE. Actually, the error comes from MPI_Comm_create > > frame #5: 0x00000001068defd4 libmpi.12.dylib`MPI_Comm_create + 3492 > frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, participate=, new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] > frame #7: 0x000000010618f8ba libpetsc.3.07.dylib`hypre_GaussElimSetup(amg_data=0x00007fe7ff857a00, level=, relax_type=9) + 74 at par_relax.c:4209 [opt] > frame #8: 0x0000000106140e93 libpetsc.3.07.dylib`hypre_BoomerAMGSetup(amg_vdata=, A=0x00007fe80842aff0, f=0x00007fe80842a980, u=0x00007fe80842a510) + 17699 at par_amg_setup.c:2108 [opt] > frame #9: 0x0000000105ec773c libpetsc.3.07.dylib`PCSetUp_HYPRE(pc=) + 2540 at hypre.c:226 [opt I thought this trace comes up after applying your patch - ierr = MatDestroy(&jac->hpmat);CHKERRQ(ierr); - ierr = MatConvert(pc->pmat,MATHYPRE,MAT_INITIAL_MATRIX,&jac->hpmat);CHKERRQ(ierr); + ierr = MatConvert(pc->pmat,MATHYPRE,jac->hpmat ? MAT_REUSE_MATRIX : MAT_INITIAL_MATRIX,&jac->hpmat);CHKERRQ(ierr); The stack before this patch was: [its a different format - so it was obtained in a different way than the above method?] preconditioners/pbp.lots_of_variables: Other MPI error, error stack: preconditioners/pbp.lots_of_variables: PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, new_comm=0x97d1068) failed preconditioners/pbp.lots_of_variables: PMPI_Comm_dup(162)..................: preconditioners/pbp.lots_of_variables: MPIR_Comm_dup_impl(57)..............: preconditioners/pbp.lots_of_variables: MPIR_Comm_copy(739).................: preconditioners/pbp.lots_of_variables: MPIR_Get_contextid_sparse_group(614): Too many communicators (0/2048 free on this process; ignore_id=0) Satish > > How did you perform the bisection? make clean + make all ? Which version of HYPRE are you using? > > > But you might want to verify [by linking with mpi trace library?] > > > > > > There are some debugging hints at https://lists.mpich.org/pipermail/discuss/2012-December/000148.html [wrt mpich] - which I haven't checked.. > > > > Satish > > > >> > >> > >> Fande, > >> > >> > >> > >>> > >>> To debug the hypre/duplication issue in MOOSE I would run in the > >>> debugger with a break point in MPI_Comm_dup() and see > >>> who keeps calling it an unreasonable amount of times. (My guess is this is > >>> a new "feature" in hypre that they will need to fix but only debugging will > >>> tell) > >>> > >>> Barry > >>> > >>> > >>>> On Apr 2, 2018, at 7:44 PM, Balay, Satish wrote: > >>>> > >>>> We do a MPI_Comm_dup() for objects related to externalpackages. > >>>> > >>>> Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is > >>>> using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think > >>>> is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 > >>>> > >>>> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( > >>> PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > >>>> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( > >>> PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > >>>> src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm); > >>> CHKERRQ(ierr); > >>>> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > >>> PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); > >>>> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > >>> PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > >>>> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > >>> PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > >>>> src/ksp/pc/impls/spai/ispai.c: ierr = > >>> MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_ > >>> spai));CHKERRQ(ierr); > >>>> src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, > >>> &comm);CHKERRQ(ierr); > >>>> src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = > >>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_ > >>> cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); > >>>> src/mat/impls/aij/mpi/mumps/mumps.c: ierr = > >>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_ > >>> mumps));CHKERRQ(ierr); > >>>> src/mat/impls/aij/mpi/pastix/pastix.c: ierr = > >>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_ > >>> comm));CHKERRQ(ierr); > >>>> src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = > >>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_ > >>> superlu));CHKERRQ(ierr); > >>>> src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > >>> PetscObject)B),&hB->comm);CHKERRQ(ierr); > >>>> src/mat/partition/impls/pmetis/pmetis.c: ierr = > >>> MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); > >>>> src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a > >>> MPI_Comm_dup() of each of these (duplicates of duplicates return the same > >>> communictor) > >>>> src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) > >>>> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& > >>> local_comm);CHKERRQ(ierr); > >>>> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& > >>> local_comm);CHKERRQ(ierr); > >>>> src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out) > >>> ;CHKERRQ(ierr); > >>>> src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm) > >>> ;CHKERRQ(ierr); > >>>> src/ts/impls/implicit/sundials/sundials.c: ierr = > >>> MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_ > >>> sundials));CHKERRQ(ierr); > >>>> > >>>> Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid > >>> these MPI_Comm_dup() calls? > >>>> > >>>> Satish > >>>> > >>>> On Tue, 3 Apr 2018, Smith, Barry F. wrote: > >>>> > >>>>> > >>>>> Are we sure this is a PETSc comm issue and not a hypre comm > >>> duplication issue > >>>>> > >>>>> frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_ > >>> GenerateSubComm(comm=-1006627852, participate=, > >>> new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] > >>>>> > >>>>> Looks like hypre is needed to generate subcomms, perhaps it generates > >>> too many? > >>>>> > >>>>> Barry > >>>>> > >>>>> > >>>>>> On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: > >>>>>> > >>>>>> I?m working with Fande on this and I would like to add a bit more. > >>> There are many circumstances where we aren?t working on COMM_WORLD at all > >>> (e.g. working on a sub-communicator) but PETSc was initialized using > >>> MPI_COMM_WORLD (think multi-level solves)? and we need to create > >>> arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We > >>> definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering > >>> duplication. > >>>>>> > >>>>>> Can you explain why PETSc needs to duplicate the communicator so much? > >>>>>> > >>>>>> Thanks for your help in tracking this down! > >>>>>> > >>>>>> Derek > >>>>>> > >>>>>> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: > >>>>>> Why we do not use user-level MPI communicators directly? What are > >>> potential risks here? > >>>>>> > >>>>>> > >>>>>> Fande, > >>>>>> > >>>>>> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay > >>> wrote: > >>>>>> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls > >>> to MPI_Comm_dup() - thus potentially avoiding such errors > >>>>>> > >>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs. > >>> anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_ > >>> PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > >>> _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi > >>> CY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_ > >>> zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > >>>>>> > >>>>>> > >>>>>> Satish > >>>>>> > >>>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: > >>>>>> > >>>>>>> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay > >>> wrote: > >>>>>>> > >>>>>>>> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? > >>>>>>>> > >>>>>>>> If so - you could try changing to PETSC_COMM_WORLD > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I do not think we are using PETSC_COMM_WORLD when creating PETSc > >>> objects. > >>>>>>> Why we can not use MPI_COMM_WORLD? > >>>>>>> > >>>>>>> > >>>>>>> Fande, > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>> Satish > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: > >>>>>>>> > >>>>>>>>> Hi All, > >>>>>>>>> > >>>>>>>>> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > >>>>>>>>> applications. I have a error message for a standard test: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> *preconditioners/pbp.lots_of_variables: MPI had an > >>>>>>>>> errorpreconditioners/pbp.lots_of_variables: > >>>>>>>>> ------------------------------------------------ > >>>>>>>> preconditioners/pbp.lots_of_variables: > >>>>>>>>> Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > >>>>>>>>> PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > >>>>>>>>> new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > >>>>>>>>> PMPI_Comm_dup(162)..................: > >>>>>>>>> preconditioners/pbp.lots_of_variables: > >>>>>>>>> MPIR_Comm_dup_impl(57)..............: > >>>>>>>>> preconditioners/pbp.lots_of_variables: > >>>>>>>>> MPIR_Comm_copy(739).................: > >>>>>>>>> preconditioners/pbp.lots_of_variables: > >>>>>>>>> MPIR_Get_contextid_sparse_group(614): Too many communicators > >>> (0/2048 > >>>>>>>> free > >>>>>>>>> on this process; ignore_id=0)* > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> I did "git bisect', and the following commit introduces this issue: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano > >>> Zampini > >>>>>>>>> >Date: Sat > >>> Nov 5 > >>>>>>>>> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > >>>>>>>>> hpmat already stores two HYPRE vectors* > >>>>>>>>> > >>>>>>>>> Before I debug line-by-line, anyone has a clue on this? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Fande, > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>> > >>> > > From fande.kong at inl.gov Tue Apr 3 10:38:02 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Tue, 3 Apr 2018 09:38:02 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: On Tue, Apr 3, 2018 at 9:32 AM, Satish Balay wrote: > On Tue, 3 Apr 2018, Stefano Zampini wrote: > > > > > > On Apr 3, 2018, at 4:58 PM, Satish Balay wrote: > > > > > > On Tue, 3 Apr 2018, Kong, Fande wrote: > > > > > >> On Tue, Apr 3, 2018 at 1:17 AM, Smith, Barry F. > wrote: > > >> > > >>> > > >>> Each external package definitely needs its own duplicated > communicator; > > >>> cannot share between packages. > > >>> > > >>> The only problem with the dups below is if they are in a loop and > get > > >>> called many times. > > >>> > > >> > > >> > > >> The "standard test" that has this issue actually has 1K fields. MOOSE > > >> creates its own field-split preconditioner (not based on the PETSc > > >> fieldsplit), and each filed is associated with one PC HYPRE. If PETSc > > >> duplicates communicators, we should easily reach the limit 2048. > > >> > > >> I also want to confirm what extra communicators are introduced in the > bad > > >> commit. > > > > > > To me it looks like there is 1 extra comm created [for MATHYPRE] for > each PCHYPRE that is created [which also creates one comm for this object]. > > > > > > > You?re right; however, it was the same before the commit. > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ > bitbucket.org_petsc_petsc_commits_49a781f5cee36db85e8d5b951eec29 > f10ac13593&d=DwIDaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi > CY&m=6_ukwovpDrK5BL_94S4ezasw2a3S15SM59R41rSY-Yw&s= > r8xHYLKF9LtJHReR6Jmfeei3OfwkQNiGrKXAgeqPVQ8&e= > Before the commit - PCHYPRE was not calling MatConvert(MATHYPRE) [this > results in an additional call to MPI_Comm_dup() for hypre calls] PCHYPRE > was calling MatHYPRE_IJMatrixCreate() directly [which I presume reusing the > comm created by the call to MPI_Comm_dup() in PCHYPRE - for hypre calls] > > > > > I don?t understand how this specific commit is related with this issue, > being the error not in the MPI_Comm_Dup which is inside MatCreate_MATHYPRE. > Actually, the error comes from MPI_Comm_create > > > > frame #5: 0x00000001068defd4 libmpi.12.dylib`MPI_Comm_create + 3492 > > frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_ > GenerateSubComm(comm=-1006627852, participate=, > new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] > > frame #7: 0x000000010618f8ba libpetsc.3.07.dylib`hypre_ > GaussElimSetup(amg_data=0x00007fe7ff857a00, level=, > relax_type=9) + 74 at par_relax.c:4209 [opt] > > frame #8: 0x0000000106140e93 libpetsc.3.07.dylib`hypre_ > BoomerAMGSetup(amg_vdata=, A=0x00007fe80842aff0, > f=0x00007fe80842a980, u=0x00007fe80842a510) + 17699 at par_amg_setup.c:2108 > [opt] > > frame #9: 0x0000000105ec773c libpetsc.3.07.dylib`PCSetUp_HYPRE(pc=) > + 2540 at hypre.c:226 [opt > > I thought this trace comes up after applying your patch > This trace comes from Mac > > - ierr = MatDestroy(&jac->hpmat);CHKERRQ(ierr); > - ierr = MatConvert(pc->pmat,MATHYPRE,MAT_INITIAL_MATRIX,&jac-> > hpmat);CHKERRQ(ierr); > + ierr = MatConvert(pc->pmat,MATHYPRE,jac->hpmat ? MAT_REUSE_MATRIX : > MAT_INITIAL_MATRIX,&jac->hpmat);CHKERRQ(ierr); > > The stack before this patch was: [its a different format - so it was > obtained in a different way than the above method?] > > preconditioners/pbp.lots_of_variables: Other MPI error, error stack: > preconditioners/pbp.lots_of_variables: PMPI_Comm_dup(177)..................: > MPI_Comm_dup(comm=0x84000001, new_comm=0x97d1068) failed > preconditioners/pbp.lots_of_variables: PMPI_Comm_dup(162)............ > ......: > preconditioners/pbp.lots_of_variables: MPIR_Comm_dup_impl(57)........ > ......: > preconditioners/pbp.lots_of_variables: MPIR_Comm_copy(739)........... > ......: > preconditioners/pbp.lots_of_variables: MPIR_Get_contextid_sparse_group(614): > Too many communicators (0/2048 free on this process; ignore_id=0) > This comes from a Linux (it is a test box), and I do not have access to it. Fande, > > Satish > > > > > How did you perform the bisection? make clean + make all ? Which version > of HYPRE are you using? > > > > > But you might want to verify [by linking with mpi trace library?] > > > > > > > > > There are some debugging hints at https://urldefense.proofpoint. > com/v2/url?u=https-3A__lists.mpich.org_pipermail_discuss_ > 2012-2DDecember_000148.html&d=DwIDaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi > CY&m=6_ukwovpDrK5BL_94S4ezasw2a3S15SM59R41rSY-Yw& > s=XUy9n2kmdq262Gwrn_RMXYR-bIyiKViCvp4fRfGCP9w&e= [wrt mpich] - which I > haven't checked.. > > > > > > Satish > > > > > >> > > >> > > >> Fande, > > >> > > >> > > >> > > >>> > > >>> To debug the hypre/duplication issue in MOOSE I would run in the > > >>> debugger with a break point in MPI_Comm_dup() and see > > >>> who keeps calling it an unreasonable amount of times. (My guess is > this is > > >>> a new "feature" in hypre that they will need to fix but only > debugging will > > >>> tell) > > >>> > > >>> Barry > > >>> > > >>> > > >>>> On Apr 2, 2018, at 7:44 PM, Balay, Satish > wrote: > > >>>> > > >>>> We do a MPI_Comm_dup() for objects related to externalpackages. > > >>>> > > >>>> Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is > > >>>> using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think > > >>>> is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs > 3.7 > > >>>> > > >>>> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > >>> PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > >>>> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > >>> PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > >>>> src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm); > > >>> CHKERRQ(ierr); > > >>>> src/ksp/pc/impls/hypre/hypre.c: ierr = > MPI_Comm_dup(PetscObjectComm(( > > >>> PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); > > >>>> src/ksp/pc/impls/hypre/hypre.c: ierr = > MPI_Comm_dup(PetscObjectComm(( > > >>> PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > >>>> src/ksp/pc/impls/hypre/hypre.c: ierr = > MPI_Comm_dup(PetscObjectComm(( > > >>> PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > >>>> src/ksp/pc/impls/spai/ispai.c: ierr = > > >>> MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_ > > >>> spai));CHKERRQ(ierr); > > >>>> src/mat/examples/tests/ex152.c: ierr = > MPI_Comm_dup(MPI_COMM_WORLD, > > >>> &comm);CHKERRQ(ierr); > > >>>> src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = > > >>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_ > > >>> cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); > > >>>> src/mat/impls/aij/mpi/mumps/mumps.c: ierr = > > >>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_ > > >>> mumps));CHKERRQ(ierr); > > >>>> src/mat/impls/aij/mpi/pastix/pastix.c: ierr = > > >>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_ > > >>> comm));CHKERRQ(ierr); > > >>>> src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = > > >>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_ > > >>> superlu));CHKERRQ(ierr); > > >>>> src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > >>> PetscObject)B),&hB->comm);CHKERRQ(ierr); > > >>>> src/mat/partition/impls/pmetis/pmetis.c: ierr = > > >>> MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); > > >>>> src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a > > >>> MPI_Comm_dup() of each of these (duplicates of duplicates return the > same > > >>> communictor) > > >>>> src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) > > >>>> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& > > >>> local_comm);CHKERRQ(ierr); > > >>>> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& > > >>> local_comm);CHKERRQ(ierr); > > >>>> src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out) > > >>> ;CHKERRQ(ierr); > > >>>> src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm) > > >>> ;CHKERRQ(ierr); > > >>>> src/ts/impls/implicit/sundials/sundials.c: ierr = > > >>> MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_ > > >>> sundials));CHKERRQ(ierr); > > >>>> > > >>>> Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid > > >>> these MPI_Comm_dup() calls? > > >>>> > > >>>> Satish > > >>>> > > >>>> On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > >>>> > > >>>>> > > >>>>> Are we sure this is a PETSc comm issue and not a hypre comm > > >>> duplication issue > > >>>>> > > >>>>> frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_ > > >>> GenerateSubComm(comm=-1006627852, participate=, > > >>> new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] > > >>>>> > > >>>>> Looks like hypre is needed to generate subcomms, perhaps it > generates > > >>> too many? > > >>>>> > > >>>>> Barry > > >>>>> > > >>>>> > > >>>>>> On Apr 2, 2018, at 7:07 PM, Derek Gaston > wrote: > > >>>>>> > > >>>>>> I?m working with Fande on this and I would like to add a bit more. > > >>> There are many circumstances where we aren?t working on COMM_WORLD > at all > > >>> (e.g. working on a sub-communicator) but PETSc was initialized using > > >>> MPI_COMM_WORLD (think multi-level solves)? and we need to create > > >>> arbitrarily many PETSc vecs/mats/solvers/preconditioners and > solve. We > > >>> definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering > > >>> duplication. > > >>>>>> > > >>>>>> Can you explain why PETSc needs to duplicate the communicator so > much? > > >>>>>> > > >>>>>> Thanks for your help in tracking this down! > > >>>>>> > > >>>>>> Derek > > >>>>>> > > >>>>>> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande > wrote: > > >>>>>> Why we do not use user-level MPI communicators directly? What are > > >>> potential risks here? > > >>>>>> > > >>>>>> > > >>>>>> Fande, > > >>>>>> > > >>>>>> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay > > >>> wrote: > > >>>>>> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize > calls > > >>> to MPI_Comm_dup() - thus potentially avoiding such errors > > >>>>>> > > >>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs. > > >>> anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_ > > >>> PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > > >>> _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi > > >>> CY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_ > > >>> zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > > >>>>>> > > >>>>>> > > >>>>>> Satish > > >>>>>> > > >>>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: > > >>>>>> > > >>>>>>> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay > > >>> wrote: > > >>>>>>> > > >>>>>>>> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc > objects? > > >>>>>>>> > > >>>>>>>> If so - you could try changing to PETSC_COMM_WORLD > > >>>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> I do not think we are using PETSC_COMM_WORLD when creating PETSc > > >>> objects. > > >>>>>>> Why we can not use MPI_COMM_WORLD? > > >>>>>>> > > >>>>>>> > > >>>>>>> Fande, > > >>>>>>> > > >>>>>>> > > >>>>>>>> > > >>>>>>>> Satish > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: > > >>>>>>>> > > >>>>>>>>> Hi All, > > >>>>>>>>> > > >>>>>>>>> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and > its > > >>>>>>>>> applications. I have a error message for a standard test: > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> *preconditioners/pbp.lots_of_variables: MPI had an > > >>>>>>>>> errorpreconditioners/pbp.lots_of_variables: > > >>>>>>>>> ------------------------------------------------ > > >>>>>>>> preconditioners/pbp.lots_of_variables: > > >>>>>>>>> Other MPI error, error stack:preconditioners/pbp. > lots_of_variables: > > >>>>>>>>> PMPI_Comm_dup(177)..................: > MPI_Comm_dup(comm=0x84000001, > > >>>>>>>>> new_comm=0x97d1068) failedpreconditioners/pbp. > lots_of_variables: > > >>>>>>>>> PMPI_Comm_dup(162)..................: > > >>>>>>>>> preconditioners/pbp.lots_of_variables: > > >>>>>>>>> MPIR_Comm_dup_impl(57)..............: > > >>>>>>>>> preconditioners/pbp.lots_of_variables: > > >>>>>>>>> MPIR_Comm_copy(739).................: > > >>>>>>>>> preconditioners/pbp.lots_of_variables: > > >>>>>>>>> MPIR_Get_contextid_sparse_group(614): Too many communicators > > >>> (0/2048 > > >>>>>>>> free > > >>>>>>>>> on this process; ignore_id=0)* > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> I did "git bisect', and the following commit introduces this > issue: > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: > Stefano > > >>> Zampini > > >>>>>>>>> >Date: > Sat > > >>> Nov 5 > > >>>>>>>>> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type > MatHYPRE > > >>>>>>>>> hpmat already stores two HYPRE vectors* > > >>>>>>>>> > > >>>>>>>>> Before I debug line-by-line, anyone has a clue on this? > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Fande, > > >>>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>> > > >>> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Apr 3 10:41:34 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 3 Apr 2018 10:41:34 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: On Tue, 3 Apr 2018, Kong, Fande wrote: > > I thought this trace comes up after applying your patch > > > > This trace comes from Mac > > Too many communicators (0/2048 free on this process; ignore_id=0) > This comes from a Linux (it is a test box), and I do not have access to it. Then its hard to compare and infer what caused either of these traces. Changing only 1 variable at a time is best to narrow down > > > How did you perform the bisection? make clean + make all ? Which version of HYPRE are you using? Also its good to know this. Satish From fdkong.jd at gmail.com Tue Apr 3 10:43:38 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Tue, 3 Apr 2018 09:43:38 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: On Tue, Apr 3, 2018 at 9:12 AM, Stefano Zampini wrote: > > On Apr 3, 2018, at 4:58 PM, Satish Balay wrote: > > On Tue, 3 Apr 2018, Kong, Fande wrote: > > On Tue, Apr 3, 2018 at 1:17 AM, Smith, Barry F. > wrote: > > > Each external package definitely needs its own duplicated communicator; > cannot share between packages. > > The only problem with the dups below is if they are in a loop and get > called many times. > > > > The "standard test" that has this issue actually has 1K fields. MOOSE > creates its own field-split preconditioner (not based on the PETSc > fieldsplit), and each filed is associated with one PC HYPRE. If PETSc > duplicates communicators, we should easily reach the limit 2048. > > I also want to confirm what extra communicators are introduced in the bad > commit. > > > To me it looks like there is 1 extra comm created [for MATHYPRE] for each > PCHYPRE that is created [which also creates one comm for this object]. > > > You?re right; however, it was the same before the commit. > I don?t understand how this specific commit is related with this issue, > being the error not in the MPI_Comm_Dup which is inside MatCreate_MATHYPRE. > Actually, the error comes from MPI_Comm_create > > > > > > * frame #5: 0x00000001068defd4 libmpi.12.dylib`MPI_Comm_create + > 3492 frame #6: 0x00000001061345d9 > libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, > participate=, new_comm_ptr=) + 409 at > gen_redcs_mat.c:531 [opt] frame #7: 0x000000010618f8ba > libpetsc.3.07.dylib`hypre_GaussElimSetup(amg_data=0x00007fe7ff857a00, > level=, relax_type=9) + 74 at par_relax.c:4209 [opt] frame > #8: 0x0000000106140e93 > libpetsc.3.07.dylib`hypre_BoomerAMGSetup(amg_vdata=, > A=0x00007fe80842aff0, f=0x00007fe80842a980, u=0x00007fe80842a510) + 17699 > at par_amg_setup.c:2108 [opt] frame #9: 0x0000000105ec773c > libpetsc.3.07.dylib`PCSetUp_HYPRE(pc=) + 2540 at hypre.c:226 > [opt* > > How did you perform the bisection? make clean + make all ? Which version > of HYPRE are you using? > I did more aggressively. "rm -rf arch-darwin-c-opt-bisect " "./configure --optionsModule=config.compilerOptions -with-debugging=no --with-shared-libraries=1 --with-mpi=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 PETSC_ARCH=arch-darwin-c-opt-bisect" HYPRE verison: self.gitcommit = 'v2.11.1-55-g2ea0e43' self.download = ['git://https://github.com/LLNL/hypre',' https://github.com/LLNL/hypre/archive/'+self.gitcommit+'.tar.gz'] I do not think this is caused by HYPRE. Fande, > > But you might want to verify [by linking with mpi trace library?] > > > There are some debugging hints at https://lists.mpich.org/ > pipermail/discuss/2012-December/000148.html [wrt mpich] - which I haven't > checked.. > > Satish > > > > Fande, > > > > > To debug the hypre/duplication issue in MOOSE I would run in the > debugger with a break point in MPI_Comm_dup() and see > who keeps calling it an unreasonable amount of times. (My guess is this is > a new "feature" in hypre that they will need to fix but only debugging will > tell) > > Barry > > > On Apr 2, 2018, at 7:44 PM, Balay, Satish wrote: > > We do a MPI_Comm_dup() for objects related to externalpackages. > > Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is > using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think > is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 > > src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm); > > CHKERRQ(ierr); > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > src/ksp/pc/impls/spai/ispai.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_ > spai));CHKERRQ(ierr); > > src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, > > &comm);CHKERRQ(ierr); > > src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_ > cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); > > src/mat/impls/aij/mpi/mumps/mumps.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_ > mumps));CHKERRQ(ierr); > > src/mat/impls/aij/mpi/pastix/pastix.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_ > comm));CHKERRQ(ierr); > > src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_ > superlu));CHKERRQ(ierr); > > src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)B),&hB->comm);CHKERRQ(ierr); > > src/mat/partition/impls/pmetis/pmetis.c: ierr = > > MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); > > src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a > > MPI_Comm_dup() of each of these (duplicates of duplicates return the same > communictor) > > src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) > src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& > > local_comm);CHKERRQ(ierr); > > src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& > > local_comm);CHKERRQ(ierr); > > src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out) > > ;CHKERRQ(ierr); > > src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm) > > ;CHKERRQ(ierr); > > src/ts/impls/implicit/sundials/sundials.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_ > sundials));CHKERRQ(ierr); > > > Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid > > these MPI_Comm_dup() calls? > > > Satish > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > Are we sure this is a PETSc comm issue and not a hypre comm > > duplication issue > > > frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_ > > GenerateSubComm(comm=-1006627852, participate=, > new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] > > > Looks like hypre is needed to generate subcomms, perhaps it generates > > too many? > > > Barry > > > On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: > > I?m working with Fande on this and I would like to add a bit more. > > There are many circumstances where we aren?t working on COMM_WORLD at all > (e.g. working on a sub-communicator) but PETSc was initialized using > MPI_COMM_WORLD (think multi-level solves)? and we need to create > arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We > definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering > duplication. > > > Can you explain why PETSc needs to duplicate the communicator so much? > > Thanks for your help in tracking this down! > > Derek > > On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: > Why we do not use user-level MPI communicators directly? What are > > potential risks here? > > > > Fande, > > On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay > > wrote: > > PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls > > to MPI_Comm_dup() - thus potentially avoiding such errors > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs. > > anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_ > PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi > CY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_ > zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > > > > Satish > > On Mon, 2 Apr 2018, Kong, Fande wrote: > > On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay > > wrote: > > > Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? > > If so - you could try changing to PETSC_COMM_WORLD > > > > I do not think we are using PETSC_COMM_WORLD when creating PETSc > > objects. > > Why we can not use MPI_COMM_WORLD? > > > Fande, > > > > Satish > > > On Mon, 2 Apr 2018, Kong, Fande wrote: > > Hi All, > > I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > applications. I have a error message for a standard test: > > > > > > > > > > *preconditioners/pbp.lots_of_variables: MPI had an > errorpreconditioners/pbp.lots_of_variables: > ------------------------------------------------ > > preconditioners/pbp.lots_of_variables: > > Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > PMPI_Comm_dup(162)..................: > preconditioners/pbp.lots_of_variables: > MPIR_Comm_dup_impl(57)..............: > preconditioners/pbp.lots_of_variables: > MPIR_Comm_copy(739).................: > preconditioners/pbp.lots_of_variables: > MPIR_Get_contextid_sparse_group(614): Too many communicators > > (0/2048 > > free > > on this process; ignore_id=0)* > > > I did "git bisect', and the following commit introduces this issue: > > > > > > > > > *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano > > Zampini > > >Date: Sat > > Nov 5 > > 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > hpmat already stores two HYPRE vectors* > > Before I debug line-by-line, anyone has a clue on this? > > > Fande, > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedmud at gmail.com Tue Apr 3 10:43:54 2018 From: friedmud at gmail.com (Derek Gaston) Date: Tue, 03 Apr 2018 15:43:54 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: One thing I want to be clear of here: is that we're not trying to solve this particular problem (where we're creating 1000 instances of Hypre to precondition each variable independently)... this particular problem is just a test (that we've had in our test suite for a long time) to stress test some of this capability. We really do have needs for thousands (tens of thousands) of simultaneous solves (each with their own Hypre instances). That's not what this particular problem is doing - but it is representative of a class of our problems we need to solve. Which does bring up a point: I have been able to do solves before with ~50,000 separate PETSc solves without issue. Is it because I was working with MVAPICH on a cluster? Does it just have a higher limit? Derek On Tue, Apr 3, 2018 at 9:13 AM Stefano Zampini wrote: > On Apr 3, 2018, at 4:58 PM, Satish Balay wrote: > > On Tue, 3 Apr 2018, Kong, Fande wrote: > > On Tue, Apr 3, 2018 at 1:17 AM, Smith, Barry F. > wrote: > > > Each external package definitely needs its own duplicated communicator; > cannot share between packages. > > The only problem with the dups below is if they are in a loop and get > called many times. > > > > The "standard test" that has this issue actually has 1K fields. MOOSE > creates its own field-split preconditioner (not based on the PETSc > fieldsplit), and each filed is associated with one PC HYPRE. If PETSc > duplicates communicators, we should easily reach the limit 2048. > > I also want to confirm what extra communicators are introduced in the bad > commit. > > > To me it looks like there is 1 extra comm created [for MATHYPRE] for each > PCHYPRE that is created [which also creates one comm for this object]. > > > You?re right; however, it was the same before the commit. > I don?t understand how this specific commit is related with this issue, > being the error not in the MPI_Comm_Dup which is inside MatCreate_MATHYPRE. > Actually, the error comes from MPI_Comm_create > > > > > > * frame #5: 0x00000001068defd4 libmpi.12.dylib`MPI_Comm_create + > 3492 frame #6: 0x00000001061345d9 > libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, > participate=, new_comm_ptr=) + 409 at > gen_redcs_mat.c:531 [opt] frame #7: 0x000000010618f8ba > libpetsc.3.07.dylib`hypre_GaussElimSetup(amg_data=0x00007fe7ff857a00, > level=, relax_type=9) + 74 at par_relax.c:4209 [opt] frame > #8: 0x0000000106140e93 > libpetsc.3.07.dylib`hypre_BoomerAMGSetup(amg_vdata=, > A=0x00007fe80842aff0, f=0x00007fe80842a980, u=0x00007fe80842a510) + 17699 > at par_amg_setup.c:2108 [opt] frame #9: 0x0000000105ec773c > libpetsc.3.07.dylib`PCSetUp_HYPRE(pc=) + 2540 at hypre.c:226 > [opt* > > How did you perform the bisection? make clean + make all ? Which version > of HYPRE are you using? > > But you might want to verify [by linking with mpi trace library?] > > > There are some debugging hints at > https://lists.mpich.org/pipermail/discuss/2012-December/000148.html [wrt > mpich] - which I haven't checked.. > > Satish > > > > Fande, > > > > > To debug the hypre/duplication issue in MOOSE I would run in the > debugger with a break point in MPI_Comm_dup() and see > who keeps calling it an unreasonable amount of times. (My guess is this is > a new "feature" in hypre that they will need to fix but only debugging will > tell) > > Barry > > > On Apr 2, 2018, at 7:44 PM, Balay, Satish wrote: > > We do a MPI_Comm_dup() for objects related to externalpackages. > > Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is > using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think > is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 > > src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); > > src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm); > > CHKERRQ(ierr); > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); > > src/ksp/pc/impls/spai/ispai.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_ > spai));CHKERRQ(ierr); > > src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, > > &comm);CHKERRQ(ierr); > > src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_ > cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); > > src/mat/impls/aij/mpi/mumps/mumps.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_ > mumps));CHKERRQ(ierr); > > src/mat/impls/aij/mpi/pastix/pastix.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_ > comm));CHKERRQ(ierr); > > src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_ > superlu));CHKERRQ(ierr); > > src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( > > PetscObject)B),&hB->comm);CHKERRQ(ierr); > > src/mat/partition/impls/pmetis/pmetis.c: ierr = > > MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); > > src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a > > MPI_Comm_dup() of each of these (duplicates of duplicates return the same > communictor) > > src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) > src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& > > local_comm);CHKERRQ(ierr); > > src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& > > local_comm);CHKERRQ(ierr); > > src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out) > > ;CHKERRQ(ierr); > > src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm) > > ;CHKERRQ(ierr); > > src/ts/impls/implicit/sundials/sundials.c: ierr = > > MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_ > sundials));CHKERRQ(ierr); > > > Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid > > these MPI_Comm_dup() calls? > > > Satish > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > Are we sure this is a PETSc comm issue and not a hypre comm > > duplication issue > > > frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_ > > GenerateSubComm(comm=-1006627852, participate=, > new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] > > > Looks like hypre is needed to generate subcomms, perhaps it generates > > too many? > > > Barry > > > On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: > > I?m working with Fande on this and I would like to add a bit more. > > There are many circumstances where we aren?t working on COMM_WORLD at all > (e.g. working on a sub-communicator) but PETSc was initialized using > MPI_COMM_WORLD (think multi-level solves)? and we need to create > arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We > definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering > duplication. > > > Can you explain why PETSc needs to duplicate the communicator so much? > > Thanks for your help in tracking this down! > > Derek > > On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: > Why we do not use user-level MPI communicators directly? What are > > potential risks here? > > > > Fande, > > On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay > > wrote: > > PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls > > to MPI_Comm_dup() - thus potentially avoiding such errors > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs. > > anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_ > PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi > CY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_ > zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= > > > > Satish > > On Mon, 2 Apr 2018, Kong, Fande wrote: > > On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay > > wrote: > > > Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? > > If so - you could try changing to PETSC_COMM_WORLD > > > > I do not think we are using PETSC_COMM_WORLD when creating PETSc > > objects. > > Why we can not use MPI_COMM_WORLD? > > > Fande, > > > > Satish > > > On Mon, 2 Apr 2018, Kong, Fande wrote: > > Hi All, > > I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its > applications. I have a error message for a standard test: > > > > > > > > > > *preconditioners/pbp.lots_of_variables: MPI had an > errorpreconditioners/pbp.lots_of_variables: > ------------------------------------------------ > > preconditioners/pbp.lots_of_variables: > > Other MPI error, error stack:preconditioners/pbp.lots_of_variables: > PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, > new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: > PMPI_Comm_dup(162)..................: > preconditioners/pbp.lots_of_variables: > MPIR_Comm_dup_impl(57)..............: > preconditioners/pbp.lots_of_variables: > MPIR_Comm_copy(739).................: > preconditioners/pbp.lots_of_variables: > MPIR_Get_contextid_sparse_group(614): Too many communicators > > (0/2048 > > free > > on this process; ignore_id=0)* > > > I did "git bisect', and the following commit introduces this issue: > > > > > > > > > *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano > > Zampini > > >Date: Sat > > Nov 5 > > 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE > hpmat already stores two HYPRE vectors* > > Before I debug line-by-line, anyone has a clue on this? > > > Fande, > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Tue Apr 3 10:50:09 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Tue, 3 Apr 2018 17:50:09 +0200 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: <6F4AE107-B379-47E8-9453-CBE42AAD823A@gmail.com> > On Apr 3, 2018, at 5:43 PM, Fande Kong wrote: > > > > On Tue, Apr 3, 2018 at 9:12 AM, Stefano Zampini > wrote: > >> On Apr 3, 2018, at 4:58 PM, Satish Balay > wrote: >> >> On Tue, 3 Apr 2018, Kong, Fande wrote: >> >>> On Tue, Apr 3, 2018 at 1:17 AM, Smith, Barry F. > wrote: >>> >>>> >>>> Each external package definitely needs its own duplicated communicator; >>>> cannot share between packages. >>>> >>>> The only problem with the dups below is if they are in a loop and get >>>> called many times. >>>> >>> >>> >>> The "standard test" that has this issue actually has 1K fields. MOOSE >>> creates its own field-split preconditioner (not based on the PETSc >>> fieldsplit), and each filed is associated with one PC HYPRE. If PETSc >>> duplicates communicators, we should easily reach the limit 2048. >>> >>> I also want to confirm what extra communicators are introduced in the bad >>> commit. >> >> To me it looks like there is 1 extra comm created [for MATHYPRE] for each PCHYPRE that is created [which also creates one comm for this object]. >> > > You?re right; however, it was the same before the commit. > I don?t understand how this specific commit is related with this issue, being the error not in the MPI_Comm_Dup which is inside MatCreate_MATHYPRE. Actually, the error comes from MPI_Comm_create > > frame #5: 0x00000001068defd4 libmpi.12.dylib`MPI_Comm_create + 3492 > frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, participate=, new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] > frame #7: 0x000000010618f8ba libpetsc.3.07.dylib`hypre_GaussElimSetup(amg_data=0x00007fe7ff857a00, level=, relax_type=9) + 74 at par_relax.c:4209 [opt] > frame #8: 0x0000000106140e93 libpetsc.3.07.dylib`hypre_BoomerAMGSetup(amg_vdata=, A=0x00007fe80842aff0, f=0x00007fe80842a980, u=0x00007fe80842a510) + 17699 at par_amg_setup.c:2108 [opt] > frame #9: 0x0000000105ec773c libpetsc.3.07.dylib`PCSetUp_HYPRE(pc=) + 2540 at hypre.c:226 [opt > > How did you perform the bisection? make clean + make all ? Which version of HYPRE are you using? > > I did more aggressively. > > "rm -rf arch-darwin-c-opt-bisect " > > "./configure --optionsModule=config.compilerOptions -with-debugging=no --with-shared-libraries=1 --with-mpi=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 PETSC_ARCH=arch-darwin-c-opt-bisect" > Good, so this removes some possible sources of errors > > HYPRE verison: > > > self.gitcommit = 'v2.11.1-55-g2ea0e43' > self.download = ['git://https://github.com/LLNL/hypre ','https://github.com/LLNL/hypre/archive/'+self.gitcommit+'.tar.gz '] > > When reconfiguring, the HYPRE version can be different too (that commit is from 11/2016, so the HYPRE version used by the PETSc configure can have been upgraded too) > I do not think this is caused by HYPRE. > > Fande, > > > >> But you might want to verify [by linking with mpi trace library?] >> >> >> There are some debugging hints at https://lists.mpich.org/pipermail/discuss/2012-December/000148.html [wrt mpich] - which I haven't checked.. >> >> Satish >> >>> >>> >>> Fande, >>> >>> >>> >>>> >>>> To debug the hypre/duplication issue in MOOSE I would run in the >>>> debugger with a break point in MPI_Comm_dup() and see >>>> who keeps calling it an unreasonable amount of times. (My guess is this is >>>> a new "feature" in hypre that they will need to fix but only debugging will >>>> tell) >>>> >>>> Barry >>>> >>>> >>>>> On Apr 2, 2018, at 7:44 PM, Balay, Satish > wrote: >>>>> >>>>> We do a MPI_Comm_dup() for objects related to externalpackages. >>>>> >>>>> Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is >>>>> using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think >>>>> is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 >>>>> >>>>> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>>> PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); >>>>> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>>> PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); >>>>> src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm); >>>> CHKERRQ(ierr); >>>>> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>>> PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); >>>>> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>>> PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); >>>>> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>>> PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); >>>>> src/ksp/pc/impls/spai/ispai.c: ierr = >>>> MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_ >>>> spai));CHKERRQ(ierr); >>>>> src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, >>>> &comm);CHKERRQ(ierr); >>>>> src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = >>>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_ >>>> cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); >>>>> src/mat/impls/aij/mpi/mumps/mumps.c: ierr = >>>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_ >>>> mumps));CHKERRQ(ierr); >>>>> src/mat/impls/aij/mpi/pastix/pastix.c: ierr = >>>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_ >>>> comm));CHKERRQ(ierr); >>>>> src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = >>>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_ >>>> superlu));CHKERRQ(ierr); >>>>> src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>>> PetscObject)B),&hB->comm);CHKERRQ(ierr); >>>>> src/mat/partition/impls/pmetis/pmetis.c: ierr = >>>> MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); >>>>> src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a >>>> MPI_Comm_dup() of each of these (duplicates of duplicates return the same >>>> communictor) >>>>> src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) >>>>> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& >>>> local_comm);CHKERRQ(ierr); >>>>> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& >>>> local_comm);CHKERRQ(ierr); >>>>> src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out) >>>> ;CHKERRQ(ierr); >>>>> src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm) >>>> ;CHKERRQ(ierr); >>>>> src/ts/impls/implicit/sundials/sundials.c: ierr = >>>> MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_ >>>> sundials));CHKERRQ(ierr); >>>>> >>>>> Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid >>>> these MPI_Comm_dup() calls? >>>>> >>>>> Satish >>>>> >>>>> On Tue, 3 Apr 2018, Smith, Barry F. wrote: >>>>> >>>>>> >>>>>> Are we sure this is a PETSc comm issue and not a hypre comm >>>> duplication issue >>>>>> >>>>>> frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_ >>>> GenerateSubComm(comm=-1006627852, participate=, >>>> new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] >>>>>> >>>>>> Looks like hypre is needed to generate subcomms, perhaps it generates >>>> too many? >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>>> On Apr 2, 2018, at 7:07 PM, Derek Gaston > wrote: >>>>>>> >>>>>>> I?m working with Fande on this and I would like to add a bit more. >>>> There are many circumstances where we aren?t working on COMM_WORLD at all >>>> (e.g. working on a sub-communicator) but PETSc was initialized using >>>> MPI_COMM_WORLD (think multi-level solves)? and we need to create >>>> arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We >>>> definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering >>>> duplication. >>>>>>> >>>>>>> Can you explain why PETSc needs to duplicate the communicator so much? >>>>>>> >>>>>>> Thanks for your help in tracking this down! >>>>>>> >>>>>>> Derek >>>>>>> >>>>>>> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande > wrote: >>>>>>> Why we do not use user-level MPI communicators directly? What are >>>> potential risks here? >>>>>>> >>>>>>> >>>>>>> Fande, >>>>>>> >>>>>>> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay > >>>> wrote: >>>>>>> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls >>>> to MPI_Comm_dup() - thus potentially avoiding such errors >>>>>>> >>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs . >>>> anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_ >>>> PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ >>>> _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi >>>> CY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_ >>>> zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= >>>>>>> >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: >>>>>>> >>>>>>>> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay > >>>> wrote: >>>>>>>> >>>>>>>>> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? >>>>>>>>> >>>>>>>>> If so - you could try changing to PETSC_COMM_WORLD >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I do not think we are using PETSC_COMM_WORLD when creating PETSc >>>> objects. >>>>>>>> Why we can not use MPI_COMM_WORLD? >>>>>>>> >>>>>>>> >>>>>>>> Fande, >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Satish >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its >>>>>>>>>> applications. I have a error message for a standard test: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *preconditioners/pbp.lots_of_variables: MPI had an >>>>>>>>>> errorpreconditioners/pbp.lots_of_variables: >>>>>>>>>> ------------------------------------------------ >>>>>>>>> preconditioners/pbp.lots_of_variables: >>>>>>>>>> Other MPI error, error stack:preconditioners/pbp.lots_of_variables: >>>>>>>>>> PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, >>>>>>>>>> new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: >>>>>>>>>> PMPI_Comm_dup(162)..................: >>>>>>>>>> preconditioners/pbp.lots_of_variables: >>>>>>>>>> MPIR_Comm_dup_impl(57)..............: >>>>>>>>>> preconditioners/pbp.lots_of_variables: >>>>>>>>>> MPIR_Comm_copy(739).................: >>>>>>>>>> preconditioners/pbp.lots_of_variables: >>>>>>>>>> MPIR_Get_contextid_sparse_group(614): Too many communicators >>>> (0/2048 >>>>>>>>> free >>>>>>>>>> on this process; ignore_id=0)* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I did "git bisect', and the following commit introduces this issue: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano >>>> Zampini >>>>>>>>>> >>Date: Sat >>>> Nov 5 >>>>>>>>>> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE >>>>>>>>>> hpmat already stores two HYPRE vectors* >>>>>>>>>> >>>>>>>>>> Before I debug line-by-line, anyone has a clue on this? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Fande, >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Tue Apr 3 11:14:44 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Tue, 3 Apr 2018 10:14:44 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: <6F4AE107-B379-47E8-9453-CBE42AAD823A@gmail.com> References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <6F4AE107-B379-47E8-9453-CBE42AAD823A@gmail.com> Message-ID: The first bad commit: *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano Zampini >Date: Sat Nov 5 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE hpmat already stores two HYPRE vectors* Hypre version: ~/projects/petsc/arch-darwin-c-opt-bisect_bad/externalpackages/git.hypre]> git branch * (HEAD detached at 83b1f19) The last good commit: *commit 63c07aad33d943fe85193412d077a1746a7c55aaAuthor: Stefano Zampini >Date: Sat Nov 5 19:30:12 2016 +0300 MatHYPRE: create new matrix type The conversion from AIJ to HYPRE has been taken from src/dm/impls/da/hypre/mhyp.c HYPRE to AIJ is new* Hypre version: /projects/petsc/arch-darwin-c-opt-bisect/externalpackages/git.hypre]> git branch * (HEAD detached at 83b1f19) We are using the same HYPRE version. I will narrow down line-by-line. Fande, On Tue, Apr 3, 2018 at 9:50 AM, Stefano Zampini wrote: > > On Apr 3, 2018, at 5:43 PM, Fande Kong wrote: > > > > On Tue, Apr 3, 2018 at 9:12 AM, Stefano Zampini > wrote: > >> >> On Apr 3, 2018, at 4:58 PM, Satish Balay wrote: >> >> On Tue, 3 Apr 2018, Kong, Fande wrote: >> >> On Tue, Apr 3, 2018 at 1:17 AM, Smith, Barry F. >> wrote: >> >> >> Each external package definitely needs its own duplicated communicator; >> cannot share between packages. >> >> The only problem with the dups below is if they are in a loop and get >> called many times. >> >> >> >> The "standard test" that has this issue actually has 1K fields. MOOSE >> creates its own field-split preconditioner (not based on the PETSc >> fieldsplit), and each filed is associated with one PC HYPRE. If PETSc >> duplicates communicators, we should easily reach the limit 2048. >> >> I also want to confirm what extra communicators are introduced in the bad >> commit. >> >> >> To me it looks like there is 1 extra comm created [for MATHYPRE] for each >> PCHYPRE that is created [which also creates one comm for this object]. >> >> >> You?re right; however, it was the same before the commit. >> I don?t understand how this specific commit is related with this issue, >> being the error not in the MPI_Comm_Dup which is inside MatCreate_MATHYPRE. >> Actually, the error comes from MPI_Comm_create >> >> >> >> >> >> * frame #5: 0x00000001068defd4 libmpi.12.dylib`MPI_Comm_create + >> 3492 frame #6: 0x00000001061345d9 >> libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, >> participate=, new_comm_ptr=) + 409 at >> gen_redcs_mat.c:531 [opt] frame #7: 0x000000010618f8ba >> libpetsc.3.07.dylib`hypre_GaussElimSetup(amg_data=0x00007fe7ff857a00, >> level=, relax_type=9) + 74 at par_relax.c:4209 [opt] frame >> #8: 0x0000000106140e93 >> libpetsc.3.07.dylib`hypre_BoomerAMGSetup(amg_vdata=, >> A=0x00007fe80842aff0, f=0x00007fe80842a980, u=0x00007fe80842a510) + 17699 >> at par_amg_setup.c:2108 [opt] frame #9: 0x0000000105ec773c >> libpetsc.3.07.dylib`PCSetUp_HYPRE(pc=) + 2540 at hypre.c:226 >> [opt* >> >> How did you perform the bisection? make clean + make all ? Which version >> of HYPRE are you using? >> > > I did more aggressively. > > "rm -rf arch-darwin-c-opt-bisect " > > "./configure --optionsModule=config.compilerOptions -with-debugging=no > --with-shared-libraries=1 --with-mpi=1 --download-fblaslapack=1 > --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > --download-hypre=1 --download-mumps=1 --download-scalapack=1 > PETSC_ARCH=arch-darwin-c-opt-bisect" > > > Good, so this removes some possible sources of errors > > > HYPRE verison: > > > self.gitcommit = 'v2.11.1-55-g2ea0e43' > self.download = ['git://https://github.com/LLNL/hypre > > ','https://github.com/LLNL/hypre/archive/'+self.gitcommit+'.tar.gz > > '] > > > > When reconfiguring, the HYPRE version can be different too (that commit > is from 11/2016, so the HYPRE version used by the PETSc configure can have > been upgraded too) > > I do not think this is caused by HYPRE. > > > Fande, > > > >> >> But you might want to verify [by linking with mpi trace library?] >> >> >> There are some debugging hints at https://lists.mpich.org/piperm >> ail/discuss/2012-December/000148.html >> >> [wrt mpich] - which I haven't checked.. >> >> Satish >> >> >> >> Fande, >> >> >> >> >> To debug the hypre/duplication issue in MOOSE I would run in the >> debugger with a break point in MPI_Comm_dup() and see >> who keeps calling it an unreasonable amount of times. (My guess is this is >> a new "feature" in hypre that they will need to fix but only debugging >> will >> tell) >> >> Barry >> >> >> On Apr 2, 2018, at 7:44 PM, Balay, Satish wrote: >> >> We do a MPI_Comm_dup() for objects related to externalpackages. >> >> Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is >> using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think >> is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 >> >> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( >> >> PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); >> >> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( >> >> PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); >> >> src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm); >> >> CHKERRQ(ierr); >> >> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >> >> PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); >> >> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >> >> PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); >> >> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >> >> PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); >> >> src/ksp/pc/impls/spai/ispai.c: ierr = >> >> MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_ >> spai));CHKERRQ(ierr); >> >> src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, >> >> &comm);CHKERRQ(ierr); >> >> src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = >> >> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_ >> cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); >> >> src/mat/impls/aij/mpi/mumps/mumps.c: ierr = >> >> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_ >> mumps));CHKERRQ(ierr); >> >> src/mat/impls/aij/mpi/pastix/pastix.c: ierr = >> >> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_ >> comm));CHKERRQ(ierr); >> >> src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = >> >> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_ >> superlu));CHKERRQ(ierr); >> >> src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >> >> PetscObject)B),&hB->comm);CHKERRQ(ierr); >> >> src/mat/partition/impls/pmetis/pmetis.c: ierr = >> >> MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); >> >> src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a >> >> MPI_Comm_dup() of each of these (duplicates of duplicates return the same >> communictor) >> >> src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) >> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& >> >> local_comm);CHKERRQ(ierr); >> >> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& >> >> local_comm);CHKERRQ(ierr); >> >> src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out) >> >> ;CHKERRQ(ierr); >> >> src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm) >> >> ;CHKERRQ(ierr); >> >> src/ts/impls/implicit/sundials/sundials.c: ierr = >> >> MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_ >> sundials));CHKERRQ(ierr); >> >> >> Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid >> >> these MPI_Comm_dup() calls? >> >> >> Satish >> >> On Tue, 3 Apr 2018, Smith, Barry F. wrote: >> >> >> Are we sure this is a PETSc comm issue and not a hypre comm >> >> duplication issue >> >> >> frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_ >> >> GenerateSubComm(comm=-1006627852, participate=, >> new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] >> >> >> Looks like hypre is needed to generate subcomms, perhaps it generates >> >> too many? >> >> >> Barry >> >> >> On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: >> >> I?m working with Fande on this and I would like to add a bit more. >> >> There are many circumstances where we aren?t working on COMM_WORLD at all >> (e.g. working on a sub-communicator) but PETSc was initialized using >> MPI_COMM_WORLD (think multi-level solves)? and we need to create >> arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We >> definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering >> duplication. >> >> >> Can you explain why PETSc needs to duplicate the communicator so much? >> >> Thanks for your help in tracking this down! >> >> Derek >> >> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: >> Why we do not use user-level MPI communicators directly? What are >> >> potential risks here? >> >> >> >> Fande, >> >> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay >> >> wrote: >> >> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls >> >> to MPI_Comm_dup() - thus potentially avoiding such errors >> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs. >> >> anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_ >> PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ >> _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi >> CY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_ >> zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= >> >> >> >> Satish >> >> On Mon, 2 Apr 2018, Kong, Fande wrote: >> >> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay >> >> wrote: >> >> >> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? >> >> If so - you could try changing to PETSC_COMM_WORLD >> >> >> >> I do not think we are using PETSC_COMM_WORLD when creating PETSc >> >> objects. >> >> Why we can not use MPI_COMM_WORLD? >> >> >> Fande, >> >> >> >> Satish >> >> >> On Mon, 2 Apr 2018, Kong, Fande wrote: >> >> Hi All, >> >> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its >> applications. I have a error message for a standard test: >> >> >> >> >> >> >> >> >> >> *preconditioners/pbp.lots_of_variables: MPI had an >> errorpreconditioners/pbp.lots_of_variables: >> ------------------------------------------------ >> >> preconditioners/pbp.lots_of_variables: >> >> Other MPI error, error stack:preconditioners/pbp.lots_of_variables: >> PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, >> new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: >> PMPI_Comm_dup(162)..................: >> preconditioners/pbp.lots_of_variables: >> MPIR_Comm_dup_impl(57)..............: >> preconditioners/pbp.lots_of_variables: >> MPIR_Comm_copy(739).................: >> preconditioners/pbp.lots_of_variables: >> MPIR_Get_contextid_sparse_group(614): Too many communicators >> >> (0/2048 >> >> free >> >> on this process; ignore_id=0)* >> >> >> I did "git bisect', and the following commit introduces this issue: >> >> >> >> >> >> >> >> >> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano >> >> Zampini >> >> >Date: Sat >> >> Nov 5 >> >> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE >> hpmat already stores two HYPRE vectors* >> >> Before I debug line-by-line, anyone has a clue on this? >> >> >> Fande, >> >> >> >> >> >> >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Apr 3 11:30:54 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 3 Apr 2018 11:30:54 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: On Tue, 3 Apr 2018, Derek Gaston wrote: > One thing I want to be clear of here: is that we're not trying to solve > this particular problem (where we're creating 1000 instances of Hypre to > precondition each variable independently)... this particular problem is > just a test (that we've had in our test suite for a long time) to stress > test some of this capability. > > We really do have needs for thousands (tens of thousands) of simultaneous > solves (each with their own Hypre instances). That's not what this > particular problem is doing - but it is representative of a class of our > problems we need to solve. > > Which does bring up a point: I have been able to do solves before with > ~50,000 separate PETSc solves without issue. Is it because I was working > with MVAPICH on a cluster? Does it just have a higher limit? Don't know - but thats easy to find out with a simple test code.. >>>>>> $ cat comm_dup_test.c #include #include int main(int argc, char** argv) { MPI_Comm newcomm; int i, err; MPI_Init(NULL, NULL); for (i=0; i<100000; i++) { err = MPI_Comm_dup(MPI_COMM_WORLD, &newcomm); if (err) { printf("%5d - fail\n",i);fflush(stdout); break; } else { printf("%5d - success\n",i);fflush(stdout); } } MPI_Finalize(); } <<<<<<< OpenMPI fails after '65531' and mpich after '2044'. MVAPICH is derived off MPICH - but its possible they have a different limit than MPICH. Satish From balay at mcs.anl.gov Tue Apr 3 11:35:04 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 3 Apr 2018 11:35:04 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: On Tue, 3 Apr 2018, Satish Balay wrote: > On Tue, 3 Apr 2018, Derek Gaston wrote: > > > One thing I want to be clear of here: is that we're not trying to solve > > this particular problem (where we're creating 1000 instances of Hypre to > > precondition each variable independently)... this particular problem is > > just a test (that we've had in our test suite for a long time) to stress > > test some of this capability. > > > > We really do have needs for thousands (tens of thousands) of simultaneous > > solves (each with their own Hypre instances). That's not what this > > particular problem is doing - but it is representative of a class of our > > problems we need to solve. > > > > Which does bring up a point: I have been able to do solves before with > > ~50,000 separate PETSc solves without issue. Is it because I was working > > with MVAPICH on a cluster? Does it just have a higher limit? > > Don't know - but thats easy to find out with a simple test code.. > > >>>>>> > $ cat comm_dup_test.c > #include > #include > > int main(int argc, char** argv) { > MPI_Comm newcomm; > int i, err; > MPI_Init(NULL, NULL); > for (i=0; i<100000; i++) { > err = MPI_Comm_dup(MPI_COMM_WORLD, &newcomm); > if (err) { > printf("%5d - fail\n",i);fflush(stdout); > break; > } else { > printf("%5d - success\n",i);fflush(stdout); > } > } > MPI_Finalize(); > } > <<<<<<< > > OpenMPI fails after '65531' and mpich after '2044'. MVAPICH is derived > off MPICH - but its possible they have a different limit than MPICH. BTW: the above is with: openmpi-2.1.2 and mpich-3.3b1 mvapich2-1.9.5 - and I get error after '2044' comm dupes Satish From fande.kong at inl.gov Tue Apr 3 12:21:06 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Tue, 3 Apr 2018 11:21:06 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: Figured out: The reason is that in MatCreate_HYPRE(Mat B), we call MPI_Comm_dup instead of PetscCommDuplicate. The PetscCommDuplicate is better, and it does not actually create a communicator if the communicator is already known to PETSc. Furthermore, I do not think we should a comm in *typedef struct { HYPRE_IJMatrix ij; HYPRE_IJVector x; HYPRE_IJVector b; MPI_Comm comm;} Mat_HYPRE;* It is an inner data of Mat, and it should already the same comm as the Mat. I do not understand why the internal data has its own comm. The following patch fixed the issue (just deleted this extra comm). diff --git a/src/mat/impls/hypre/mhypre.c b/src/mat/impls/hypre/mhypre.c index dc19892..d8cfe3d 100644 --- a/src/mat/impls/hypre/mhypre.c +++ b/src/mat/impls/hypre/mhypre.c @@ -74,7 +74,7 @@ static PetscErrorCode MatHYPRE_CreateFromMat(Mat A, Mat_HYPRE *hA) rend = A->rmap->rend; cstart = A->cmap->rstart; cend = A->cmap->rend; - PetscStackCallStandard(HYPRE_IJMatrixCreate,(hA->comm,rstart,rend-1,cstart,cend-1,&hA->ij)); + PetscStackCallStandard(HYPRE_IJMatrixCreate,(PetscObjectComm((PetscObject)A),rstart,rend-1,cstart,cend-1,&hA->ij)); PetscStackCallStandard(HYPRE_IJMatrixSetObjectType,(hA->ij,HYPRE_PARCSR)); { PetscBool same; @@ -434,7 +434,7 @@ PetscErrorCode MatDestroy_HYPRE(Mat A) if (hA->x) PetscStackCallStandard(HYPRE_IJVectorDestroy,(hA->x)); if (hA->b) PetscStackCallStandard(HYPRE_IJVectorDestroy,(hA->b)); if (hA->ij) PetscStackCallStandard(HYPRE_IJMatrixDestroy,(hA->ij)); - if (hA->comm) { ierr = MPI_Comm_free(&hA->comm);CHKERRQ(ierr);} + /*if (hA->comm) { ierr = MPI_Comm_free(&hA->comm);CHKERRQ(ierr);}*/ ierr = PetscObjectComposeFunction((PetscObject)A,"MatConvert_hypre_aij_C",NULL);CHKERRQ(ierr); ierr = PetscFree(A->data);CHKERRQ(ierr); PetscFunctionReturn(0); @@ -500,7 +500,8 @@ PETSC_EXTERN PetscErrorCode MatCreate_HYPRE(Mat B) B->ops->destroy = MatDestroy_HYPRE; B->ops->assemblyend = MatAssemblyEnd_HYPRE; - ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); + /*ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); */ + /*ierr = PetscCommDuplicate(PetscObjectComm((PetscObject)B),&hB->comm,NULL);CHKERRQ(ierr);*/ ierr = PetscObjectChangeTypeName((PetscObject)B,MATHYPRE);CHKERRQ(ierr); ierr = PetscObjectComposeFunction((PetscObject)B,"MatConvert_hypre_aij_C",MatConvert_HYPRE_AIJ);CHKERRQ(ierr); PetscFunctionReturn(0); diff --git a/src/mat/impls/hypre/mhypre.h b/src/mat/impls/hypre/mhypre.h index 3d9ddd2..1189020 100644 --- a/src/mat/impls/hypre/mhypre.h +++ b/src/mat/impls/hypre/mhypre.h @@ -10,7 +10,7 @@ typedef struct { HYPRE_IJMatrix ij; HYPRE_IJVector x; HYPRE_IJVector b; - MPI_Comm comm; + /*MPI_Comm comm;*/ } Mat_HYPRE; Fande, On Tue, Apr 3, 2018 at 10:35 AM, Satish Balay wrote: > On Tue, 3 Apr 2018, Satish Balay wrote: > > > On Tue, 3 Apr 2018, Derek Gaston wrote: > > > > > One thing I want to be clear of here: is that we're not trying to solve > > > this particular problem (where we're creating 1000 instances of Hypre > to > > > precondition each variable independently)... this particular problem is > > > just a test (that we've had in our test suite for a long time) to > stress > > > test some of this capability. > > > > > > We really do have needs for thousands (tens of thousands) of > simultaneous > > > solves (each with their own Hypre instances). That's not what this > > > particular problem is doing - but it is representative of a class of > our > > > problems we need to solve. > > > > > > Which does bring up a point: I have been able to do solves before with > > > ~50,000 separate PETSc solves without issue. Is it because I was > working > > > with MVAPICH on a cluster? Does it just have a higher limit? > > > > Don't know - but thats easy to find out with a simple test code.. > > > > >>>>>> > > $ cat comm_dup_test.c > > #include > > #include > > > > int main(int argc, char** argv) { > > MPI_Comm newcomm; > > int i, err; > > MPI_Init(NULL, NULL); > > for (i=0; i<100000; i++) { > > err = MPI_Comm_dup(MPI_COMM_WORLD, &newcomm); > > if (err) { > > printf("%5d - fail\n",i);fflush(stdout); > > break; > > } else { > > printf("%5d - success\n",i);fflush(stdout); > > } > > } > > MPI_Finalize(); > > } > > <<<<<<< > > > > OpenMPI fails after '65531' and mpich after '2044'. MVAPICH is derived > > off MPICH - but its possible they have a different limit than MPICH. > > BTW: the above is with: openmpi-2.1.2 and mpich-3.3b1 > > mvapich2-1.9.5 - and I get error after '2044' comm dupes > > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 3 12:24:20 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 3 Apr 2018 17:24:20 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <6F4AE107-B379-47E8-9453-CBE42AAD823A@gmail.com> Message-ID: <3B83B981-869D-4BB6-BA6E-E5A122E0BA20@anl.gov> Fande, Please try the branch https://bitbucket.org/petsc/petsc/pull-requests/921/boomeramg-unlike-2-other-hypre/diff It does not "solve" the problem but it should get your current test that now fails to run again, Barry > On Apr 3, 2018, at 10:14 AM, Kong, Fande wrote: > > The first bad commit: > > commit 49a781f5cee36db85e8d5b951eec29f10ac13593 > Author: Stefano Zampini > Date: Sat Nov 5 20:15:19 2016 +0300 > > PCHYPRE: use internal Mat of type MatHYPRE > > hpmat already stores two HYPRE vectors > > > Hypre version: > > ~/projects/petsc/arch-darwin-c-opt-bisect_bad/externalpackages/git.hypre]> git branch > * (HEAD detached at 83b1f19) > > > > The last good commit: > > commit 63c07aad33d943fe85193412d077a1746a7c55aa > Author: Stefano Zampini > Date: Sat Nov 5 19:30:12 2016 +0300 > > MatHYPRE: create new matrix type > > The conversion from AIJ to HYPRE has been taken from src/dm/impls/da/hypre/mhyp.c > HYPRE to AIJ is new > > Hypre version: > > /projects/petsc/arch-darwin-c-opt-bisect/externalpackages/git.hypre]> git branch > * (HEAD detached at 83b1f19) > > > > > > We are using the same HYPRE version. > > > I will narrow down line-by-line. > > > Fande, > > > On Tue, Apr 3, 2018 at 9:50 AM, Stefano Zampini wrote: > >> On Apr 3, 2018, at 5:43 PM, Fande Kong wrote: >> >> >> >> On Tue, Apr 3, 2018 at 9:12 AM, Stefano Zampini wrote: >> >>> On Apr 3, 2018, at 4:58 PM, Satish Balay wrote: >>> >>> On Tue, 3 Apr 2018, Kong, Fande wrote: >>> >>>> On Tue, Apr 3, 2018 at 1:17 AM, Smith, Barry F. wrote: >>>> >>>>> >>>>> Each external package definitely needs its own duplicated communicator; >>>>> cannot share between packages. >>>>> >>>>> The only problem with the dups below is if they are in a loop and get >>>>> called many times. >>>>> >>>> >>>> >>>> The "standard test" that has this issue actually has 1K fields. MOOSE >>>> creates its own field-split preconditioner (not based on the PETSc >>>> fieldsplit), and each filed is associated with one PC HYPRE. If PETSc >>>> duplicates communicators, we should easily reach the limit 2048. >>>> >>>> I also want to confirm what extra communicators are introduced in the bad >>>> commit. >>> >>> To me it looks like there is 1 extra comm created [for MATHYPRE] for each PCHYPRE that is created [which also creates one comm for this object]. >>> >> >> You?re right; however, it was the same before the commit. >> I don?t understand how this specific commit is related with this issue, being the error not in the MPI_Comm_Dup which is inside MatCreate_MATHYPRE. Actually, the error comes from MPI_Comm_create >> >> frame #5: 0x00000001068defd4 libmpi.12.dylib`MPI_Comm_create + 3492 >> frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_GenerateSubComm(comm=-1006627852, participate=, new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] >> frame #7: 0x000000010618f8ba libpetsc.3.07.dylib`hypre_GaussElimSetup(amg_data=0x00007fe7ff857a00, level=, relax_type=9) + 74 at par_relax.c:4209 [opt] >> frame #8: 0x0000000106140e93 libpetsc.3.07.dylib`hypre_BoomerAMGSetup(amg_vdata=, A=0x00007fe80842aff0, f=0x00007fe80842a980, u=0x00007fe80842a510) + 17699 at par_amg_setup.c:2108 [opt] >> frame #9: 0x0000000105ec773c libpetsc.3.07.dylib`PCSetUp_HYPRE(pc=) + 2540 at hypre.c:226 [opt >> >> How did you perform the bisection? make clean + make all ? Which version of HYPRE are you using? >> >> I did more aggressively. >> >> "rm -rf arch-darwin-c-opt-bisect " >> >> "./configure --optionsModule=config.compilerOptions -with-debugging=no --with-shared-libraries=1 --with-mpi=1 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 PETSC_ARCH=arch-darwin-c-opt-bisect" >> > > Good, so this removes some possible sources of errors >> >> HYPRE verison: >> >> >> self.gitcommit = 'v2.11.1-55-g2ea0e43' >> self.download = ['git://https://github.com/LLNL/hypre','https://github.com/LLNL/hypre/archive/'+self.gitcommit+'.tar.gz'] >> >> > > When reconfiguring, the HYPRE version can be different too (that commit is from 11/2016, so the HYPRE version used by the PETSc configure can have been upgraded too) > >> I do not think this is caused by HYPRE. >> >> Fande, >> >> >> >>> But you might want to verify [by linking with mpi trace library?] >>> >>> >>> There are some debugging hints at https://lists.mpich.org/pipermail/discuss/2012-December/000148.html [wrt mpich] - which I haven't checked.. >>> >>> Satish >>> >>>> >>>> >>>> Fande, >>>> >>>> >>>> >>>>> >>>>> To debug the hypre/duplication issue in MOOSE I would run in the >>>>> debugger with a break point in MPI_Comm_dup() and see >>>>> who keeps calling it an unreasonable amount of times. (My guess is this is >>>>> a new "feature" in hypre that they will need to fix but only debugging will >>>>> tell) >>>>> >>>>> Barry >>>>> >>>>> >>>>>> On Apr 2, 2018, at 7:44 PM, Balay, Satish wrote: >>>>>> >>>>>> We do a MPI_Comm_dup() for objects related to externalpackages. >>>>>> >>>>>> Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is >>>>>> using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think >>>>>> is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7 >>>>>> >>>>>> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>>>> PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); >>>>>> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>>>> PetscObject)B),&(ex->hcomm));CHKERRQ(ierr); >>>>>> src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm); >>>>> CHKERRQ(ierr); >>>>>> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>>>> PetscObject)pc),&(jac->comm_hypre));CHKERRQ(ierr); >>>>>> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>>>> PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); >>>>>> src/ksp/pc/impls/hypre/hypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>>>> PetscObject)pc),&(ex->hcomm));CHKERRQ(ierr); >>>>>> src/ksp/pc/impls/spai/ispai.c: ierr = >>>>> MPI_Comm_dup(PetscObjectComm((PetscObject)pc),&(ispai->comm_ >>>>> spai));CHKERRQ(ierr); >>>>>> src/mat/examples/tests/ex152.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, >>>>> &comm);CHKERRQ(ierr); >>>>>> src/mat/impls/aij/mpi/mkl_cpardiso/mkl_cpardiso.c: ierr = >>>>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mat_mkl_ >>>>> cpardiso->comm_mkl_cpardiso));CHKERRQ(ierr); >>>>>> src/mat/impls/aij/mpi/mumps/mumps.c: ierr = >>>>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(mumps->comm_ >>>>> mumps));CHKERRQ(ierr); >>>>>> src/mat/impls/aij/mpi/pastix/pastix.c: ierr = >>>>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->pastix_ >>>>> comm));CHKERRQ(ierr); >>>>>> src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c: ierr = >>>>> MPI_Comm_dup(PetscObjectComm((PetscObject)A),&(lu->comm_ >>>>> superlu));CHKERRQ(ierr); >>>>>> src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm(( >>>>> PetscObject)B),&hB->comm);CHKERRQ(ierr); >>>>>> src/mat/partition/impls/pmetis/pmetis.c: ierr = >>>>> MPI_Comm_dup(pcomm,&comm);CHKERRQ(ierr); >>>>>> src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a >>>>> MPI_Comm_dup() of each of these (duplicates of duplicates return the same >>>>> communictor) >>>>>> src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out) >>>>>> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& >>>>> local_comm);CHKERRQ(ierr); >>>>>> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,& >>>>> local_comm);CHKERRQ(ierr); >>>>>> src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out) >>>>> ;CHKERRQ(ierr); >>>>>> src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm) >>>>> ;CHKERRQ(ierr); >>>>>> src/ts/impls/implicit/sundials/sundials.c: ierr = >>>>> MPI_Comm_dup(PetscObjectComm((PetscObject)ts),&(cvode->comm_ >>>>> sundials));CHKERRQ(ierr); >>>>>> >>>>>> Perhaps we need a PetscCommDuplicateExternalPkg() to somehow avoid >>>>> these MPI_Comm_dup() calls? >>>>>> >>>>>> Satish >>>>>> >>>>>> On Tue, 3 Apr 2018, Smith, Barry F. wrote: >>>>>> >>>>>>> >>>>>>> Are we sure this is a PETSc comm issue and not a hypre comm >>>>> duplication issue >>>>>>> >>>>>>> frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_ >>>>> GenerateSubComm(comm=-1006627852, participate=, >>>>> new_comm_ptr=) + 409 at gen_redcs_mat.c:531 [opt] >>>>>>> >>>>>>> Looks like hypre is needed to generate subcomms, perhaps it generates >>>>> too many? >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>>> On Apr 2, 2018, at 7:07 PM, Derek Gaston wrote: >>>>>>>> >>>>>>>> I?m working with Fande on this and I would like to add a bit more. >>>>> There are many circumstances where we aren?t working on COMM_WORLD at all >>>>> (e.g. working on a sub-communicator) but PETSc was initialized using >>>>> MPI_COMM_WORLD (think multi-level solves)? and we need to create >>>>> arbitrarily many PETSc vecs/mats/solvers/preconditioners and solve. We >>>>> definitely can?t rely on using PETSC_COMM_WORLD to avoid triggering >>>>> duplication. >>>>>>>> >>>>>>>> Can you explain why PETSc needs to duplicate the communicator so much? >>>>>>>> >>>>>>>> Thanks for your help in tracking this down! >>>>>>>> >>>>>>>> Derek >>>>>>>> >>>>>>>> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande wrote: >>>>>>>> Why we do not use user-level MPI communicators directly? What are >>>>> potential risks here? >>>>>>>> >>>>>>>> >>>>>>>> Fande, >>>>>>>> >>>>>>>> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay >>>>> wrote: >>>>>>>> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls >>>>> to MPI_Comm_dup() - thus potentially avoiding such errors >>>>>>>> >>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs. >>>>> anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_ >>>>> PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ >>>>> _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmi >>>>> CY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_ >>>>> zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e= >>>>>>>> >>>>>>>> >>>>>>>> Satish >>>>>>>> >>>>>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: >>>>>>>> >>>>>>>>> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay >>>>> wrote: >>>>>>>>> >>>>>>>>>> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects? >>>>>>>>>> >>>>>>>>>> If so - you could try changing to PETSC_COMM_WORLD >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I do not think we are using PETSC_COMM_WORLD when creating PETSc >>>>> objects. >>>>>>>>> Why we can not use MPI_COMM_WORLD? >>>>>>>>> >>>>>>>>> >>>>>>>>> Fande, >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Satish >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, 2 Apr 2018, Kong, Fande wrote: >>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its >>>>>>>>>>> applications. I have a error message for a standard test: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *preconditioners/pbp.lots_of_variables: MPI had an >>>>>>>>>>> errorpreconditioners/pbp.lots_of_variables: >>>>>>>>>>> ------------------------------------------------ >>>>>>>>>> preconditioners/pbp.lots_of_variables: >>>>>>>>>>> Other MPI error, error stack:preconditioners/pbp.lots_of_variables: >>>>>>>>>>> PMPI_Comm_dup(177)..................: MPI_Comm_dup(comm=0x84000001, >>>>>>>>>>> new_comm=0x97d1068) failedpreconditioners/pbp.lots_of_variables: >>>>>>>>>>> PMPI_Comm_dup(162)..................: >>>>>>>>>>> preconditioners/pbp.lots_of_variables: >>>>>>>>>>> MPIR_Comm_dup_impl(57)..............: >>>>>>>>>>> preconditioners/pbp.lots_of_variables: >>>>>>>>>>> MPIR_Comm_copy(739).................: >>>>>>>>>>> preconditioners/pbp.lots_of_variables: >>>>>>>>>>> MPIR_Get_contextid_sparse_group(614): Too many communicators >>>>> (0/2048 >>>>>>>>>> free >>>>>>>>>>> on this process; ignore_id=0)* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I did "git bisect', and the following commit introduces this issue: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *commit 49a781f5cee36db85e8d5b951eec29f10ac13593Author: Stefano >>>>> Zampini >>>>>>>>>>> >Date: Sat >>>>> Nov 5 >>>>>>>>>>> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE >>>>>>>>>>> hpmat already stores two HYPRE vectors* >>>>>>>>>>> >>>>>>>>>>> Before I debug line-by-line, anyone has a clue on this? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Fande, >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> >> >> > > From balay at mcs.anl.gov Tue Apr 3 12:27:28 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 3 Apr 2018 12:27:28 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: On Tue, 3 Apr 2018, Kong, Fande wrote: > Figured out: > > The reason is that in MatCreate_HYPRE(Mat B), we call MPI_Comm_dup > instead of PetscCommDuplicate. The PetscCommDuplicate is better, and it > does not actually create a communicator if the communicator is already > known to PETSc. > > Furthermore, I do not think we should a comm in > > > > > > > *typedef struct { HYPRE_IJMatrix ij; HYPRE_IJVector x; HYPRE_IJVector > b; MPI_Comm comm;} Mat_HYPRE;* > > It is an inner data of Mat, and it should already the same comm as the Mat. > I do not understand why the internal data has its own comm. As mentioned before - we create a separate comm using MPI_Comm_dup() to pass it down to hypre routines [corresponding to this petsc object]. PetscCommDuplicate() is used for all comm usage required within PETSc. The problem with this change is - now 2 libraries are using the same communicator - but don't know how to manage 'tags' in separate space - thus potentially resulting in messages from hypre picked up by petsc or vice versa. Satish > > The following patch fixed the issue (just deleted this extra comm). > > diff --git a/src/mat/impls/hypre/mhypre.c b/src/mat/impls/hypre/mhypre.c > index dc19892..d8cfe3d 100644 > --- a/src/mat/impls/hypre/mhypre.c > +++ b/src/mat/impls/hypre/mhypre.c > @@ -74,7 +74,7 @@ static PetscErrorCode MatHYPRE_CreateFromMat(Mat A, > Mat_HYPRE *hA) > rend = A->rmap->rend; > cstart = A->cmap->rstart; > cend = A->cmap->rend; > - > PetscStackCallStandard(HYPRE_IJMatrixCreate,(hA->comm,rstart,rend-1,cstart,cend-1,&hA->ij)); > + > PetscStackCallStandard(HYPRE_IJMatrixCreate,(PetscObjectComm((PetscObject)A),rstart,rend-1,cstart,cend-1,&hA->ij)); > > PetscStackCallStandard(HYPRE_IJMatrixSetObjectType,(hA->ij,HYPRE_PARCSR)); > { > PetscBool same; > @@ -434,7 +434,7 @@ PetscErrorCode MatDestroy_HYPRE(Mat A) > if (hA->x) PetscStackCallStandard(HYPRE_IJVectorDestroy,(hA->x)); > if (hA->b) PetscStackCallStandard(HYPRE_IJVectorDestroy,(hA->b)); > if (hA->ij) PetscStackCallStandard(HYPRE_IJMatrixDestroy,(hA->ij)); > - if (hA->comm) { ierr = MPI_Comm_free(&hA->comm);CHKERRQ(ierr);} > + /*if (hA->comm) { ierr = MPI_Comm_free(&hA->comm);CHKERRQ(ierr);}*/ > ierr = > PetscObjectComposeFunction((PetscObject)A,"MatConvert_hypre_aij_C",NULL);CHKERRQ(ierr); > ierr = PetscFree(A->data);CHKERRQ(ierr); > PetscFunctionReturn(0); > @@ -500,7 +500,8 @@ PETSC_EXTERN PetscErrorCode MatCreate_HYPRE(Mat B) > B->ops->destroy = MatDestroy_HYPRE; > B->ops->assemblyend = MatAssemblyEnd_HYPRE; > > - ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); > + /*ierr = > MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); */ > + /*ierr = > PetscCommDuplicate(PetscObjectComm((PetscObject)B),&hB->comm,NULL);CHKERRQ(ierr);*/ > ierr = PetscObjectChangeTypeName((PetscObject)B,MATHYPRE);CHKERRQ(ierr); > ierr = > PetscObjectComposeFunction((PetscObject)B,"MatConvert_hypre_aij_C",MatConvert_HYPRE_AIJ);CHKERRQ(ierr); > PetscFunctionReturn(0); > diff --git a/src/mat/impls/hypre/mhypre.h b/src/mat/impls/hypre/mhypre.h > index 3d9ddd2..1189020 100644 > --- a/src/mat/impls/hypre/mhypre.h > +++ b/src/mat/impls/hypre/mhypre.h > @@ -10,7 +10,7 @@ typedef struct { > HYPRE_IJMatrix ij; > HYPRE_IJVector x; > HYPRE_IJVector b; > - MPI_Comm comm; > + /*MPI_Comm comm;*/ > } Mat_HYPRE; > > > > Fande, > > > > > On Tue, Apr 3, 2018 at 10:35 AM, Satish Balay wrote: > > > On Tue, 3 Apr 2018, Satish Balay wrote: > > > > > On Tue, 3 Apr 2018, Derek Gaston wrote: > > > > > > > One thing I want to be clear of here: is that we're not trying to solve > > > > this particular problem (where we're creating 1000 instances of Hypre > > to > > > > precondition each variable independently)... this particular problem is > > > > just a test (that we've had in our test suite for a long time) to > > stress > > > > test some of this capability. > > > > > > > > We really do have needs for thousands (tens of thousands) of > > simultaneous > > > > solves (each with their own Hypre instances). That's not what this > > > > particular problem is doing - but it is representative of a class of > > our > > > > problems we need to solve. > > > > > > > > Which does bring up a point: I have been able to do solves before with > > > > ~50,000 separate PETSc solves without issue. Is it because I was > > working > > > > with MVAPICH on a cluster? Does it just have a higher limit? > > > > > > Don't know - but thats easy to find out with a simple test code.. > > > > > > >>>>>> > > > $ cat comm_dup_test.c > > > #include > > > #include > > > > > > int main(int argc, char** argv) { > > > MPI_Comm newcomm; > > > int i, err; > > > MPI_Init(NULL, NULL); > > > for (i=0; i<100000; i++) { > > > err = MPI_Comm_dup(MPI_COMM_WORLD, &newcomm); > > > if (err) { > > > printf("%5d - fail\n",i);fflush(stdout); > > > break; > > > } else { > > > printf("%5d - success\n",i);fflush(stdout); > > > } > > > } > > > MPI_Finalize(); > > > } > > > <<<<<<< > > > > > > OpenMPI fails after '65531' and mpich after '2044'. MVAPICH is derived > > > off MPICH - but its possible they have a different limit than MPICH. > > > > BTW: the above is with: openmpi-2.1.2 and mpich-3.3b1 > > > > mvapich2-1.9.5 - and I get error after '2044' comm dupes > > > > Satish > > > From bsmith at mcs.anl.gov Tue Apr 3 12:29:16 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 3 Apr 2018 17:29:16 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> Fande, The reason for MPI_Comm_dup() and the inner communicator is that this communicator is used by hypre and so cannot "just" be a PETSc communicator. We cannot have PETSc and hypre using the same communicator since they may capture each others messages etc. See my pull request that I think should resolve the issue in the short term, Barry > On Apr 3, 2018, at 11:21 AM, Kong, Fande wrote: > > Figured out: > > The reason is that in MatCreate_HYPRE(Mat B), we call MPI_Comm_dup instead of PetscCommDuplicate. The PetscCommDuplicate is better, and it does not actually create a communicator if the communicator is already known to PETSc. > > Furthermore, I do not think we should a comm in > > typedef struct { > HYPRE_IJMatrix ij; > HYPRE_IJVector x; > HYPRE_IJVector b; > MPI_Comm comm; > } Mat_HYPRE; > > It is an inner data of Mat, and it should already the same comm as the Mat. I do not understand why the internal data has its own comm. > > The following patch fixed the issue (just deleted this extra comm). > > diff --git a/src/mat/impls/hypre/mhypre.c b/src/mat/impls/hypre/mhypre.c > index dc19892..d8cfe3d 100644 > --- a/src/mat/impls/hypre/mhypre.c > +++ b/src/mat/impls/hypre/mhypre.c > @@ -74,7 +74,7 @@ static PetscErrorCode MatHYPRE_CreateFromMat(Mat A, Mat_HYPRE *hA) > rend = A->rmap->rend; > cstart = A->cmap->rstart; > cend = A->cmap->rend; > - PetscStackCallStandard(HYPRE_IJMatrixCreate,(hA->comm,rstart,rend-1,cstart,cend-1,&hA->ij)); > + PetscStackCallStandard(HYPRE_IJMatrixCreate,(PetscObjectComm((PetscObject)A),rstart,rend-1,cstart,cend-1,&hA->ij)); > PetscStackCallStandard(HYPRE_IJMatrixSetObjectType,(hA->ij,HYPRE_PARCSR)); > { > PetscBool same; > @@ -434,7 +434,7 @@ PetscErrorCode MatDestroy_HYPRE(Mat A) > if (hA->x) PetscStackCallStandard(HYPRE_IJVectorDestroy,(hA->x)); > if (hA->b) PetscStackCallStandard(HYPRE_IJVectorDestroy,(hA->b)); > if (hA->ij) PetscStackCallStandard(HYPRE_IJMatrixDestroy,(hA->ij)); > - if (hA->comm) { ierr = MPI_Comm_free(&hA->comm);CHKERRQ(ierr);} > + /*if (hA->comm) { ierr = MPI_Comm_free(&hA->comm);CHKERRQ(ierr);}*/ > ierr = PetscObjectComposeFunction((PetscObject)A,"MatConvert_hypre_aij_C",NULL);CHKERRQ(ierr); > ierr = PetscFree(A->data);CHKERRQ(ierr); > PetscFunctionReturn(0); > @@ -500,7 +500,8 @@ PETSC_EXTERN PetscErrorCode MatCreate_HYPRE(Mat B) > B->ops->destroy = MatDestroy_HYPRE; > B->ops->assemblyend = MatAssemblyEnd_HYPRE; > > - ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); > + /*ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); */ > + /*ierr = PetscCommDuplicate(PetscObjectComm((PetscObject)B),&hB->comm,NULL);CHKERRQ(ierr);*/ > ierr = PetscObjectChangeTypeName((PetscObject)B,MATHYPRE);CHKERRQ(ierr); > ierr = PetscObjectComposeFunction((PetscObject)B,"MatConvert_hypre_aij_C",MatConvert_HYPRE_AIJ);CHKERRQ(ierr); > PetscFunctionReturn(0); > diff --git a/src/mat/impls/hypre/mhypre.h b/src/mat/impls/hypre/mhypre.h > index 3d9ddd2..1189020 100644 > --- a/src/mat/impls/hypre/mhypre.h > +++ b/src/mat/impls/hypre/mhypre.h > @@ -10,7 +10,7 @@ typedef struct { > HYPRE_IJMatrix ij; > HYPRE_IJVector x; > HYPRE_IJVector b; > - MPI_Comm comm; > + /*MPI_Comm comm;*/ > } Mat_HYPRE; > > > > Fande, > > > > > On Tue, Apr 3, 2018 at 10:35 AM, Satish Balay wrote: > On Tue, 3 Apr 2018, Satish Balay wrote: > > > On Tue, 3 Apr 2018, Derek Gaston wrote: > > > > > One thing I want to be clear of here: is that we're not trying to solve > > > this particular problem (where we're creating 1000 instances of Hypre to > > > precondition each variable independently)... this particular problem is > > > just a test (that we've had in our test suite for a long time) to stress > > > test some of this capability. > > > > > > We really do have needs for thousands (tens of thousands) of simultaneous > > > solves (each with their own Hypre instances). That's not what this > > > particular problem is doing - but it is representative of a class of our > > > problems we need to solve. > > > > > > Which does bring up a point: I have been able to do solves before with > > > ~50,000 separate PETSc solves without issue. Is it because I was working > > > with MVAPICH on a cluster? Does it just have a higher limit? > > > > Don't know - but thats easy to find out with a simple test code.. > > > > >>>>>> > > $ cat comm_dup_test.c > > #include > > #include > > > > int main(int argc, char** argv) { > > MPI_Comm newcomm; > > int i, err; > > MPI_Init(NULL, NULL); > > for (i=0; i<100000; i++) { > > err = MPI_Comm_dup(MPI_COMM_WORLD, &newcomm); > > if (err) { > > printf("%5d - fail\n",i);fflush(stdout); > > break; > > } else { > > printf("%5d - success\n",i);fflush(stdout); > > } > > } > > MPI_Finalize(); > > } > > <<<<<<< > > > > OpenMPI fails after '65531' and mpich after '2044'. MVAPICH is derived > > off MPICH - but its possible they have a different limit than MPICH. > > BTW: the above is with: openmpi-2.1.2 and mpich-3.3b1 > > mvapich2-1.9.5 - and I get error after '2044' comm dupes > > Satish > From fande.kong at inl.gov Tue Apr 3 12:41:01 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Tue, 3 Apr 2018 11:41:01 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> Message-ID: On Tue, Apr 3, 2018 at 11:29 AM, Smith, Barry F. wrote: > > Fande, > > The reason for MPI_Comm_dup() and the inner communicator is that this > communicator is used by hypre and so cannot "just" be a PETSc communicator. > We cannot have PETSc and hypre using the same communicator since they may > capture each others messages etc. > > See my pull request that I think should resolve the issue in the > short term, > Yes, it helps as well. The question becomes we can not have more than 2000 AMG solvers in one application because each Hypre owns its communicator. There is no way to have all AMG solvers share the same HYPRE-sided communicator? Just like what we are dong for PETSc objects? Fande, > > Barry > > > > On Apr 3, 2018, at 11:21 AM, Kong, Fande wrote: > > > > Figured out: > > > > The reason is that in MatCreate_HYPRE(Mat B), we call MPI_Comm_dup > instead of PetscCommDuplicate. The PetscCommDuplicate is better, and it > does not actually create a communicator if the communicator is already > known to PETSc. > > > > Furthermore, I do not think we should a comm in > > > > typedef struct { > > HYPRE_IJMatrix ij; > > HYPRE_IJVector x; > > HYPRE_IJVector b; > > MPI_Comm comm; > > } Mat_HYPRE; > > > > It is an inner data of Mat, and it should already the same comm as the > Mat. I do not understand why the internal data has its own comm. > > > > The following patch fixed the issue (just deleted this extra comm). > > > > diff --git a/src/mat/impls/hypre/mhypre.c b/src/mat/impls/hypre/mhypre.c > > index dc19892..d8cfe3d 100644 > > --- a/src/mat/impls/hypre/mhypre.c > > +++ b/src/mat/impls/hypre/mhypre.c > > @@ -74,7 +74,7 @@ static PetscErrorCode MatHYPRE_CreateFromMat(Mat A, > Mat_HYPRE *hA) > > rend = A->rmap->rend; > > cstart = A->cmap->rstart; > > cend = A->cmap->rend; > > - PetscStackCallStandard(HYPRE_IJMatrixCreate,(hA->comm, > rstart,rend-1,cstart,cend-1,&hA->ij)); > > + PetscStackCallStandard(HYPRE_IJMatrixCreate,( > PetscObjectComm((PetscObject)A),rstart,rend-1,cstart,cend-1,&hA->ij)); > > PetscStackCallStandard(HYPRE_IJMatrixSetObjectType,(hA->ij, > HYPRE_PARCSR)); > > { > > PetscBool same; > > @@ -434,7 +434,7 @@ PetscErrorCode MatDestroy_HYPRE(Mat A) > > if (hA->x) PetscStackCallStandard(HYPRE_IJVectorDestroy,(hA->x)); > > if (hA->b) PetscStackCallStandard(HYPRE_IJVectorDestroy,(hA->b)); > > if (hA->ij) PetscStackCallStandard(HYPRE_IJMatrixDestroy,(hA->ij)); > > - if (hA->comm) { ierr = MPI_Comm_free(&hA->comm);CHKERRQ(ierr);} > > + /*if (hA->comm) { ierr = MPI_Comm_free(&hA->comm);CHKERRQ(ierr);}*/ > > ierr = PetscObjectComposeFunction((PetscObject)A,"MatConvert_ > hypre_aij_C",NULL);CHKERRQ(ierr); > > ierr = PetscFree(A->data);CHKERRQ(ierr); > > PetscFunctionReturn(0); > > @@ -500,7 +500,8 @@ PETSC_EXTERN PetscErrorCode MatCreate_HYPRE(Mat B) > > B->ops->destroy = MatDestroy_HYPRE; > > B->ops->assemblyend = MatAssemblyEnd_HYPRE; > > > > - ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm); > CHKERRQ(ierr); > > + /*ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); > */ > > + /*ierr = PetscCommDuplicate(PetscObjectComm((PetscObject) > B),&hB->comm,NULL);CHKERRQ(ierr);*/ > > ierr = PetscObjectChangeTypeName((PetscObject)B,MATHYPRE); > CHKERRQ(ierr); > > ierr = PetscObjectComposeFunction((PetscObject)B,"MatConvert_ > hypre_aij_C",MatConvert_HYPRE_AIJ);CHKERRQ(ierr); > > PetscFunctionReturn(0); > > diff --git a/src/mat/impls/hypre/mhypre.h b/src/mat/impls/hypre/mhypre.h > > index 3d9ddd2..1189020 100644 > > --- a/src/mat/impls/hypre/mhypre.h > > +++ b/src/mat/impls/hypre/mhypre.h > > @@ -10,7 +10,7 @@ typedef struct { > > HYPRE_IJMatrix ij; > > HYPRE_IJVector x; > > HYPRE_IJVector b; > > - MPI_Comm comm; > > + /*MPI_Comm comm;*/ > > } Mat_HYPRE; > > > > > > > > Fande, > > > > > > > > > > On Tue, Apr 3, 2018 at 10:35 AM, Satish Balay wrote: > > On Tue, 3 Apr 2018, Satish Balay wrote: > > > > > On Tue, 3 Apr 2018, Derek Gaston wrote: > > > > > > > One thing I want to be clear of here: is that we're not trying to > solve > > > > this particular problem (where we're creating 1000 instances of > Hypre to > > > > precondition each variable independently)... this particular problem > is > > > > just a test (that we've had in our test suite for a long time) to > stress > > > > test some of this capability. > > > > > > > > We really do have needs for thousands (tens of thousands) of > simultaneous > > > > solves (each with their own Hypre instances). That's not what this > > > > particular problem is doing - but it is representative of a class of > our > > > > problems we need to solve. > > > > > > > > Which does bring up a point: I have been able to do solves before > with > > > > ~50,000 separate PETSc solves without issue. Is it because I was > working > > > > with MVAPICH on a cluster? Does it just have a higher limit? > > > > > > Don't know - but thats easy to find out with a simple test code.. > > > > > > >>>>>> > > > $ cat comm_dup_test.c > > > #include > > > #include > > > > > > int main(int argc, char** argv) { > > > MPI_Comm newcomm; > > > int i, err; > > > MPI_Init(NULL, NULL); > > > for (i=0; i<100000; i++) { > > > err = MPI_Comm_dup(MPI_COMM_WORLD, &newcomm); > > > if (err) { > > > printf("%5d - fail\n",i);fflush(stdout); > > > break; > > > } else { > > > printf("%5d - success\n",i);fflush(stdout); > > > } > > > } > > > MPI_Finalize(); > > > } > > > <<<<<<< > > > > > > OpenMPI fails after '65531' and mpich after '2044'. MVAPICH is derived > > > off MPICH - but its possible they have a different limit than MPICH. > > > > BTW: the above is with: openmpi-2.1.2 and mpich-3.3b1 > > > > mvapich2-1.9.5 - and I get error after '2044' comm dupes > > > > Satish > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedmud at gmail.com Tue Apr 3 12:47:35 2018 From: friedmud at gmail.com (Derek Gaston) Date: Tue, 03 Apr 2018 17:47:35 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: On Tue, Apr 3, 2018 at 10:31 AM Satish Balay wrote: > On Tue, 3 Apr 2018, Derek Gaston wrote: > > Which does bring up a point: I have been able to do solves before with > > ~50,000 separate PETSc solves without issue. Is it because I was working > > with MVAPICH on a cluster? Does it just have a higher limit? > > Don't know - but thats easy to find out with a simple test code.. > I get 2044 using mvapich on my cluster too. The only thing I can think of as to why those massive problems work for me is that any one MPI process is not involved in more than ~2000 processors (because the communicators are split as you go down the hierarchy). At most, a single MPI process will see ~hundreds of PETSc solves but not thousands. That said: it's just because of the current nature of the solves I'm doing - it's definitely possible to have that not be the case with MOOSE. Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedmud at gmail.com Tue Apr 3 12:48:46 2018 From: friedmud at gmail.com (Derek Gaston) Date: Tue, 03 Apr 2018 17:48:46 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: Sorry, should read: "any one MPI process is not involved in more than ~2000 *communicators*" Derek On Tue, Apr 3, 2018 at 11:47 AM Derek Gaston wrote: > On Tue, Apr 3, 2018 at 10:31 AM Satish Balay wrote: > >> On Tue, 3 Apr 2018, Derek Gaston wrote: >> > Which does bring up a point: I have been able to do solves before with >> > ~50,000 separate PETSc solves without issue. Is it because I was >> working >> > with MVAPICH on a cluster? Does it just have a higher limit? >> >> Don't know - but thats easy to find out with a simple test code.. >> > > I get 2044 using mvapich on my cluster too. > > The only thing I can think of as to why those massive problems work for me > is that any one MPI process is not involved in more than ~2000 processors > (because the communicators are split as you go down the hierarchy). At > most, a single MPI process will see ~hundreds of PETSc solves but not > thousands. > > That said: it's just because of the current nature of the solves I'm doing > - it's definitely possible to have that not be the case with MOOSE. > > Derek > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Apr 3 12:52:22 2018 From: jed at jedbrown.org (Jed Brown) Date: Tue, 03 Apr 2018 11:52:22 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> Message-ID: <87lge4rxd5.fsf@jedbrown.org> Derek Gaston writes: > Sorry, should read: "any one MPI process is not involved in more than ~2000 > *communicators*" Yes, as intended. Only the ranks in a communicator's group need to know about the existence of that communicator. > Derek > > On Tue, Apr 3, 2018 at 11:47 AM Derek Gaston wrote: > >> On Tue, Apr 3, 2018 at 10:31 AM Satish Balay wrote: >> >>> On Tue, 3 Apr 2018, Derek Gaston wrote: >>> > Which does bring up a point: I have been able to do solves before with >>> > ~50,000 separate PETSc solves without issue. Is it because I was >>> working >>> > with MVAPICH on a cluster? Does it just have a higher limit? >>> >>> Don't know - but thats easy to find out with a simple test code.. >>> >> >> I get 2044 using mvapich on my cluster too. >> >> The only thing I can think of as to why those massive problems work for me >> is that any one MPI process is not involved in more than ~2000 processors >> (because the communicators are split as you go down the hierarchy). At >> most, a single MPI process will see ~hundreds of PETSc solves but not >> thousands. >> >> That said: it's just because of the current nature of the solves I'm doing >> - it's definitely possible to have that not be the case with MOOSE. >> >> Derek >> From bsmith at mcs.anl.gov Tue Apr 3 12:53:25 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 3 Apr 2018 17:53:25 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> Message-ID: <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> > On Apr 3, 2018, at 11:41 AM, Kong, Fande wrote: > > > > On Tue, Apr 3, 2018 at 11:29 AM, Smith, Barry F. wrote: > > Fande, > > The reason for MPI_Comm_dup() and the inner communicator is that this communicator is used by hypre and so cannot "just" be a PETSc communicator. We cannot have PETSc and hypre using the same communicator since they may capture each others messages etc. > > See my pull request that I think should resolve the issue in the short term, > > Yes, it helps as well. Good, we'll get it in to master after testing > > The question becomes we can not have more than 2000 AMG solvers in one application because each Hypre owns its communicator. There is no way to have all AMG solvers share the same HYPRE-sided communicator? Just like what we are dong for PETSc objects? I cannot answer this question. This we need to consult with the hypre team. Note that PETSc does one MPI_Comm_dup() for each hypre matrix. Internally hypre does at least one MPI_Comm_create() per hypre boomerAMG solver. So even if PETSc does not do the MPI_Comm_dup() you will still be limited due to hypre's MPI_Comm_create. I will compose an email to hypre cc:ing everyone to get information from them. Barry Another note: If you are creating 2 thousand subcommunicators (independent of using hypre etc) the limit on communicators is still going to hit you, so if you dream of millions of cores and thousands of sub communicators you will need to convince the MPI developers to support a larger number of communicators. > > > Fande, > > > > Barry > > > > On Apr 3, 2018, at 11:21 AM, Kong, Fande wrote: > > > > Figured out: > > > > The reason is that in MatCreate_HYPRE(Mat B), we call MPI_Comm_dup instead of PetscCommDuplicate. The PetscCommDuplicate is better, and it does not actually create a communicator if the communicator is already known to PETSc. > > > > Furthermore, I do not think we should a comm in > > > > typedef struct { > > HYPRE_IJMatrix ij; > > HYPRE_IJVector x; > > HYPRE_IJVector b; > > MPI_Comm comm; > > } Mat_HYPRE; > > > > It is an inner data of Mat, and it should already the same comm as the Mat. I do not understand why the internal data has its own comm. > > > > The following patch fixed the issue (just deleted this extra comm). > > > > diff --git a/src/mat/impls/hypre/mhypre.c b/src/mat/impls/hypre/mhypre.c > > index dc19892..d8cfe3d 100644 > > --- a/src/mat/impls/hypre/mhypre.c > > +++ b/src/mat/impls/hypre/mhypre.c > > @@ -74,7 +74,7 @@ static PetscErrorCode MatHYPRE_CreateFromMat(Mat A, Mat_HYPRE *hA) > > rend = A->rmap->rend; > > cstart = A->cmap->rstart; > > cend = A->cmap->rend; > > - PetscStackCallStandard(HYPRE_IJMatrixCreate,(hA->comm,rstart,rend-1,cstart,cend-1,&hA->ij)); > > + PetscStackCallStandard(HYPRE_IJMatrixCreate,(PetscObjectComm((PetscObject)A),rstart,rend-1,cstart,cend-1,&hA->ij)); > > PetscStackCallStandard(HYPRE_IJMatrixSetObjectType,(hA->ij,HYPRE_PARCSR)); > > { > > PetscBool same; > > @@ -434,7 +434,7 @@ PetscErrorCode MatDestroy_HYPRE(Mat A) > > if (hA->x) PetscStackCallStandard(HYPRE_IJVectorDestroy,(hA->x)); > > if (hA->b) PetscStackCallStandard(HYPRE_IJVectorDestroy,(hA->b)); > > if (hA->ij) PetscStackCallStandard(HYPRE_IJMatrixDestroy,(hA->ij)); > > - if (hA->comm) { ierr = MPI_Comm_free(&hA->comm);CHKERRQ(ierr);} > > + /*if (hA->comm) { ierr = MPI_Comm_free(&hA->comm);CHKERRQ(ierr);}*/ > > ierr = PetscObjectComposeFunction((PetscObject)A,"MatConvert_hypre_aij_C",NULL);CHKERRQ(ierr); > > ierr = PetscFree(A->data);CHKERRQ(ierr); > > PetscFunctionReturn(0); > > @@ -500,7 +500,8 @@ PETSC_EXTERN PetscErrorCode MatCreate_HYPRE(Mat B) > > B->ops->destroy = MatDestroy_HYPRE; > > B->ops->assemblyend = MatAssemblyEnd_HYPRE; > > > > - ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); > > + /*ierr = MPI_Comm_dup(PetscObjectComm((PetscObject)B),&hB->comm);CHKERRQ(ierr); */ > > + /*ierr = PetscCommDuplicate(PetscObjectComm((PetscObject)B),&hB->comm,NULL);CHKERRQ(ierr);*/ > > ierr = PetscObjectChangeTypeName((PetscObject)B,MATHYPRE);CHKERRQ(ierr); > > ierr = PetscObjectComposeFunction((PetscObject)B,"MatConvert_hypre_aij_C",MatConvert_HYPRE_AIJ);CHKERRQ(ierr); > > PetscFunctionReturn(0); > > diff --git a/src/mat/impls/hypre/mhypre.h b/src/mat/impls/hypre/mhypre.h > > index 3d9ddd2..1189020 100644 > > --- a/src/mat/impls/hypre/mhypre.h > > +++ b/src/mat/impls/hypre/mhypre.h > > @@ -10,7 +10,7 @@ typedef struct { > > HYPRE_IJMatrix ij; > > HYPRE_IJVector x; > > HYPRE_IJVector b; > > - MPI_Comm comm; > > + /*MPI_Comm comm;*/ > > } Mat_HYPRE; > > > > > > > > Fande, > > > > > > > > > > On Tue, Apr 3, 2018 at 10:35 AM, Satish Balay wrote: > > On Tue, 3 Apr 2018, Satish Balay wrote: > > > > > On Tue, 3 Apr 2018, Derek Gaston wrote: > > > > > > > One thing I want to be clear of here: is that we're not trying to solve > > > > this particular problem (where we're creating 1000 instances of Hypre to > > > > precondition each variable independently)... this particular problem is > > > > just a test (that we've had in our test suite for a long time) to stress > > > > test some of this capability. > > > > > > > > We really do have needs for thousands (tens of thousands) of simultaneous > > > > solves (each with their own Hypre instances). That's not what this > > > > particular problem is doing - but it is representative of a class of our > > > > problems we need to solve. > > > > > > > > Which does bring up a point: I have been able to do solves before with > > > > ~50,000 separate PETSc solves without issue. Is it because I was working > > > > with MVAPICH on a cluster? Does it just have a higher limit? > > > > > > Don't know - but thats easy to find out with a simple test code.. > > > > > > >>>>>> > > > $ cat comm_dup_test.c > > > #include > > > #include > > > > > > int main(int argc, char** argv) { > > > MPI_Comm newcomm; > > > int i, err; > > > MPI_Init(NULL, NULL); > > > for (i=0; i<100000; i++) { > > > err = MPI_Comm_dup(MPI_COMM_WORLD, &newcomm); > > > if (err) { > > > printf("%5d - fail\n",i);fflush(stdout); > > > break; > > > } else { > > > printf("%5d - success\n",i);fflush(stdout); > > > } > > > } > > > MPI_Finalize(); > > > } > > > <<<<<<< > > > > > > OpenMPI fails after '65531' and mpich after '2044'. MVAPICH is derived > > > off MPICH - but its possible they have a different limit than MPICH. > > > > BTW: the above is with: openmpi-2.1.2 and mpich-3.3b1 > > > > mvapich2-1.9.5 - and I get error after '2044' comm dupes > > > > Satish > > > > From balay at mcs.anl.gov Tue Apr 3 12:59:09 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 3 Apr 2018 12:59:09 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> Message-ID: On Tue, 3 Apr 2018, Smith, Barry F. wrote: > Note that PETSc does one MPI_Comm_dup() for each hypre matrix. Internally hypre does at least one MPI_Comm_create() per hypre boomerAMG solver. So even if PETSc does not do the MPI_Comm_dup() you will still be limited due to hypre's MPI_Comm_create. > > I will compose an email to hypre cc:ing everyone to get information from them. Actually I don't see any calls to MPI_Comm_dup() in hypre sources [there are stubs for it for non-mpi build] There was that call to MPI_Comm_create() in the stack trace [via hypre_BoomerAMGSetup] Satish From bsmith at mcs.anl.gov Tue Apr 3 13:23:23 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 3 Apr 2018 18:23:23 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> Message-ID: > On Apr 3, 2018, at 11:59 AM, Balay, Satish wrote: > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > >> Note that PETSc does one MPI_Comm_dup() for each hypre matrix. Internally hypre does at least one MPI_Comm_create() per hypre boomerAMG solver. So even if PETSc does not do the MPI_Comm_dup() you will still be limited due to hypre's MPI_Comm_create. >> >> I will compose an email to hypre cc:ing everyone to get information from them. > > Actually I don't see any calls to MPI_Comm_dup() in hypre sources [there are stubs for it for non-mpi build] > > There was that call to MPI_Comm_create() in the stack trace [via hypre_BoomerAMGSetup] This is what I said. The MPI_Comm_create() is called for each solver and hence uses a slot for each solver. Barry > > Satish From friedmud at gmail.com Tue Apr 3 13:48:38 2018 From: friedmud at gmail.com (Derek Gaston) Date: Tue, 03 Apr 2018 18:48:38 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> Message-ID: Do you think there is any possibility of getting Hypre to use disjoint tags from PETSc so you can just use the same comm? Maybe a configure option to Hypre to tell it what number to start at for its tags? Derek On Tue, Apr 3, 2018 at 11:59 AM Satish Balay wrote: > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > Note that PETSc does one MPI_Comm_dup() for each hypre matrix. > Internally hypre does at least one MPI_Comm_create() per hypre boomerAMG > solver. So even if PETSc does not do the MPI_Comm_dup() you will still be > limited due to hypre's MPI_Comm_create. > > > > I will compose an email to hypre cc:ing everyone to get information > from them. > > Actually I don't see any calls to MPI_Comm_dup() in hypre sources [there > are stubs for it for non-mpi build] > > There was that call to MPI_Comm_create() in the stack trace [via > hypre_BoomerAMGSetup] > > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Apr 3 14:45:00 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 3 Apr 2018 14:45:00 -0500 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> Message-ID: On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > > On Apr 3, 2018, at 11:59 AM, Balay, Satish wrote: > > > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > >> Note that PETSc does one MPI_Comm_dup() for each hypre matrix. Internally hypre does at least one MPI_Comm_create() per hypre boomerAMG solver. So even if PETSc does not do the MPI_Comm_dup() you will still be limited due to hypre's MPI_Comm_create. > >> > >> I will compose an email to hypre cc:ing everyone to get information from them. > > > > Actually I don't see any calls to MPI_Comm_dup() in hypre sources [there are stubs for it for non-mpi build] > > > > There was that call to MPI_Comm_create() in the stack trace [via hypre_BoomerAMGSetup] > > This is what I said. The MPI_Comm_create() is called for each solver and hence uses a slot for each solver. Ops sorry - misread the text.. Satish From fande.kong at inl.gov Tue Apr 3 15:56:29 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Tue, 3 Apr 2018 14:56:29 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> Message-ID: I think we could add an inner comm for external package. If the same comm is passed in again, we just retrieve the same communicator, instead of MPI_Comm_dup(), for that external package (at least HYPRE team claimed this will be fine). I did not see any issue with this idea so far. I might be missing something here Fande, On Tue, Apr 3, 2018 at 1:45 PM, Satish Balay wrote: > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > > > > > > On Apr 3, 2018, at 11:59 AM, Balay, Satish wrote: > > > > > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > > > >> Note that PETSc does one MPI_Comm_dup() for each hypre matrix. > Internally hypre does at least one MPI_Comm_create() per hypre boomerAMG > solver. So even if PETSc does not do the MPI_Comm_dup() you will still be > limited due to hypre's MPI_Comm_create. > > >> > > >> I will compose an email to hypre cc:ing everyone to get > information from them. > > > > > > Actually I don't see any calls to MPI_Comm_dup() in hypre sources > [there are stubs for it for non-mpi build] > > > > > > There was that call to MPI_Comm_create() in the stack trace [via > hypre_BoomerAMGSetup] > > > > This is what I said. The MPI_Comm_create() is called for each solver > and hence uses a slot for each solver. > > Ops sorry - misread the text.. > > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Tue Apr 3 16:04:48 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Tue, 3 Apr 2018 23:04:48 +0200 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> Message-ID: What about PetscCommGetPkgComm(MPI_Comm comm ,const char* package, MPI_Comm* pkgcomm) with a key for each of the external packages PETSc can use? > On Apr 3, 2018, at 10:56 PM, Kong, Fande wrote: > > I think we could add an inner comm for external package. If the same comm is passed in again, we just retrieve the same communicator, instead of MPI_Comm_dup(), for that external package (at least HYPRE team claimed this will be fine). I did not see any issue with this idea so far. > > I might be missing something here > > > Fande, > > On Tue, Apr 3, 2018 at 1:45 PM, Satish Balay > wrote: > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > > > > > > On Apr 3, 2018, at 11:59 AM, Balay, Satish > wrote: > > > > > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: > > > > > >> Note that PETSc does one MPI_Comm_dup() for each hypre matrix. Internally hypre does at least one MPI_Comm_create() per hypre boomerAMG solver. So even if PETSc does not do the MPI_Comm_dup() you will still be limited due to hypre's MPI_Comm_create. > > >> > > >> I will compose an email to hypre cc:ing everyone to get information from them. > > > > > > Actually I don't see any calls to MPI_Comm_dup() in hypre sources [there are stubs for it for non-mpi build] > > > > > > There was that call to MPI_Comm_create() in the stack trace [via hypre_BoomerAMGSetup] > > > > This is what I said. The MPI_Comm_create() is called for each solver and hence uses a slot for each solver. > > Ops sorry - misread the text.. > > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Tue Apr 3 16:11:09 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Tue, 3 Apr 2018 15:11:09 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> Message-ID: It looks nice for me. Fande, On Tue, Apr 3, 2018 at 3:04 PM, Stefano Zampini wrote: > What about > > PetscCommGetPkgComm(MPI_Comm comm ,const char* package, MPI_Comm* pkgcomm) > > with a key for each of the external packages PETSc can use? > > > On Apr 3, 2018, at 10:56 PM, Kong, Fande wrote: > > I think we could add an inner comm for external package. If the same comm > is passed in again, we just retrieve the same communicator, instead of > MPI_Comm_dup(), for that external package (at least HYPRE team claimed > this will be fine). I did not see any issue with this idea so far. > > I might be missing something here > > > Fande, > > On Tue, Apr 3, 2018 at 1:45 PM, Satish Balay wrote: > >> On Tue, 3 Apr 2018, Smith, Barry F. wrote: >> >> > >> > >> > > On Apr 3, 2018, at 11:59 AM, Balay, Satish wrote: >> > > >> > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: >> > > >> > >> Note that PETSc does one MPI_Comm_dup() for each hypre matrix. >> Internally hypre does at least one MPI_Comm_create() per hypre boomerAMG >> solver. So even if PETSc does not do the MPI_Comm_dup() you will still be >> limited due to hypre's MPI_Comm_create. >> > >> >> > >> I will compose an email to hypre cc:ing everyone to get >> information from them. >> > > >> > > Actually I don't see any calls to MPI_Comm_dup() in hypre sources >> [there are stubs for it for non-mpi build] >> > > >> > > There was that call to MPI_Comm_create() in the stack trace [via >> hypre_BoomerAMGSetup] >> > >> > This is what I said. The MPI_Comm_create() is called for each solver >> and hence uses a slot for each solver. >> >> Ops sorry - misread the text.. >> >> Satish >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Apr 3 16:20:26 2018 From: jed at jedbrown.org (Jed Brown) Date: Tue, 03 Apr 2018 15:20:26 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> Message-ID: <87370crnqd.fsf@jedbrown.org> Derek Gaston writes: > Do you think there is any possibility of getting Hypre to use disjoint tags > from PETSc so you can just use the same comm? Maybe a configure option to > Hypre to tell it what number to start at for its tags? Why have malloc when we could just coordinate each of our libraries to use non-overlapping memory segments??? From friedmud at gmail.com Tue Apr 3 16:56:13 2018 From: friedmud at gmail.com (Derek Gaston) Date: Tue, 03 Apr 2018 21:56:13 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: <87370crnqd.fsf@jedbrown.org> References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> <87370crnqd.fsf@jedbrown.org> Message-ID: Sounds great to me - what library do I download that we're all going to use for managing the memory pool? :-) Seriously though: why doesn't MPI give us an ability to get unique tag IDs for a given communicator? I like the way libMesh deals with this: https://github.com/libMesh/libmesh/blob/master/include/parallel/parallel_implementation.h#L1343 I would definitely sign on for all of us to use the same library for getting unique tag IDs... and then we would need a lot less communicators... Derek On Tue, Apr 3, 2018 at 3:20 PM Jed Brown wrote: > Derek Gaston writes: > > > Do you think there is any possibility of getting Hypre to use disjoint > tags > > from PETSc so you can just use the same comm? Maybe a configure option > to > > Hypre to tell it what number to start at for its tags? > > Why have malloc when we could just coordinate each of our libraries to > use non-overlapping memory segments??? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 3 16:57:51 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 3 Apr 2018 21:57:51 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> Message-ID: <7C519057-B077-452B-AA1E-983962E43D7C@anl.gov> I would only implement it for hypre since we don't know if it is safe for other packages. Best to insure the comm is freed as soon as possible (not waiting for PetscFinalize()) > On Apr 3, 2018, at 3:04 PM, Stefano Zampini wrote: > > What about > > PetscCommGetPkgComm(MPI_Comm comm ,const char* package, MPI_Comm* pkgcomm) > > with a key for each of the external packages PETSc can use? > > >> On Apr 3, 2018, at 10:56 PM, Kong, Fande wrote: >> >> I think we could add an inner comm for external package. If the same comm is passed in again, we just retrieve the same communicator, instead of MPI_Comm_dup(), for that external package (at least HYPRE team claimed this will be fine). I did not see any issue with this idea so far. >> >> I might be missing something here >> >> >> Fande, >> >> On Tue, Apr 3, 2018 at 1:45 PM, Satish Balay wrote: >> On Tue, 3 Apr 2018, Smith, Barry F. wrote: >> >> > >> > >> > > On Apr 3, 2018, at 11:59 AM, Balay, Satish wrote: >> > > >> > > On Tue, 3 Apr 2018, Smith, Barry F. wrote: >> > > >> > >> Note that PETSc does one MPI_Comm_dup() for each hypre matrix. Internally hypre does at least one MPI_Comm_create() per hypre boomerAMG solver. So even if PETSc does not do the MPI_Comm_dup() you will still be limited due to hypre's MPI_Comm_create. >> > >> >> > >> I will compose an email to hypre cc:ing everyone to get information from them. >> > > >> > > Actually I don't see any calls to MPI_Comm_dup() in hypre sources [there are stubs for it for non-mpi build] >> > > >> > > There was that call to MPI_Comm_create() in the stack trace [via hypre_BoomerAMGSetup] >> > >> > This is what I said. The MPI_Comm_create() is called for each solver and hence uses a slot for each solver. >> >> Ops sorry - misread the text.. >> >> Satish >> > From jed at jedbrown.org Tue Apr 3 17:06:57 2018 From: jed at jedbrown.org (Jed Brown) Date: Tue, 03 Apr 2018 16:06:57 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> <87370crnqd.fsf@jedbrown.org> Message-ID: <87efjwq70e.fsf@jedbrown.org> Derek Gaston writes: > Sounds great to me - what library do I download that we're all going to use > for managing the memory pool? :-) > > Seriously though: why doesn't MPI give us an ability to get unique tag IDs > for a given communicator? It's called a dup'd communicator. > I like the way libMesh deals with this: > https://github.com/libMesh/libmesh/blob/master/include/parallel/parallel_implementation.h#L1343 PETSc does something similar, but using attributes inside the MPI_Comm instead of as a wrapper that goes around the communicator. https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscCommGetNewTag.html > I would definitely sign on for all of us to use the same library for > getting unique tag IDs... and then we would need a lot less communicators... Communicators should be cheap. One per library per "size" isn't a huge number of communicators. From knepley at gmail.com Tue Apr 3 17:11:19 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 3 Apr 2018 18:11:19 -0400 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: <87efjwq70e.fsf@jedbrown.org> References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> <87370crnqd.fsf@jedbrown.org> <87efjwq70e.fsf@jedbrown.org> Message-ID: On Tue, Apr 3, 2018 at 6:06 PM, Jed Brown wrote: > Derek Gaston writes: > > > Sounds great to me - what library do I download that we're all going to > use > > for managing the memory pool? :-) > > > > Seriously though: why doesn't MPI give us an ability to get unique tag > IDs > > for a given communicator? > > It's called a dup'd communicator. > > > I like the way libMesh deals with this: > > https://github.com/libMesh/libmesh/blob/master/include/ > parallel/parallel_implementation.h#L1343 > > PETSc does something similar, but using attributes inside the MPI_Comm > instead of as a wrapper that goes around the communicator. > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/ > PetscCommGetNewTag.html > > > I would definitely sign on for all of us to use the same library for > > getting unique tag IDs... and then we would need a lot less > communicators... > > Communicators should be cheap. One per library per "size" isn't a huge > number of communicators. > And this https://experts.illinois.edu/en/publications/mpi-on-millions-of-cores still got published? I guess the reviewers never wanted any more than 2K communicators ;) Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Apr 3 17:15:59 2018 From: jed at jedbrown.org (Jed Brown) Date: Tue, 03 Apr 2018 16:15:59 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> <87370crnqd.fsf@jedbrown.org> <87efjwq70e.fsf@jedbrown.org> Message-ID: <878ta4q6lc.fsf@jedbrown.org> Matthew Knepley writes: > On Tue, Apr 3, 2018 at 6:06 PM, Jed Brown wrote: > >> Derek Gaston writes: >> >> > Sounds great to me - what library do I download that we're all going to >> use >> > for managing the memory pool? :-) >> > >> > Seriously though: why doesn't MPI give us an ability to get unique tag >> IDs >> > for a given communicator? >> >> It's called a dup'd communicator. >> >> > I like the way libMesh deals with this: >> > https://github.com/libMesh/libmesh/blob/master/include/ >> parallel/parallel_implementation.h#L1343 >> >> PETSc does something similar, but using attributes inside the MPI_Comm >> instead of as a wrapper that goes around the communicator. >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/ >> PetscCommGetNewTag.html >> >> > I would definitely sign on for all of us to use the same library for >> > getting unique tag IDs... and then we would need a lot less >> communicators... >> >> Communicators should be cheap. One per library per "size" isn't a huge >> number of communicators. >> > > And this > https://experts.illinois.edu/en/publications/mpi-on-millions-of-cores still > got published? I guess > the reviewers never wanted any more than 2K communicators ;) These are unrelated concerns. A million is only 2^20 and 20 is much less thann 2000. The issue is that communicator etiquette between libraries isn't expressed in some visible statement of best practices, perhaps even a recommendation in the MPI standard. If hypre dup'd communicators like PETSc, then we would all have less code and it would be nowhere near 2000 even in the massive MOOSE systems. From friedmud at gmail.com Tue Apr 3 17:16:55 2018 From: friedmud at gmail.com (Derek Gaston) Date: Tue, 03 Apr 2018 22:16:55 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: <87efjwq70e.fsf@jedbrown.org> References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> <87370crnqd.fsf@jedbrown.org> <87efjwq70e.fsf@jedbrown.org> Message-ID: On Tue, Apr 3, 2018 at 4:06 PM Jed Brown wrote: > Communicators should be cheap. One per library per "size" isn't a huge > number of communicators. > I agree - but that's not what we're getting here. We're getting one per "object" (Mat / Preconditioner, etc.) associated with the library per "size". If we can fix that I agree that there's no problem (we use a lot of libraries... but not 2000 separate ones simultaneously!). Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 3 17:17:19 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 3 Apr 2018 18:17:19 -0400 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: <878ta4q6lc.fsf@jedbrown.org> References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> <87370crnqd.fsf@jedbrown.org> <87efjwq70e.fsf@jedbrown.org> <878ta4q6lc.fsf@jedbrown.org> Message-ID: On Tue, Apr 3, 2018 at 6:15 PM, Jed Brown wrote: > Matthew Knepley writes: > > > On Tue, Apr 3, 2018 at 6:06 PM, Jed Brown wrote: > > > >> Derek Gaston writes: > >> > >> > Sounds great to me - what library do I download that we're all going > to > >> use > >> > for managing the memory pool? :-) > >> > > >> > Seriously though: why doesn't MPI give us an ability to get unique tag > >> IDs > >> > for a given communicator? > >> > >> It's called a dup'd communicator. > >> > >> > I like the way libMesh deals with this: > >> > https://github.com/libMesh/libmesh/blob/master/include/ > >> parallel/parallel_implementation.h#L1343 > >> > >> PETSc does something similar, but using attributes inside the MPI_Comm > >> instead of as a wrapper that goes around the communicator. > >> > >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/ > >> PetscCommGetNewTag.html > >> > >> > I would definitely sign on for all of us to use the same library for > >> > getting unique tag IDs... and then we would need a lot less > >> communicators... > >> > >> Communicators should be cheap. One per library per "size" isn't a huge > >> number of communicators. > >> > > > > And this > > https://experts.illinois.edu/en/publications/mpi-on-millions-of-cores > still > > got published? I guess > > the reviewers never wanted any more than 2K communicators ;) > > These are unrelated concerns. A million is only 2^20 and 20 is much > less thann 2000. The issue is that communicator etiquette between > libraries isn't expressed in some visible statement of best practices, > perhaps even a recommendation in the MPI standard. If hypre dup'd > communicators like PETSc, then we would all have less code and it would > be nowhere near 2000 even in the massive MOOSE systems. > I understand. I was pointing out that even Bill went for the obvious target, rather than the usability standard. And if they agreed with the last line, they should have pointed it out. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Apr 3 17:20:17 2018 From: jed at jedbrown.org (Jed Brown) Date: Tue, 03 Apr 2018 16:20:17 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> <87370crnqd.fsf@jedbrown.org> <87efjwq70e.fsf@jedbrown.org> Message-ID: <871sfwq6e6.fsf@jedbrown.org> Derek Gaston writes: > On Tue, Apr 3, 2018 at 4:06 PM Jed Brown wrote: > >> Communicators should be cheap. One per library per "size" isn't a huge >> number of communicators. >> > > I agree - but that's not what we're getting here. We're getting one per > "object" (Mat / Preconditioner, etc.) associated with the library per > "size". If we can fix that I agree that there's no problem (we use a lot > of libraries... but not 2000 separate ones simultaneously!). So PETSc needs to dup and attach a hypre communicator because they aren't interested in doing it themselves. Not hard to implement, just mildly annoying. From knepley at gmail.com Tue Apr 3 17:22:11 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 3 Apr 2018 18:22:11 -0400 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: <871sfwq6e6.fsf@jedbrown.org> References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> <87370crnqd.fsf@jedbrown.org> <87efjwq70e.fsf@jedbrown.org> <871sfwq6e6.fsf@jedbrown.org> Message-ID: On Tue, Apr 3, 2018 at 6:20 PM, Jed Brown wrote: > Derek Gaston writes: > > > On Tue, Apr 3, 2018 at 4:06 PM Jed Brown wrote: > > > >> Communicators should be cheap. One per library per "size" isn't a huge > >> number of communicators. > >> > > > > I agree - but that's not what we're getting here. We're getting one per > > "object" (Mat / Preconditioner, etc.) associated with the library per > > "size". If we can fix that I agree that there's no problem (we use a lot > > of libraries... but not 2000 separate ones simultaneously!). > > So PETSc needs to dup and attach a hypre communicator because they > aren't interested in doing it themselves. Not hard to implement, just > mildly annoying. > Can't someone tell them its an xSDK requirement? Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 3 17:27:38 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 3 Apr 2018 22:27:38 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> <87370crnqd.fsf@jedbrown.org> <87efjwq70e.fsf@jedbrown.org> <871sfwq6e6.fsf@jedbrown.org> Message-ID: <56907EE4-5FD6-4448-8FBC-A6500AAAA642@anl.gov> My mistake, xSDK only says you need to use the given communicators; it doesn't say you need to use a duped version. M3. Each xSDK-compatible package that utilizes MPI must restrict its MPI operations to MPI communicators that are provided to it and not use directly MPI_COMM_WORLD. The package should use configure tests or version tests to detect MPI 2 or MPI 3 features that may not be available; it should not be assumed that a full MPI 2 or MPI 3 implementation is available. The package can change the MPI error-handling mode by default but should have an option to prevent it from changing the MPI error handling (which may have been set by another package or the application). The package should also behave appropriately regardless of the MPI error handling being used. There is no requirement that the package provide a sequential (non-MPI) version, although this functionality is welcome, too. If the package provides a sequential version, there is no requirement that it be compatible or usable with other xSDK-compliant packages running without MPI. Barry > On Apr 3, 2018, at 4:22 PM, Matthew Knepley wrote: > > On Tue, Apr 3, 2018 at 6:20 PM, Jed Brown wrote: > Derek Gaston writes: > > > On Tue, Apr 3, 2018 at 4:06 PM Jed Brown wrote: > > > >> Communicators should be cheap. One per library per "size" isn't a huge > >> number of communicators. > >> > > > > I agree - but that's not what we're getting here. We're getting one per > > "object" (Mat / Preconditioner, etc.) associated with the library per > > "size". If we can fix that I agree that there's no problem (we use a lot > > of libraries... but not 2000 separate ones simultaneously!). > > So PETSc needs to dup and attach a hypre communicator because they > aren't interested in doing it themselves. Not hard to implement, just > mildly annoying. > > Can't someone tell them its an xSDK requirement? > > Matt > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From jed at jedbrown.org Tue Apr 3 17:28:44 2018 From: jed at jedbrown.org (Jed Brown) Date: Tue, 03 Apr 2018 16:28:44 -0600 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> <87370crnqd.fsf@jedbrown.org> <87efjwq70e.fsf@jedbrown.org> <871sfwq6e6.fsf@jedbrown.org> Message-ID: <87y3i3q603.fsf@jedbrown.org> Matthew Knepley writes: > On Tue, Apr 3, 2018 at 6:20 PM, Jed Brown wrote: > >> Derek Gaston writes: >> >> > On Tue, Apr 3, 2018 at 4:06 PM Jed Brown wrote: >> > >> >> Communicators should be cheap. One per library per "size" isn't a huge >> >> number of communicators. >> >> >> > >> > I agree - but that's not what we're getting here. We're getting one per >> > "object" (Mat / Preconditioner, etc.) associated with the library per >> > "size". If we can fix that I agree that there's no problem (we use a lot >> > of libraries... but not 2000 separate ones simultaneously!). >> >> So PETSc needs to dup and attach a hypre communicator because they >> aren't interested in doing it themselves. Not hard to implement, just >> mildly annoying. >> > > Can't someone tell them its an xSDK requirement? xSDX is busy with threads... ;-) From friedmud at gmail.com Tue Apr 3 17:37:10 2018 From: friedmud at gmail.com (Derek Gaston) Date: Tue, 03 Apr 2018 22:37:10 +0000 Subject: [petsc-users] A bad commit affects MOOSE In-Reply-To: References: <63C431BC-B701-48E8-A7F3-C8A294257A00@anl.gov> <072D98D5-75F4-4526-A726-4DC618BC26D0@gmail.com> <46C8B2BF-3D24-47C4-8F39-6CC1BF51429F@anl.gov> <8EC9030D-D8CA-4D36-B955-FCD6D10556A2@mcs.anl.gov> <87370crnqd.fsf@jedbrown.org> <87efjwq70e.fsf@jedbrown.org> <871sfwq6e6.fsf@jedbrown.org> Message-ID: It's interesting... I'm now thinking about MOOSE. MOOSE doesn't dup the comm you give it either. It simply takes it and uses it. It assumes you've already done the duping. I like it that way because the calling code can do whatever it needs to do to get that communicator (for instance, split it). Duping it on the way in would be unnecessary in many circumstances where MOOSE is getting passed a "clean" comm (for instance, from a split that is meant only for that instance of MOOSE). It's up to the caller to make sure to pass in a clean comm for MOOSE to use. So: I don't fault Hypre... it seems to work under the same assumption. I can see both sides. At the same time, I can see the PETSc model because it simplifies things for the calling library - you don't have to worry about passing in a "clean" comm... PETSc generates its own internally. I think both models are valid - and there are tradeoffs both ways. We just need to do the "annoying" thing and keep track of a clean comm that we're passing in to Hypre. When MOOSE is running sub-solves of itself the calling instance of MOOSE creates and keeps track of those clean comms for each sub-solve... Derek On Tue, Apr 3, 2018 at 4:22 PM Matthew Knepley wrote: > On Tue, Apr 3, 2018 at 6:20 PM, Jed Brown wrote: > >> Derek Gaston writes: >> >> > On Tue, Apr 3, 2018 at 4:06 PM Jed Brown wrote: >> > >> >> Communicators should be cheap. One per library per "size" isn't a huge >> >> number of communicators. >> >> >> > >> > I agree - but that's not what we're getting here. We're getting one per >> > "object" (Mat / Preconditioner, etc.) associated with the library per >> > "size". If we can fix that I agree that there's no problem (we use a >> lot >> > of libraries... but not 2000 separate ones simultaneously!). >> >> So PETSc needs to dup and attach a hypre communicator because they >> aren't interested in doing it themselves. Not hard to implement, just >> mildly annoying. >> > > Can't someone tell them its an xSDK requirement? > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Tue Apr 3 20:24:10 2018 From: epscodes at gmail.com (Xiangdong) Date: Tue, 3 Apr 2018 21:24:10 -0400 Subject: [petsc-users] using hypre's GPU feature through PETSc Message-ID: Hello everyone, The latest Hypre has GPU capability. Can I use hypre GPU feature through PETSc? If it is possible, what configuration changes should I make in PETSc? Thank you. Best, Xiangdong -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Wed Apr 4 04:11:17 2018 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 4 Apr 2018 17:11:17 +0800 Subject: [petsc-users] Obtaining compiling and building information from a.out In-Reply-To: <54708017-a3da-34ac-b2df-19f1c52ae46d@gmail.com> References: <87f7f663-3184-0b62-e092-ae6628729883@gmail.com> <54708017-a3da-34ac-b2df-19f1c52ae46d@gmail.com> Message-ID: <04e30b6d-080c-a8c4-b704-d517e3ef8f91@gmail.com> Hi, I emailed as shown below a while ago but I have yet to get a response. Can someone help? Thank you very much. Yours sincerely, ================================================ TAY Wee-Beng (Zheng Weiming) ??? Personal research webpage: http://tayweebeng.wixsite.com/website Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA linkedin: www.linkedin.com/in/tay-weebeng ================================================ On 29/3/2018 1:48 PM, TAY wee-beng wrote: > > On 28/3/2018 5:45 PM, Matthew Knepley wrote: >> On Tue, Mar 27, 2018 at 10:17 PM, TAY wee-beng > > wrote: >> >> Hi Dave, >> >> I looked at the output using log_view and re-compile. However, >> although I use the same options "-xHost -g -O3 -openmp" (some >> filenames and dir names are now different but they are actually >> the same), I still get different results timing. I have attach >> both, fast and slow. So what else can I do to pinpoint the >> differences? >> >> The solver options must be different. In Fast, there is almost no >> time in LUFactor, but in Slow its half the time. >> >> ? ?Matt > Hi Matt, > > Finally found that it was because I was using KSPRICHARDSON in the new > code. KSPGMRES in the old code is much faster. > > I also tried cg, lsqr, fgmres. CG and GMRES seem similar in term of > best performance. Are there any other recommendations to try? The > Poisson eqn is symmetric. > > I also tried -pc_type mg -pc_mg_nlevels 2 instead of -poisson_pc_type > gamg -poisson_pc_gamg_agg_nsmooths 1. > > It reduces the runtime from 2.25 to 1.46min. > > However, I found that my subroutine is very similar to my momentum > subroutine, which is based on KSPSolve: > > 0. DMDACreate, DMDACreate3d etc > 1. Assemble matrix > 2. KSPSetOperators, KSPSetType etc > 3. KSPSolve > > However, the 2D Poisson example in ex5f.F90 uses SNESSetDM, SNESSolve > etc. So does it matter if I use SNESSolve or KSPSolve if I'm using > -pc_type mg? > >> Thank you very much. >> >> Yours sincerely, >> >> ================================================ >> TAY Wee-Beng (Zheng Weiming) ??? >> Personal research webpage:http://tayweebeng.wixsite.com/website >> >> Youtube research showcase:https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >> >> linkedin:www.linkedin.com/in/tay-weebeng >> >> ================================================ >> >> On 27/3/2018 5:22 PM, Dave May wrote: >>> >>> >>> On 27 March 2018 at 10:16, TAY wee-beng >> > wrote: >>> >>> Hi, >>> >>> I have been compiling and building different version of my >>> CFD with the intel 2016, 2018 compilers, and also different >>> compiling options. >>> >>> I tested a version of my a.out and it is much faster than >>> the other a.out, using only 3 min instead of more than 10min >>> to solve a certain case using GAMG. >>> >>> However, I can't recall how it was compiled. I only know >>> that I used the intel 2016 compiler. >>> >>> So is there any way I can find out how the a.out was >>> compiled? Like what options were used? >>> >>> >>> Since you posted to the list I presume "a.out" links against >>> petsc... >>> If so, run your code with >>> ? -log_view >>> >>> Upon calling PetscFinalize(), you will get all the options given >>> to PETSc configure, plus the CFLAGS, link lines, etc. >>> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 4 06:55:51 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 4 Apr 2018 07:55:51 -0400 Subject: [petsc-users] Obtaining compiling and building information from a.out In-Reply-To: <54708017-a3da-34ac-b2df-19f1c52ae46d@gmail.com> References: <87f7f663-3184-0b62-e092-ae6628729883@gmail.com> <54708017-a3da-34ac-b2df-19f1c52ae46d@gmail.com> Message-ID: On Thu, Mar 29, 2018 at 1:48 AM, TAY wee-beng wrote: > > On 28/3/2018 5:45 PM, Matthew Knepley wrote: > > On Tue, Mar 27, 2018 at 10:17 PM, TAY wee-beng wrote: > >> Hi Dave, >> >> I looked at the output using log_view and re-compile. However, although I >> use the same options "-xHost -g -O3 -openmp" (some filenames and dir names >> are now different but they are actually the same), I still get different >> results timing. I have attach both, fast and slow. So what else can I do to >> pinpoint the differences? >> > The solver options must be different. In Fast, there is almost no time in > LUFactor, but in Slow its half the time. > > Matt > > Hi Matt, > > Finally found that it was because I was using KSPRICHARDSON in the new > code. KSPGMRES in the old code is much faster. > > I also tried cg, lsqr, fgmres. CG and GMRES seem similar in term of best > performance. Are there any other recommendations to try? The Poisson eqn is > symmetric. > Krylov solvers fundamentally do not matter for solving elliptic problems. It comes down to your preconditioner. > I also tried -pc_type mg -pc_mg_nlevels 2 instead of -poisson_pc_type gamg > -poisson_pc_gamg_agg_nsmooths 1. > > It reduces the runtime from 2.25 to 1.46min. > Yes, GMG has faster setup time than AMG. The solve time should be similar. > However, I found that my subroutine is very similar to my momentum > subroutine, which is based on KSPSolve: > > 0. DMDACreate, DMDACreate3d etc > 1. Assemble matrix > 2. KSPSetOperators, KSPSetType etc > 3. KSPSolve > > However, the 2D Poisson example in ex5f.F90 uses SNESSetDM, SNESSolve etc. > So does it matter if I use SNESSolve or KSPSolve if I'm using -pc_type mg? > Not really. Thanks, Matt > Thank you very much. >> >> Yours sincerely, >> >> ================================================ >> TAY Wee-Beng (Zheng Weiming) ??? >> Personal research webpage: http://tayweebeng.wixsite.com/website >> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >> linkedin: www.linkedin.com/in/tay-weebeng >> ================================================ >> >> On 27/3/2018 5:22 PM, Dave May wrote: >> >> >> >> On 27 March 2018 at 10:16, TAY wee-beng wrote: >> >>> Hi, >>> >>> I have been compiling and building different version of my CFD with the >>> intel 2016, 2018 compilers, and also different compiling options. >>> >>> I tested a version of my a.out and it is much faster than the other >>> a.out, using only 3 min instead of more than 10min to solve a certain case >>> using GAMG. >>> >>> However, I can't recall how it was compiled. I only know that I used the >>> intel 2016 compiler. >>> >>> So is there any way I can find out how the a.out was compiled? Like what >>> options were used? >> >> >> Since you posted to the list I presume "a.out" links against petsc... >> If so, run your code with >> -log_view >> >> Upon calling PetscFinalize(), you will get all the options given to PETSc >> configure, plus the CFLAGS, link lines, etc. >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tisaac at cc.gatech.edu Wed Apr 4 08:18:42 2018 From: tisaac at cc.gatech.edu (Tobin Isaac) Date: Wed, 4 Apr 2018 09:18:42 -0400 Subject: [petsc-users] DMForestTransferVec with -petscspace_order 0 In-Reply-To: References: <91696bcc-a34a-1142-8322-817bc497263b@univ-amu.fr> <20180403013341.6kewwzmm7evmoytf@gatech.edu> Message-ID: <20180404131842.7ofcr7k6scngrjsl@gatech.edu> Hi Yann, On Tue, Apr 03, 2018 at 05:29:39PM +0200, Yann Jobic wrote: > Hi, > > Thanks for the fast answer ! And sorry for my late one... > As a test, i'm using ex2.c in test/forest directory. > > I'm using 2 git depot (master), one from this week end, and one of today. I > see some different behaviors. > > For the git from the week end : v3.8.4-1-g756c7f9 > In 2D, everything is ok (petscspace_order = 0,1,2) : mpiexec -n 1 ./ex2 > -petscspace_poly_tensor -petscspace_order 2 -dim 2 > In 3D, only -petscspace_order 2 works. For the other ones, i put the full > message in a log file. This looks like a version of petsc/maint commit. I have not tried to fix this (a new maintenance release is imminent). > > For the git from today : v3.8.4-2420-g8f4cb0b > In 2D, petscspace_order is ok for 2 and 1, but do not work for 0. > In 3D, same as 2D, it's working for petscspace_order 2 and 1, but not for 0. > (see log file) > Many thanks for the help ! I have fixed this in the branch `tisaac/fix-height-ds` on the repo. It does not require any interface changes, so as soon as it is approved, it will be integrated into the master branch and to-be-released maintained version v3.8.9. Cheers, Toby > > Regards, > > Yann > > > Le 03/04/2018 ? 03:33, Tobin Isaac a ?crit?: > > Hi Yann, > > > > Thanks for pointing this out to us. Matt and I are the two most > > actively developing in this area. We have been working on separate > > threads and this looks like an issue where we need to sync up. I > > think there is a simple fix, but it would be helpful to know which > > version petsc you're working from: I'm seeing different > > behavior. Could you please send along more complete output? > > > > Cheers, > > Toby > > > > On Mon, Apr 02, 2018 at 04:42:29PM +0200, yann JOBIC wrote: > > > Hi, > > > > > > I'm using DMForestTransferVec, as in "dm/impls/forest/examples/tests/ex2.c". > > > > > > I would like to use it with a space approximation order at zero > > > (-petscspace_order 0). However, in this case, it's not working (valgrind > > > output of ex2.c from forest test) : > > > > > > ==8604== Conditional jump or move depends on uninitialised value(s) > > > ==8604==??? at 0x47D74F: DMPlexVecGetClosure (plex.c:4035) > > > ==8604==??? by 0x557612: DMPlexLocatePoint_General_3D_Internal > > > (plexgeometry.c:153) > > > ==8604==??? by 0x559B85: DMPlexLocatePoint_Internal (plexgeometry.c:383) > > > ==8604==??? by 0x611EED: DMPlexComputeInjectorReferenceTree > > > (plextree.c:3247) > > > ==8604==??? by 0x6148DB: DMPlexReferenceTreeGetInjector (plextree.c:3454) > > > ==8604==??? by 0x61EAD8: DMPlexTransferVecTree_Inject (plextree.c:4319) > > > ==8604==??? by 0x620CC1: DMPlexTransferVecTree (plextree.c:4527) > > > ==8604==??? by 0x7F23D8: DMForestTransferVec_p8est (pforest.c:4239) > > > ==8604==??? by 0x429B48: DMForestTransferVec (forest.c:985) > > > ==8604==??? by 0x40BB8A: main (transf_test.c:136) > > > > > > With the error message : > > > > > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > > > probably memory access out of range > > > > > > It's working fine in 2D (p4est). The problem arise in 3D (p8est). > > > > > > Is it an expected behavior ? Am i doing something wrong ? > > > > > > Thanks in advance, > > > > > > Yann > > > > > -- > ___________________________ > > Yann JOBIC > HPC engineer > IUSTI-CNRS UMR 7343 - Polytech Marseille > Technop?le de Ch?teau Gombert > 5 rue Enrico Fermi > 13453 Marseille cedex 13 > Tel : (33) 4 91 10 69 43 > Fax : (33) 4 91 10 69 69 > > > > --- > L'absence de virus dans ce courrier ?lectronique a ?t? v?rifi?e par le logiciel antivirus Avast. > https://www.avast.com/antivirus > yann at crabe:~/projet/AMR/moulinette$ mpiexec -n 1 ./ex2 -petscspace_poly_tensor -petscspace_order 0 -dim 3 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] DMPlexVecGetClosure line 4020 /home/yann/src/petsc-git/src/dm/impls/plex/plex.c > [0]PETSC ERROR: [0] DMPlexLocatePoint_General_3D_Internal line 150 /home/yann/src/petsc-git/src/dm/impls/plex/plexgeometry.c > [0]PETSC ERROR: [0] DMPlexLocatePoint_Internal line 361 /home/yann/src/petsc-git/src/dm/impls/plex/plexgeometry.c > [0]PETSC ERROR: [0] DMPlexComputeInjectorReferenceTree line 3008 /home/yann/src/petsc-git/src/dm/impls/plex/plextree.c > [0]PETSC ERROR: [0] DMPlexReferenceTreeGetInjector line 3449 /home/yann/src/petsc-git/src/dm/impls/plex/plextree.c > [0]PETSC ERROR: [0] DMPlexTransferVecTree_Inject line 4309 /home/yann/src/petsc-git/src/dm/impls/plex/plextree.c > [0]PETSC ERROR: [0] DMPlexTransferVecTree line 4486 /home/yann/src/petsc-git/src/dm/impls/plex/plextree.c > [0]PETSC ERROR: [0] DMForestTransferVec_p8est line 4213 /home/yann/src/petsc-git/src/dm/impls/forest/p4est/pforest.c > [0]PETSC ERROR: [0] DMForestTransferVec line 978 /home/yann/src/petsc-git/src/dm/impls/forest/forest.c > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.8.4, unknown > [0]PETSC ERROR: ./ex2 on a depot-git named crabe by yann Tue Apr 3 17:18:02 2018 > [0]PETSC ERROR: Configure options --prefix=/local/lib/petsc/depot-git/gcc/openmpi_gcc_all --with-single-library=0 --with-debugging=1 --download-scalapack=1 --download-metis=1 --download-parmetis=1 --download-ptscotch=1 --download-mumps=1 --download-hypre=1 --download-superlu=1 --download-superlu_dist=1 --download-fblaslapack=1 --download-metis=1 --download-cmake=1 --download-ml=1 --download-p4est=1 --download-netcdf=1 --download-pragmatic=1 --download-eigen=1 --download-parms=1 --download-triangle=1 --download-hdf5=1 --with-zlib=1 --download-szlib=1 --download-chaco=1 --download-spai=1 --download-suitesparse=1 --with-shared-libraries=0 PETSC_ARCH=depot-git > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > ------------------------------------------------------- > Primary job terminated normally, but 1 process returned > a non-zero exit code.. Per user-direction, the job has been aborted. > ------------------------------------------------------- > -------------------------------------------------------------------------- > mpiexec detected that one or more processes exited with non-zero status, thus causing > the job to be terminated. The first process to do so was: > > Process name: [[36533,1],0] > Exit code: 59 > -------------------------------------------------------------------------- > jobic at stargate:~/projet/AMR/moulinette$ mpirun -np 1 ./ex2 -petscspace_poly_tensor -dim 3 -petscspace_order 0 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Null Object: Parameter # 2 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2420-g8f4cb0b GIT Date: 2018-04-02 16:00:52 -0500 > [0]PETSC ERROR: ./ex2 on a named stargate by jobic Tue Apr 3 17:22:04 2018 > [0]PETSC ERROR: Configure options --download-chaco=1 --download-cmake=1 --download-eigen=1 --download-fblaslapack=1 --download-hdf5=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mumps=1 --download-netcdf=1 --download-p4est=1 --download-parmetis=1 --download-parms=1 --download-pragmatic=1 --download-ptscotch=1 --download-scalapack=1 --download-spai=1 --download-suitesparse=1 --download-superlu=1 --download-superlu_dist=1 --download-szlib=1 --download-triangle=1 --prefix=/local/lib/petsc/git/gcc/openmpi_gcc_all --with-debugging=1 --with-shared-libraries=0 --with-single-library=0 --with-zlib=1 PETSC_ARCH=openmpi_gcc_all > [0]PETSC ERROR: #1 PetscFESetDualSpace() line 3615 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtfe.c > [0]PETSC ERROR: #2 PetscFEGetHeightSubspace() line 6630 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtfe.c > [0]PETSC ERROR: #3 PetscDSGetHeightSubspace() line 2880 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtds.c > [0]PETSC ERROR: #4 DMProjectLocal_Generic_Plex() line 325 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/plex/plexproject.c > [0]PETSC ERROR: #5 DMProjectFunctionLocal_Plex() line 428 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/plex/plexproject.c > [0]PETSC ERROR: #6 DMProjectFunctionLocal() line 6265 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #7 DMProjectFunctionLocal_p8est() line 4385 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/forest/p4est/pforest.c > [0]PETSC ERROR: #8 DMProjectFunctionLocal() line 6265 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #9 DMProjectFunction() line 6250 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #10 main() line 161 in /home/jobic/projet/AMR/moulinette/ex2.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -dim 3 > [0]PETSC ERROR: -petscspace_order 0 > [0]PETSC ERROR: -petscspace_poly_tensor > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 85. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > > > > > > With valgrind : > jobic at stargate:~/projet/AMR/moulinette$ valgrind mpirun -np 1 ./ex2 -petscspace_poly_tensor -dim 3 -petscspace_order 0 > ==9730== Memcheck, a memory error detector > ==9730== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. > ==9730== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info > ==9730== Command: mpirun -np 1 ./ex2 -petscspace_poly_tensor -dim 3 -petscspace_order 0 > ==9730== > ==10212== Warning: invalid file descriptor 1024 in syscall close() > ==10212== Warning: invalid file descriptor 1025 in syscall close() > ==10212== Warning: invalid file descriptor 1026 in syscall close() > ==10212== Warning: invalid file descriptor 1027 in syscall close() > ==10212== Use --log-fd= to select an alternative log fd. > ==10212== Warning: invalid file descriptor 1028 in syscall close() > ==10212== Warning: invalid file descriptor 1029 in syscall close() > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Null Object: Parameter # 2 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2420-g8f4cb0b GIT Date: 2018-04-02 16:00:52 -0500 > [0]PETSC ERROR: ./ex2 on a named stargate by jobic Tue Apr 3 17:27:58 2018 > [0]PETSC ERROR: Configure options --download-chaco=1 --download-cmake=1 --download-eigen=1 --download-fblaslapack=1 --download-hdf5=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mumps=1 --download-netcdf=1 --download-p4est=1 --download-parmetis=1 --download-parms=1 --download-pragmatic=1 --download-ptscotch=1 --download-scalapack=1 --download-spai=1 --download-suitesparse=1 --download-superlu=1 --download-superlu_dist=1 --download-szlib=1 --download-triangle=1 --prefix=/local/lib/petsc/git/gcc/openmpi_gcc_all --with-debugging=1 --with-shared-libraries=0 --with-single-library=0 --with-zlib=1 PETSC_ARCH=openmpi_gcc_all > [0]PETSC ERROR: #1 PetscFESetDualSpace() line 3615 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtfe.c > [0]PETSC ERROR: #2 PetscFEGetHeightSubspace() line 6630 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtfe.c > [0]PETSC ERROR: #3 PetscDSGetHeightSubspace() line 2880 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtds.c > [0]PETSC ERROR: #4 DMProjectLocal_Generic_Plex() line 325 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/plex/plexproject.c > [0]PETSC ERROR: #5 DMProjectFunctionLocal_Plex() line 428 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/plex/plexproject.c > [0]PETSC ERROR: #6 DMProjectFunctionLocal() line 6265 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #7 DMProjectFunctionLocal_p8est() line 4385 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/forest/p4est/pforest.c > [0]PETSC ERROR: #8 DMProjectFunctionLocal() line 6265 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #9 DMProjectFunction() line 6250 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #10 main() line 161 in /home/jobic/projet/AMR/moulinette/ex2.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -dim 3 > [0]PETSC ERROR: -petscspace_order 0 > [0]PETSC ERROR: -petscspace_poly_tensor > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 85. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > ==9730== > ==9730== HEAP SUMMARY: > ==9730== in use at exit: 55,755 bytes in 116 blocks > ==9730== total heap usage: 19,779 allocs, 19,663 frees, 6,633,193 bytes allocated > ==9730== > ==9730== LEAK SUMMARY: > ==9730== definitely lost: 2,675 bytes in 11 blocks > ==9730== indirectly lost: 48,765 bytes in 75 blocks > ==9730== possibly lost: 200 bytes in 1 blocks > ==9730== still reachable: 4,115 bytes in 29 blocks > ==9730== suppressed: 0 bytes in 0 blocks > ==9730== Rerun with --leak-check=full to see details of leaked memory > ==9730== > ==9730== For counts of detected and suppressed errors, rerun with: -v > ==9730== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) From yann.jobic at univ-amu.fr Wed Apr 4 10:31:30 2018 From: yann.jobic at univ-amu.fr (Yann Jobic) Date: Wed, 4 Apr 2018 17:31:30 +0200 Subject: [petsc-users] DMForestTransferVec with -petscspace_order 0 In-Reply-To: <20180404131842.7ofcr7k6scngrjsl@gatech.edu> References: <91696bcc-a34a-1142-8322-817bc497263b@univ-amu.fr> <20180403013341.6kewwzmm7evmoytf@gatech.edu> <20180404131842.7ofcr7k6scngrjsl@gatech.edu> Message-ID: Hi Isaac, It's working !! Many thanks ! Regards, Yann Le 04/04/2018 ? 15:18, Tobin Isaac a ?crit?: > Hi Yann, > > On Tue, Apr 03, 2018 at 05:29:39PM +0200, Yann Jobic wrote: >> Hi, >> >> Thanks for the fast answer ! And sorry for my late one... >> As a test, i'm using ex2.c in test/forest directory. >> >> I'm using 2 git depot (master), one from this week end, and one of today. I >> see some different behaviors. >> >> For the git from the week end : v3.8.4-1-g756c7f9 >> In 2D, everything is ok (petscspace_order = 0,1,2) : mpiexec -n 1 ./ex2 >> -petscspace_poly_tensor -petscspace_order 2 -dim 2 >> In 3D, only -petscspace_order 2 works. For the other ones, i put the full >> message in a log file. > This looks like a version of petsc/maint commit. I have not tried to > fix this (a new maintenance release is imminent). > >> For the git from today : v3.8.4-2420-g8f4cb0b >> In 2D, petscspace_order is ok for 2 and 1, but do not work for 0. >> In 3D, same as 2D, it's working for petscspace_order 2 and 1, but not for 0. >> (see log file) >> Many thanks for the help ! > I have fixed this in the branch `tisaac/fix-height-ds` on the repo. > It does not require any interface changes, so as soon as it is > approved, it will be integrated into the master branch and > to-be-released maintained version v3.8.9. > > Cheers, > Toby > >> Regards, >> >> Yann >> >> >> Le 03/04/2018 ? 03:33, Tobin Isaac a ?crit?: >>> Hi Yann, >>> >>> Thanks for pointing this out to us. Matt and I are the two most >>> actively developing in this area. We have been working on separate >>> threads and this looks like an issue where we need to sync up. I >>> think there is a simple fix, but it would be helpful to know which >>> version petsc you're working from: I'm seeing different >>> behavior. Could you please send along more complete output? >>> >>> Cheers, >>> Toby >>> >>> On Mon, Apr 02, 2018 at 04:42:29PM +0200, yann JOBIC wrote: >>>> Hi, >>>> >>>> I'm using DMForestTransferVec, as in "dm/impls/forest/examples/tests/ex2.c". >>>> >>>> I would like to use it with a space approximation order at zero >>>> (-petscspace_order 0). However, in this case, it's not working (valgrind >>>> output of ex2.c from forest test) : >>>> >>>> ==8604== Conditional jump or move depends on uninitialised value(s) >>>> ==8604==??? at 0x47D74F: DMPlexVecGetClosure (plex.c:4035) >>>> ==8604==??? by 0x557612: DMPlexLocatePoint_General_3D_Internal >>>> (plexgeometry.c:153) >>>> ==8604==??? by 0x559B85: DMPlexLocatePoint_Internal (plexgeometry.c:383) >>>> ==8604==??? by 0x611EED: DMPlexComputeInjectorReferenceTree >>>> (plextree.c:3247) >>>> ==8604==??? by 0x6148DB: DMPlexReferenceTreeGetInjector (plextree.c:3454) >>>> ==8604==??? by 0x61EAD8: DMPlexTransferVecTree_Inject (plextree.c:4319) >>>> ==8604==??? by 0x620CC1: DMPlexTransferVecTree (plextree.c:4527) >>>> ==8604==??? by 0x7F23D8: DMForestTransferVec_p8est (pforest.c:4239) >>>> ==8604==??? by 0x429B48: DMForestTransferVec (forest.c:985) >>>> ==8604==??? by 0x40BB8A: main (transf_test.c:136) >>>> >>>> With the error message : >>>> >>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>>> probably memory access out of range >>>> >>>> It's working fine in 2D (p4est). The problem arise in 3D (p8est). >>>> >>>> Is it an expected behavior ? Am i doing something wrong ? >>>> >>>> Thanks in advance, >>>> >>>> Yann >>>> >> -- >> ___________________________ >> >> Yann JOBIC >> HPC engineer >> IUSTI-CNRS UMR 7343 - Polytech Marseille >> Technop?le de Ch?teau Gombert >> 5 rue Enrico Fermi >> 13453 Marseille cedex 13 >> Tel : (33) 4 91 10 69 43 >> Fax : (33) 4 91 10 69 69 >> >> >> >> --- >> L'absence de virus dans ce courrier ?lectronique a ?t? v?rifi?e par le logiciel antivirus Avast. >> https://www.avast.com/antivirus >> yann at crabe:~/projet/AMR/moulinette$ mpiexec -n 1 ./ex2 -petscspace_poly_tensor -petscspace_order 0 -dim 3 >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] DMPlexVecGetClosure line 4020 /home/yann/src/petsc-git/src/dm/impls/plex/plex.c >> [0]PETSC ERROR: [0] DMPlexLocatePoint_General_3D_Internal line 150 /home/yann/src/petsc-git/src/dm/impls/plex/plexgeometry.c >> [0]PETSC ERROR: [0] DMPlexLocatePoint_Internal line 361 /home/yann/src/petsc-git/src/dm/impls/plex/plexgeometry.c >> [0]PETSC ERROR: [0] DMPlexComputeInjectorReferenceTree line 3008 /home/yann/src/petsc-git/src/dm/impls/plex/plextree.c >> [0]PETSC ERROR: [0] DMPlexReferenceTreeGetInjector line 3449 /home/yann/src/petsc-git/src/dm/impls/plex/plextree.c >> [0]PETSC ERROR: [0] DMPlexTransferVecTree_Inject line 4309 /home/yann/src/petsc-git/src/dm/impls/plex/plextree.c >> [0]PETSC ERROR: [0] DMPlexTransferVecTree line 4486 /home/yann/src/petsc-git/src/dm/impls/plex/plextree.c >> [0]PETSC ERROR: [0] DMForestTransferVec_p8est line 4213 /home/yann/src/petsc-git/src/dm/impls/forest/p4est/pforest.c >> [0]PETSC ERROR: [0] DMForestTransferVec line 978 /home/yann/src/petsc-git/src/dm/impls/forest/forest.c >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Signal received >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.8.4, unknown >> [0]PETSC ERROR: ./ex2 on a depot-git named crabe by yann Tue Apr 3 17:18:02 2018 >> [0]PETSC ERROR: Configure options --prefix=/local/lib/petsc/depot-git/gcc/openmpi_gcc_all --with-single-library=0 --with-debugging=1 --download-scalapack=1 --download-metis=1 --download-parmetis=1 --download-ptscotch=1 --download-mumps=1 --download-hypre=1 --download-superlu=1 --download-superlu_dist=1 --download-fblaslapack=1 --download-metis=1 --download-cmake=1 --download-ml=1 --download-p4est=1 --download-netcdf=1 --download-pragmatic=1 --download-eigen=1 --download-parms=1 --download-triangle=1 --download-hdf5=1 --with-zlib=1 --download-szlib=1 --download-chaco=1 --download-spai=1 --download-suitesparse=1 --with-shared-libraries=0 PETSC_ARCH=depot-git >> [0]PETSC ERROR: #1 User provided function() line 0 in unknown file >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 59. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -------------------------------------------------------------------------- >> ------------------------------------------------------- >> Primary job terminated normally, but 1 process returned >> a non-zero exit code.. Per user-direction, the job has been aborted. >> ------------------------------------------------------- >> -------------------------------------------------------------------------- >> mpiexec detected that one or more processes exited with non-zero status, thus causing >> the job to be terminated. The first process to do so was: >> >> Process name: [[36533,1],0] >> Exit code: 59 >> -------------------------------------------------------------------------- >> jobic at stargate:~/projet/AMR/moulinette$ mpirun -np 1 ./ex2 -petscspace_poly_tensor -dim 3 -petscspace_order 0 >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Null argument, when expecting valid pointer >> [0]PETSC ERROR: Null Object: Parameter # 2 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2420-g8f4cb0b GIT Date: 2018-04-02 16:00:52 -0500 >> [0]PETSC ERROR: ./ex2 on a named stargate by jobic Tue Apr 3 17:22:04 2018 >> [0]PETSC ERROR: Configure options --download-chaco=1 --download-cmake=1 --download-eigen=1 --download-fblaslapack=1 --download-hdf5=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mumps=1 --download-netcdf=1 --download-p4est=1 --download-parmetis=1 --download-parms=1 --download-pragmatic=1 --download-ptscotch=1 --download-scalapack=1 --download-spai=1 --download-suitesparse=1 --download-superlu=1 --download-superlu_dist=1 --download-szlib=1 --download-triangle=1 --prefix=/local/lib/petsc/git/gcc/openmpi_gcc_all --with-debugging=1 --with-shared-libraries=0 --with-single-library=0 --with-zlib=1 PETSC_ARCH=openmpi_gcc_all >> [0]PETSC ERROR: #1 PetscFESetDualSpace() line 3615 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtfe.c >> [0]PETSC ERROR: #2 PetscFEGetHeightSubspace() line 6630 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtfe.c >> [0]PETSC ERROR: #3 PetscDSGetHeightSubspace() line 2880 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtds.c >> [0]PETSC ERROR: #4 DMProjectLocal_Generic_Plex() line 325 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/plex/plexproject.c >> [0]PETSC ERROR: #5 DMProjectFunctionLocal_Plex() line 428 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/plex/plexproject.c >> [0]PETSC ERROR: #6 DMProjectFunctionLocal() line 6265 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c >> [0]PETSC ERROR: #7 DMProjectFunctionLocal_p8est() line 4385 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/forest/p4est/pforest.c >> [0]PETSC ERROR: #8 DMProjectFunctionLocal() line 6265 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c >> [0]PETSC ERROR: #9 DMProjectFunction() line 6250 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c >> [0]PETSC ERROR: #10 main() line 161 in /home/jobic/projet/AMR/moulinette/ex2.c >> [0]PETSC ERROR: PETSc Option Table entries: >> [0]PETSC ERROR: -dim 3 >> [0]PETSC ERROR: -petscspace_order 0 >> [0]PETSC ERROR: -petscspace_poly_tensor >> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 85. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -------------------------------------------------------------------------- >> >> >> >> >> >> With valgrind : >> jobic at stargate:~/projet/AMR/moulinette$ valgrind mpirun -np 1 ./ex2 -petscspace_poly_tensor -dim 3 -petscspace_order 0 >> ==9730== Memcheck, a memory error detector >> ==9730== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. >> ==9730== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info >> ==9730== Command: mpirun -np 1 ./ex2 -petscspace_poly_tensor -dim 3 -petscspace_order 0 >> ==9730== >> ==10212== Warning: invalid file descriptor 1024 in syscall close() >> ==10212== Warning: invalid file descriptor 1025 in syscall close() >> ==10212== Warning: invalid file descriptor 1026 in syscall close() >> ==10212== Warning: invalid file descriptor 1027 in syscall close() >> ==10212== Use --log-fd= to select an alternative log fd. >> ==10212== Warning: invalid file descriptor 1028 in syscall close() >> ==10212== Warning: invalid file descriptor 1029 in syscall close() >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Null argument, when expecting valid pointer >> [0]PETSC ERROR: Null Object: Parameter # 2 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2420-g8f4cb0b GIT Date: 2018-04-02 16:00:52 -0500 >> [0]PETSC ERROR: ./ex2 on a named stargate by jobic Tue Apr 3 17:27:58 2018 >> [0]PETSC ERROR: Configure options --download-chaco=1 --download-cmake=1 --download-eigen=1 --download-fblaslapack=1 --download-hdf5=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mumps=1 --download-netcdf=1 --download-p4est=1 --download-parmetis=1 --download-parms=1 --download-pragmatic=1 --download-ptscotch=1 --download-scalapack=1 --download-spai=1 --download-suitesparse=1 --download-superlu=1 --download-superlu_dist=1 --download-szlib=1 --download-triangle=1 --prefix=/local/lib/petsc/git/gcc/openmpi_gcc_all --with-debugging=1 --with-shared-libraries=0 --with-single-library=0 --with-zlib=1 PETSC_ARCH=openmpi_gcc_all >> [0]PETSC ERROR: #1 PetscFESetDualSpace() line 3615 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtfe.c >> [0]PETSC ERROR: #2 PetscFEGetHeightSubspace() line 6630 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtfe.c >> [0]PETSC ERROR: #3 PetscDSGetHeightSubspace() line 2880 in /home/devel/src_linux/git.petsc/petsc/src/dm/dt/interface/dtds.c >> [0]PETSC ERROR: #4 DMProjectLocal_Generic_Plex() line 325 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/plex/plexproject.c >> [0]PETSC ERROR: #5 DMProjectFunctionLocal_Plex() line 428 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/plex/plexproject.c >> [0]PETSC ERROR: #6 DMProjectFunctionLocal() line 6265 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c >> [0]PETSC ERROR: #7 DMProjectFunctionLocal_p8est() line 4385 in /home/devel/src_linux/git.petsc/petsc/src/dm/impls/forest/p4est/pforest.c >> [0]PETSC ERROR: #8 DMProjectFunctionLocal() line 6265 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c >> [0]PETSC ERROR: #9 DMProjectFunction() line 6250 in /home/devel/src_linux/git.petsc/petsc/src/dm/interface/dm.c >> [0]PETSC ERROR: #10 main() line 161 in /home/jobic/projet/AMR/moulinette/ex2.c >> [0]PETSC ERROR: PETSc Option Table entries: >> [0]PETSC ERROR: -dim 3 >> [0]PETSC ERROR: -petscspace_order 0 >> [0]PETSC ERROR: -petscspace_poly_tensor >> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 85. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -------------------------------------------------------------------------- >> ==9730== >> ==9730== HEAP SUMMARY: >> ==9730== in use at exit: 55,755 bytes in 116 blocks >> ==9730== total heap usage: 19,779 allocs, 19,663 frees, 6,633,193 bytes allocated >> ==9730== >> ==9730== LEAK SUMMARY: >> ==9730== definitely lost: 2,675 bytes in 11 blocks >> ==9730== indirectly lost: 48,765 bytes in 75 blocks >> ==9730== possibly lost: 200 bytes in 1 blocks >> ==9730== still reachable: 4,115 bytes in 29 blocks >> ==9730== suppressed: 0 bytes in 0 blocks >> ==9730== Rerun with --leak-check=full to see details of leaked memory >> ==9730== >> ==9730== For counts of detected and suppressed errors, rerun with: -v >> ==9730== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) -- ___________________________ Yann JOBIC HPC engineer IUSTI-CNRS UMR 7343 - Polytech Marseille Technop?le de Ch?teau Gombert 5 rue Enrico Fermi 13453 Marseille cedex 13 Tel : (33) 4 91 10 69 43 Fax : (33) 4 91 10 69 69 From mfadams at lbl.gov Wed Apr 4 11:45:39 2018 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 4 Apr 2018 12:45:39 -0400 Subject: [petsc-users] using hypre's GPU feature through PETSc In-Reply-To: References: Message-ID: On Tue, Apr 3, 2018 at 9:24 PM, Xiangdong wrote: > Hello everyone, > > The latest Hypre has GPU capability. Can I use hypre GPU feature through > PETSc? If it is possible, what configuration changes should I make in PETSc? > It should work. I don't if anyone has used it yet. We recently verified that the OpenMP in hypre works. > > Thank you. > > Best, > Xiangdong > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 4 12:27:37 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 4 Apr 2018 17:27:37 +0000 Subject: [petsc-users] using hypre's GPU feature through PETSc In-Reply-To: References: Message-ID: Our config/BuildSystem/config/packages/hypre.py does not yet have code to communicate GPU information for building hypre. One would need to go through the hypre install documentation to determine the hypre configure flags for enabling the GPU and add support to hypre.py for passing these flags to hypre configure. Barry An example of the type of flag needed can be found in superlu_dist.py help.addArgument('SUPERLU_DIST', '-download-superlu_dist-gpu=', nargs.ArgBool(None, 0, 'Install Superlu_DIST to use GPUs')) > On Apr 3, 2018, at 7:24 PM, Xiangdong wrote: > > Hello everyone, > > The latest Hypre has GPU capability. Can I use hypre GPU feature through PETSc? If it is possible, what configuration changes should I make in PETSc? > > Thank you. > > Best, > Xiangdong > > From fande.kong at inl.gov Wed Apr 4 16:52:21 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Wed, 4 Apr 2018 15:52:21 -0600 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: Updated: If switch to a different version of git. The configuration is going to work. This one works: * petsc]$ git --version git version 1.8.5.2* This new version does not work: petsc]$ git --version git version 2.16.2 Fande. On Wed, Mar 7, 2018 at 2:56 PM, Satish Balay wrote: > On Wed, 7 Mar 2018, Kong, Fande wrote: > > > > I meant just the 3 lines - not the whole function. > > > > I knew this. "3 lines" does not work at all. > > > > I forgot the error message. > > Then you are likely to use the wrong [git] snapshot - and not the > snapshot listed by self.gitcommit - for that package. > > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Apr 4 16:55:52 2018 From: jed at jedbrown.org (Jed Brown) Date: Wed, 04 Apr 2018 15:55:52 -0600 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: <87lge2ljpz.fsf@jedbrown.org> "Kong, Fande" writes: > Updated: > > If switch to a different version of git. The configuration is going to > work. > > > This one works: > > > * petsc]$ git --version git version 1.8.5.2* > > > This new version does not work: > > petsc]$ git --version > git version 2.16.2 Can you reproduce the error in your terminal or does it only happen through configure? From fande.kong at inl.gov Wed Apr 4 16:57:41 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Wed, 4 Apr 2018 15:57:41 -0600 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: <87lge2ljpz.fsf@jedbrown.org> References: <87lge2ljpz.fsf@jedbrown.org> Message-ID: On Wed, Apr 4, 2018 at 3:55 PM, Jed Brown wrote: > "Kong, Fande" writes: > > > Updated: > > > > If switch to a different version of git. The configuration is going to > > work. > > > > > > This one works: > > > > > > * petsc]$ git --version git version 1.8.5.2* > > > > > > This new version does not work: > > > > petsc]$ git --version > > git version 2.16.2 > > Can you reproduce the error in your terminal or does it only happen > through configure? > It only happens using configure. Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Apr 4 17:00:34 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 4 Apr 2018 17:00:34 -0500 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: >>>>>>>> [balay at fedora ~]$ cat /etc/fedora-release Fedora release 27 (Twenty Seven) [balay at fedora ~]$ git --version git version 2.14.3 [balay at fedora ~]$ balay at asterix /home/balay $ cat /etc/fedora-release Fedora release 28 (Twenty Eight) balay at asterix /home/balay $ git --version git version 2.17.0.rc2 balay at asterix /home/balay <<<<<<<< The first one is a nightlybuild testbox - and the second one is my laptop - and I haven't seen this issue with either of them. [or any of our other nightlybuilds - or a bugreport form any other user] was your '2.16.2' version installed from source? Satish On Wed, 4 Apr 2018, Kong, Fande wrote: > Updated: > > If switch to a different version of git. The configuration is going to > work. > > > This one works: > > > * petsc]$ git --version git version 1.8.5.2* > > > This new version does not work: > > petsc]$ git --version > git version 2.16.2 > > > > Fande. > > > On Wed, Mar 7, 2018 at 2:56 PM, Satish Balay wrote: > > > On Wed, 7 Mar 2018, Kong, Fande wrote: > > > > > > I meant just the 3 lines - not the whole function. > > > > > > I knew this. "3 lines" does not work at all. > > > > > > I forgot the error message. > > > > Then you are likely to use the wrong [git] snapshot - and not the > > snapshot listed by self.gitcommit - for that package. > > > > Satish > > > From balay at mcs.anl.gov Wed Apr 4 17:03:42 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 4 Apr 2018 17:03:42 -0500 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: On Wed, 4 Apr 2018, Satish Balay wrote: > was your '2.16.2' version installed from source? >>> Checking for program /usr/bin/git...found Defined make macro "GIT" to "git" Executing: git --version stdout: git version 2.16.2 <<<< I gues its the OS default package >>>>> Machine platform: ('Linux', 'falcon1', '3.0.101-108.13-default', '#1 SMP Wed Oct 11 12:30:40 UTC 2017 (294ccfe)', 'x86_64', 'x86_64') Python version: 2.6.9 (unknown, Aug 5 2016, 11:15:31) [GCC 4.3.4 [gcc-4_3-branch revision 152973]] <<<< What OS/version is on this machine? I can try reproducing in a VM Satish From balay at mcs.anl.gov Wed Apr 4 17:08:42 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 4 Apr 2018 17:08:42 -0500 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: On Wed, 4 Apr 2018, Satish Balay wrote: > On Wed, 4 Apr 2018, Satish Balay wrote: > > > was your '2.16.2' version installed from source? > > >>> > Checking for program /usr/bin/git...found > Defined make macro "GIT" to "git" > Executing: git --version > stdout: git version 2.16.2 > <<<< > > I gues its the OS default package > > >>>>> > Machine platform: > ('Linux', 'falcon1', '3.0.101-108.13-default', '#1 SMP Wed Oct 11 12:30:40 UTC 2017 (294ccfe)', 'x86_64', 'x86_64') > Python version: > 2.6.9 (unknown, Aug 5 2016, 11:15:31) > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] > <<<< > > What OS/version is on this machine? I can try reproducing in a VM It is strange that the kernel is old [3.0 - perhaps LTS OS] , python is old [2.6] - but git is new? [2.16?] https://mirrors.edge.kernel.org/pub/software/scm/git/ git-2.16.2.tar.gz 16-Feb-2018 17:48 7M Satish From fande.kong at inl.gov Wed Apr 4 17:10:33 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Wed, 4 Apr 2018 16:10:33 -0600 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: moose]$ uname -a Linux falcon1 3.0.101-108.13-default #1 SMP Wed Oct 11 12:30:40 UTC 2017 (294ccfe) x86_64 x86_64 x86_64 GNU/Linux moose]$ lsb_release -a LSB Version: core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch Distributor ID: SUSE LINUX Description: SUSE Linux Enterprise Server 11 (x86_64) Release: 11 Codename: n/a On Wed, Apr 4, 2018 at 4:08 PM, Satish Balay wrote: > On Wed, 4 Apr 2018, Satish Balay wrote: > > > On Wed, 4 Apr 2018, Satish Balay wrote: > > > > > was your '2.16.2' version installed from source? > > > > >>> > > Checking for program /usr/bin/git...found > > Defined make macro "GIT" to "git" > > Executing: git --version > > stdout: git version 2.16.2 > > <<<< > > > > I gues its the OS default package > > > > >>>>> > > Machine platform: > > ('Linux', 'falcon1', '3.0.101-108.13-default', '#1 SMP Wed Oct 11 > 12:30:40 UTC 2017 (294ccfe)', 'x86_64', 'x86_64') > > Python version: > > 2.6.9 (unknown, Aug 5 2016, 11:15:31) > > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] > > <<<< > > > > What OS/version is on this machine? I can try reproducing in a VM > > It is strange that the kernel is old [3.0 - perhaps LTS OS] , python is > old [2.6] - but git is new? [2.16?] > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ > mirrors.edge.kernel.org_pub_software_scm_git_&d=DwIBAg&c= > 54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_ > JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=WF6ZxIh9Z9Hh2kYY0w70aNrgfPicp6 > kgIh5BvezPiEY&s=KWMsO7XC0-pQuQ_mD03tNJWEpxSTATlZW_DmX0QofGw&e= > git-2.16.2.tar.gz 16-Feb-2018 17:48 > 7M > > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Apr 4 17:20:59 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 4 Apr 2018 17:20:59 -0500 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: Ok - I don't think I have access to this OS. And I see its from 2009 [sure its enterprise os - with regular backport updates] But its wierd that you have such a new version of git at /usr/bin/git >From what we know so far - the problem appears to be some bad interaction of python-2.6 with this old OS [i.e old glibc etc..] - and this new git version [binary built locally or on a different OS and installed locally ?]. Satish On Wed, 4 Apr 2018, Kong, Fande wrote: > moose]$ uname -a > Linux falcon1 3.0.101-108.13-default #1 SMP Wed Oct 11 12:30:40 UTC 2017 > (294ccfe) x86_64 x86_64 x86_64 GNU/Linux > > > moose]$ lsb_release -a > LSB Version: > core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch > Distributor ID: SUSE LINUX > Description: SUSE Linux Enterprise Server 11 (x86_64) > Release: 11 > Codename: n/a > > > > On Wed, Apr 4, 2018 at 4:08 PM, Satish Balay wrote: > > > On Wed, 4 Apr 2018, Satish Balay wrote: > > > > > On Wed, 4 Apr 2018, Satish Balay wrote: > > > > > > > was your '2.16.2' version installed from source? > > > > > > >>> > > > Checking for program /usr/bin/git...found > > > Defined make macro "GIT" to "git" > > > Executing: git --version > > > stdout: git version 2.16.2 > > > <<<< > > > > > > I gues its the OS default package > > > > > > >>>>> > > > Machine platform: > > > ('Linux', 'falcon1', '3.0.101-108.13-default', '#1 SMP Wed Oct 11 > > 12:30:40 UTC 2017 (294ccfe)', 'x86_64', 'x86_64') > > > Python version: > > > 2.6.9 (unknown, Aug 5 2016, 11:15:31) > > > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] > > > <<<< > > > > > > What OS/version is on this machine? I can try reproducing in a VM > > > > It is strange that the kernel is old [3.0 - perhaps LTS OS] , python is > > old [2.6] - but git is new? [2.16?] > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ > > mirrors.edge.kernel.org_pub_software_scm_git_&d=DwIBAg&c= > > 54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_ > > JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=WF6ZxIh9Z9Hh2kYY0w70aNrgfPicp6 > > kgIh5BvezPiEY&s=KWMsO7XC0-pQuQ_mD03tNJWEpxSTATlZW_DmX0QofGw&e= > > git-2.16.2.tar.gz 16-Feb-2018 17:48 > > 7M > > > > Satish > > > From fande.kong at inl.gov Wed Apr 4 17:28:50 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Wed, 4 Apr 2018 16:28:50 -0600 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: Hi Cormac, Do you know anything on "git"? How did you guys build git on the falcon1? The default git on Falcon1 does not work with petsc any more. Fande, On Wed, Apr 4, 2018 at 4:20 PM, Satish Balay wrote: > Ok - I don't think I have access to this OS. > > And I see its from 2009 [sure its enterprise os - with regular backport > updates] > > But its wierd that you have such a new version of git at /usr/bin/git > > From what we know so far - the problem appears to be some bad > interaction of python-2.6 with this old OS [i.e old glibc etc..] - and > this new git version [binary built locally or on a different OS and > installed locally ?]. > > Satish > > On Wed, 4 Apr 2018, Kong, Fande wrote: > > > moose]$ uname -a > > Linux falcon1 3.0.101-108.13-default #1 SMP Wed Oct 11 12:30:40 UTC 2017 > > (294ccfe) x86_64 x86_64 x86_64 GNU/Linux > > > > > > moose]$ lsb_release -a > > LSB Version: > > core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2. > 0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64: > desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch: > graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64: > graphics-4.0-noarch > > Distributor ID: SUSE LINUX > > Description: SUSE Linux Enterprise Server 11 (x86_64) > > Release: 11 > > Codename: n/a > > > > > > > > On Wed, Apr 4, 2018 at 4:08 PM, Satish Balay wrote: > > > > > On Wed, 4 Apr 2018, Satish Balay wrote: > > > > > > > On Wed, 4 Apr 2018, Satish Balay wrote: > > > > > > > > > was your '2.16.2' version installed from source? > > > > > > > > >>> > > > > Checking for program /usr/bin/git...found > > > > Defined make macro "GIT" to "git" > > > > Executing: git --version > > > > stdout: git version 2.16.2 > > > > <<<< > > > > > > > > I gues its the OS default package > > > > > > > > >>>>> > > > > Machine platform: > > > > ('Linux', 'falcon1', '3.0.101-108.13-default', '#1 SMP Wed Oct 11 > > > 12:30:40 UTC 2017 (294ccfe)', 'x86_64', 'x86_64') > > > > Python version: > > > > 2.6.9 (unknown, Aug 5 2016, 11:15:31) > > > > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] > > > > <<<< > > > > > > > > What OS/version is on this machine? I can try reproducing in a VM > > > > > > It is strange that the kernel is old [3.0 - perhaps LTS OS] , python is > > > old [2.6] - but git is new? [2.16?] > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ > > > mirrors.edge.kernel.org_pub_software_scm_git_&d=DwIBAg&c= > > > 54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_ > > > JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=WF6ZxIh9Z9Hh2kYY0w70aNrgfPicp6 > > > kgIh5BvezPiEY&s=KWMsO7XC0-pQuQ_mD03tNJWEpxSTATlZW_DmX0QofGw&e= > > > git-2.16.2.tar.gz 16-Feb-2018 17:48 > > > 7M > > > > > > Satish > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cormac.garvey at inl.gov Wed Apr 4 17:39:24 2018 From: cormac.garvey at inl.gov (Garvey, Cormac T) Date: Wed, 4 Apr 2018 16:39:24 -0600 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: We needed to rebuilt git on the INL falcon cluster because github server changed such that it no longer accepted TLSv1. The default git on the falcon cluster /usr/bin/git is just a wrapper script, so users would not need to load any modules to use git. When you load load git on falcon1 or falcon2 does it still fail? module load git/2.16.2-GCCcore-5.4.0 Thanks, Cormac. On Wed, Apr 4, 2018 at 4:28 PM, Kong, Fande wrote: > Hi Cormac, > > Do you know anything on "git"? How did you guys build git on the falcon1? > The default git on Falcon1 does not work with petsc any more. > > > Fande, > > > > On Wed, Apr 4, 2018 at 4:20 PM, Satish Balay wrote: > >> Ok - I don't think I have access to this OS. >> >> And I see its from 2009 [sure its enterprise os - with regular backport >> updates] >> >> But its wierd that you have such a new version of git at /usr/bin/git >> >> From what we know so far - the problem appears to be some bad >> interaction of python-2.6 with this old OS [i.e old glibc etc..] - and >> this new git version [binary built locally or on a different OS and >> installed locally ?]. >> >> Satish >> >> On Wed, 4 Apr 2018, Kong, Fande wrote: >> >> > moose]$ uname -a >> > Linux falcon1 3.0.101-108.13-default #1 SMP Wed Oct 11 12:30:40 UTC 2017 >> > (294ccfe) x86_64 x86_64 x86_64 GNU/Linux >> > >> > >> > moose]$ lsb_release -a >> > LSB Version: >> > core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0- >> x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:des >> ktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:grap >> hics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch >> > Distributor ID: SUSE LINUX >> > Description: SUSE Linux Enterprise Server 11 (x86_64) >> > Release: 11 >> > Codename: n/a >> > >> > >> > >> > On Wed, Apr 4, 2018 at 4:08 PM, Satish Balay wrote: >> > >> > > On Wed, 4 Apr 2018, Satish Balay wrote: >> > > >> > > > On Wed, 4 Apr 2018, Satish Balay wrote: >> > > > >> > > > > was your '2.16.2' version installed from source? >> > > > >> > > > >>> >> > > > Checking for program /usr/bin/git...found >> > > > Defined make macro "GIT" to "git" >> > > > Executing: git --version >> > > > stdout: git version 2.16.2 >> > > > <<<< >> > > > >> > > > I gues its the OS default package >> > > > >> > > > >>>>> >> > > > Machine platform: >> > > > ('Linux', 'falcon1', '3.0.101-108.13-default', '#1 SMP Wed Oct 11 >> > > 12:30:40 UTC 2017 (294ccfe)', 'x86_64', 'x86_64') >> > > > Python version: >> > > > 2.6.9 (unknown, Aug 5 2016, 11:15:31) >> > > > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] >> > > > <<<< >> > > > >> > > > What OS/version is on this machine? I can try reproducing in a VM >> > > >> > > It is strange that the kernel is old [3.0 - perhaps LTS OS] , python >> is >> > > old [2.6] - but git is new? [2.16?] >> > > >> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ >> > > mirrors.edge.kernel.org_pub_software_scm_git_&d=DwIBAg&c= >> > > 54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_ >> > > JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=WF6ZxIh9Z9Hh2kYY0w70aNrgfPicp6 >> > > kgIh5BvezPiEY&s=KWMsO7XC0-pQuQ_mD03tNJWEpxSTATlZW_DmX0QofGw&e= >> > > git-2.16.2.tar.gz 16-Feb-2018 17:48 >> > > 7M >> > > >> > > Satish >> > > >> > >> >> > -- Cormac Garvey HPC Software Consultant Scientific Computing Idaho National Laboratory Ph: 208-526-6294 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Wed Apr 4 17:43:07 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Wed, 4 Apr 2018 16:43:07 -0600 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: Thanks, Cormac, *module load git/1.8.5.2-GCC-4.8.3 * works for me. Did not try "module load git/2.16.2-GCCcore-5.4.0" yet. I will try, and get it back here. Fande On Wed, Apr 4, 2018 at 4:39 PM, Garvey, Cormac T wrote: > > We needed to rebuilt git on the INL falcon cluster because github server > changed such that it no longer accepted TLSv1. > > The default git on the falcon cluster /usr/bin/git is just a wrapper > script, so users would not need to load any modules to > use git. > > When you load load git on falcon1 or falcon2 does it still fail? > > module load git/2.16.2-GCCcore-5.4.0 > > Thanks, > Cormac. > > > > On Wed, Apr 4, 2018 at 4:28 PM, Kong, Fande wrote: > >> Hi Cormac, >> >> Do you know anything on "git"? How did you guys build git on the >> falcon1? The default git on Falcon1 does not work with petsc any more. >> >> >> Fande, >> >> >> >> On Wed, Apr 4, 2018 at 4:20 PM, Satish Balay wrote: >> >>> Ok - I don't think I have access to this OS. >>> >>> And I see its from 2009 [sure its enterprise os - with regular backport >>> updates] >>> >>> But its wierd that you have such a new version of git at /usr/bin/git >>> >>> From what we know so far - the problem appears to be some bad >>> interaction of python-2.6 with this old OS [i.e old glibc etc..] - and >>> this new git version [binary built locally or on a different OS and >>> installed locally ?]. >>> >>> Satish >>> >>> On Wed, 4 Apr 2018, Kong, Fande wrote: >>> >>> > moose]$ uname -a >>> > Linux falcon1 3.0.101-108.13-default #1 SMP Wed Oct 11 12:30:40 UTC >>> 2017 >>> > (294ccfe) x86_64 x86_64 x86_64 GNU/Linux >>> > >>> > >>> > moose]$ lsb_release -a >>> > LSB Version: >>> > core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86 >>> _64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:deskt >>> op-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphic >>> s-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch >>> > Distributor ID: SUSE LINUX >>> > Description: SUSE Linux Enterprise Server 11 (x86_64) >>> > Release: 11 >>> > Codename: n/a >>> > >>> > >>> > >>> > On Wed, Apr 4, 2018 at 4:08 PM, Satish Balay >>> wrote: >>> > >>> > > On Wed, 4 Apr 2018, Satish Balay wrote: >>> > > >>> > > > On Wed, 4 Apr 2018, Satish Balay wrote: >>> > > > >>> > > > > was your '2.16.2' version installed from source? >>> > > > >>> > > > >>> >>> > > > Checking for program /usr/bin/git...found >>> > > > Defined make macro "GIT" to "git" >>> > > > Executing: git --version >>> > > > stdout: git version 2.16.2 >>> > > > <<<< >>> > > > >>> > > > I gues its the OS default package >>> > > > >>> > > > >>>>> >>> > > > Machine platform: >>> > > > ('Linux', 'falcon1', '3.0.101-108.13-default', '#1 SMP Wed Oct 11 >>> > > 12:30:40 UTC 2017 (294ccfe)', 'x86_64', 'x86_64') >>> > > > Python version: >>> > > > 2.6.9 (unknown, Aug 5 2016, 11:15:31) >>> > > > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] >>> > > > <<<< >>> > > > >>> > > > What OS/version is on this machine? I can try reproducing in a VM >>> > > >>> > > It is strange that the kernel is old [3.0 - perhaps LTS OS] , python >>> is >>> > > old [2.6] - but git is new? [2.16?] >>> > > >>> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ >>> > > mirrors.edge.kernel.org_pub_software_scm_git_&d=DwIBAg&c= >>> > > 54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_ >>> > > JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=WF6ZxIh9Z9Hh2kYY0w70aNrgfPicp6 >>> > > kgIh5BvezPiEY&s=KWMsO7XC0-pQuQ_mD03tNJWEpxSTATlZW_DmX0QofGw&e= >>> > > git-2.16.2.tar.gz 16-Feb-2018 17:48 >>> > > 7M >>> > > >>> > > Satish >>> > > >>> > >>> >>> >> > > > -- > Cormac Garvey > HPC Software Consultant > Scientific Computing > Idaho National Laboratory > Ph: 208-526-6294 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Wed Apr 4 17:48:56 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Wed, 4 Apr 2018 16:48:56 -0600 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: module load git/2.16.2-GCCcore-5.4.0" also works. Could you somehow make the default git work as well? Hence we do not need to have this extra "module load for git" Fande, On Wed, Apr 4, 2018 at 4:43 PM, Kong, Fande wrote: > Thanks, Cormac, > > *module load git/1.8.5.2-GCC-4.8.3 * > > works for me. > > Did not try "module load git/2.16.2-GCCcore-5.4.0" yet. > > I will try, and get it back here. > > > > Fande > > On Wed, Apr 4, 2018 at 4:39 PM, Garvey, Cormac T > wrote: > >> >> We needed to rebuilt git on the INL falcon cluster because github server >> changed such that it no longer accepted TLSv1. >> >> The default git on the falcon cluster /usr/bin/git is just a wrapper >> script, so users would not need to load any modules to >> use git. >> >> When you load load git on falcon1 or falcon2 does it still fail? >> >> module load git/2.16.2-GCCcore-5.4.0 >> >> Thanks, >> Cormac. >> >> >> >> On Wed, Apr 4, 2018 at 4:28 PM, Kong, Fande wrote: >> >>> Hi Cormac, >>> >>> Do you know anything on "git"? How did you guys build git on the >>> falcon1? The default git on Falcon1 does not work with petsc any more. >>> >>> >>> Fande, >>> >>> >>> >>> On Wed, Apr 4, 2018 at 4:20 PM, Satish Balay wrote: >>> >>>> Ok - I don't think I have access to this OS. >>>> >>>> And I see its from 2009 [sure its enterprise os - with regular backport >>>> updates] >>>> >>>> But its wierd that you have such a new version of git at /usr/bin/git >>>> >>>> From what we know so far - the problem appears to be some bad >>>> interaction of python-2.6 with this old OS [i.e old glibc etc..] - and >>>> this new git version [binary built locally or on a different OS and >>>> installed locally ?]. >>>> >>>> Satish >>>> >>>> On Wed, 4 Apr 2018, Kong, Fande wrote: >>>> >>>> > moose]$ uname -a >>>> > Linux falcon1 3.0.101-108.13-default #1 SMP Wed Oct 11 12:30:40 UTC >>>> 2017 >>>> > (294ccfe) x86_64 x86_64 x86_64 GNU/Linux >>>> > >>>> > >>>> > moose]$ lsb_release -a >>>> > LSB Version: >>>> > core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86 >>>> _64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:deskto >>>> p-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics >>>> -3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch >>>> > Distributor ID: SUSE LINUX >>>> > Description: SUSE Linux Enterprise Server 11 (x86_64) >>>> > Release: 11 >>>> > Codename: n/a >>>> > >>>> > >>>> > >>>> > On Wed, Apr 4, 2018 at 4:08 PM, Satish Balay >>>> wrote: >>>> > >>>> > > On Wed, 4 Apr 2018, Satish Balay wrote: >>>> > > >>>> > > > On Wed, 4 Apr 2018, Satish Balay wrote: >>>> > > > >>>> > > > > was your '2.16.2' version installed from source? >>>> > > > >>>> > > > >>> >>>> > > > Checking for program /usr/bin/git...found >>>> > > > Defined make macro "GIT" to "git" >>>> > > > Executing: git --version >>>> > > > stdout: git version 2.16.2 >>>> > > > <<<< >>>> > > > >>>> > > > I gues its the OS default package >>>> > > > >>>> > > > >>>>> >>>> > > > Machine platform: >>>> > > > ('Linux', 'falcon1', '3.0.101-108.13-default', '#1 SMP Wed Oct 11 >>>> > > 12:30:40 UTC 2017 (294ccfe)', 'x86_64', 'x86_64') >>>> > > > Python version: >>>> > > > 2.6.9 (unknown, Aug 5 2016, 11:15:31) >>>> > > > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] >>>> > > > <<<< >>>> > > > >>>> > > > What OS/version is on this machine? I can try reproducing in a VM >>>> > > >>>> > > It is strange that the kernel is old [3.0 - perhaps LTS OS] , >>>> python is >>>> > > old [2.6] - but git is new? [2.16?] >>>> > > >>>> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ >>>> > > mirrors.edge.kernel.org_pub_software_scm_git_&d=DwIBAg&c= >>>> > > 54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_ >>>> > > JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=WF6ZxIh9Z9Hh2kYY0w70aNrgfPicp6 >>>> > > kgIh5BvezPiEY&s=KWMsO7XC0-pQuQ_mD03tNJWEpxSTATlZW_DmX0QofGw&e= >>>> > > git-2.16.2.tar.gz 16-Feb-2018 17:48 >>>> > > 7M >>>> > > >>>> > > Satish >>>> > > >>>> > >>>> >>>> >>> >> >> >> -- >> Cormac Garvey >> HPC Software Consultant >> Scientific Computing >> Idaho National Laboratory >> Ph: 208-526-6294 >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cormac.garvey at inl.gov Wed Apr 4 18:04:57 2018 From: cormac.garvey at inl.gov (Garvey, Cormac T) Date: Wed, 4 Apr 2018 17:04:57 -0600 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: I though it was fixed, yes I will look into it again. Do you get an error just doing a git clone on falcon1 and falcon2? On Wed, Apr 4, 2018 at 4:48 PM, Kong, Fande wrote: > module load git/2.16.2-GCCcore-5.4.0" also works. > > Could you somehow make the default git work as well? Hence we do not need > to have this extra "module load for git" > > Fande, > > On Wed, Apr 4, 2018 at 4:43 PM, Kong, Fande wrote: > >> Thanks, Cormac, >> >> *module load git/1.8.5.2-GCC-4.8.3 * >> >> works for me. >> >> Did not try "module load git/2.16.2-GCCcore-5.4.0" yet. >> >> I will try, and get it back here. >> >> >> >> Fande >> >> On Wed, Apr 4, 2018 at 4:39 PM, Garvey, Cormac T >> wrote: >> >>> >>> We needed to rebuilt git on the INL falcon cluster because github server >>> changed such that it no longer accepted TLSv1. >>> >>> The default git on the falcon cluster /usr/bin/git is just a wrapper >>> script, so users would not need to load any modules to >>> use git. >>> >>> When you load load git on falcon1 or falcon2 does it still fail? >>> >>> module load git/2.16.2-GCCcore-5.4.0 >>> >>> Thanks, >>> Cormac. >>> >>> >>> >>> On Wed, Apr 4, 2018 at 4:28 PM, Kong, Fande wrote: >>> >>>> Hi Cormac, >>>> >>>> Do you know anything on "git"? How did you guys build git on the >>>> falcon1? The default git on Falcon1 does not work with petsc any more. >>>> >>>> >>>> Fande, >>>> >>>> >>>> >>>> On Wed, Apr 4, 2018 at 4:20 PM, Satish Balay wrote: >>>> >>>>> Ok - I don't think I have access to this OS. >>>>> >>>>> And I see its from 2009 [sure its enterprise os - with regular >>>>> backport updates] >>>>> >>>>> But its wierd that you have such a new version of git at /usr/bin/git >>>>> >>>>> From what we know so far - the problem appears to be some bad >>>>> interaction of python-2.6 with this old OS [i.e old glibc etc..] - and >>>>> this new git version [binary built locally or on a different OS and >>>>> installed locally ?]. >>>>> >>>>> Satish >>>>> >>>>> On Wed, 4 Apr 2018, Kong, Fande wrote: >>>>> >>>>> > moose]$ uname -a >>>>> > Linux falcon1 3.0.101-108.13-default #1 SMP Wed Oct 11 12:30:40 UTC >>>>> 2017 >>>>> > (294ccfe) x86_64 x86_64 x86_64 GNU/Linux >>>>> > >>>>> > >>>>> > moose]$ lsb_release -a >>>>> > LSB Version: >>>>> > core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86 >>>>> _64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:deskto >>>>> p-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics >>>>> -3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch >>>>> > Distributor ID: SUSE LINUX >>>>> > Description: SUSE Linux Enterprise Server 11 (x86_64) >>>>> > Release: 11 >>>>> > Codename: n/a >>>>> > >>>>> > >>>>> > >>>>> > On Wed, Apr 4, 2018 at 4:08 PM, Satish Balay >>>>> wrote: >>>>> > >>>>> > > On Wed, 4 Apr 2018, Satish Balay wrote: >>>>> > > >>>>> > > > On Wed, 4 Apr 2018, Satish Balay wrote: >>>>> > > > >>>>> > > > > was your '2.16.2' version installed from source? >>>>> > > > >>>>> > > > >>> >>>>> > > > Checking for program /usr/bin/git...found >>>>> > > > Defined make macro "GIT" to "git" >>>>> > > > Executing: git --version >>>>> > > > stdout: git version 2.16.2 >>>>> > > > <<<< >>>>> > > > >>>>> > > > I gues its the OS default package >>>>> > > > >>>>> > > > >>>>> >>>>> > > > Machine platform: >>>>> > > > ('Linux', 'falcon1', '3.0.101-108.13-default', '#1 SMP Wed Oct 11 >>>>> > > 12:30:40 UTC 2017 (294ccfe)', 'x86_64', 'x86_64') >>>>> > > > Python version: >>>>> > > > 2.6.9 (unknown, Aug 5 2016, 11:15:31) >>>>> > > > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] >>>>> > > > <<<< >>>>> > > > >>>>> > > > What OS/version is on this machine? I can try reproducing in a VM >>>>> > > >>>>> > > It is strange that the kernel is old [3.0 - perhaps LTS OS] , >>>>> python is >>>>> > > old [2.6] - but git is new? [2.16?] >>>>> > > >>>>> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ >>>>> > > mirrors.edge.kernel.org_pub_software_scm_git_&d=DwIBAg&c= >>>>> > > 54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_ >>>>> > > JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=WF6ZxIh9Z9Hh2kYY0w70aNrgfPicp6 >>>>> > > kgIh5BvezPiEY&s=KWMsO7XC0-pQuQ_mD03tNJWEpxSTATlZW_DmX0QofGw&e= >>>>> > > git-2.16.2.tar.gz 16-Feb-2018 >>>>> 17:48 >>>>> > > 7M >>>>> > > >>>>> > > Satish >>>>> > > >>>>> > >>>>> >>>>> >>>> >>> >>> >>> -- >>> Cormac Garvey >>> HPC Software Consultant >>> Scientific Computing >>> Idaho National Laboratory >>> Ph: 208-526-6294 >>> >>> >> > -- Cormac Garvey HPC Software Consultant Scientific Computing Idaho National Laboratory Ph: 208-526-6294 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Wed Apr 4 18:09:58 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Wed, 4 Apr 2018 17:09:58 -0600 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: The default git gives me: *Could not execute "['git', 'rev-parse', '--git-dir']"* when I am configuring PETSc. The manually loaded *gits* work just fine. Fande, On Wed, Apr 4, 2018 at 5:04 PM, Garvey, Cormac T wrote: > I though it was fixed, yes I will look into it again. > > Do you get an error just doing a git clone on falcon1 and falcon2? > > On Wed, Apr 4, 2018 at 4:48 PM, Kong, Fande wrote: > >> module load git/2.16.2-GCCcore-5.4.0" also works. >> >> Could you somehow make the default git work as well? Hence we do not need >> to have this extra "module load for git" >> >> Fande, >> >> On Wed, Apr 4, 2018 at 4:43 PM, Kong, Fande wrote: >> >>> Thanks, Cormac, >>> >>> *module load git/1.8.5.2-GCC-4.8.3 * >>> >>> works for me. >>> >>> Did not try "module load git/2.16.2-GCCcore-5.4.0" yet. >>> >>> I will try, and get it back here. >>> >>> >>> >>> Fande >>> >>> On Wed, Apr 4, 2018 at 4:39 PM, Garvey, Cormac T >>> wrote: >>> >>>> >>>> We needed to rebuilt git on the INL falcon cluster because github >>>> server changed such that it no longer accepted TLSv1. >>>> >>>> The default git on the falcon cluster /usr/bin/git is just a wrapper >>>> script, so users would not need to load any modules to >>>> use git. >>>> >>>> When you load load git on falcon1 or falcon2 does it still fail? >>>> >>>> module load git/2.16.2-GCCcore-5.4.0 >>>> >>>> Thanks, >>>> Cormac. >>>> >>>> >>>> >>>> On Wed, Apr 4, 2018 at 4:28 PM, Kong, Fande wrote: >>>> >>>>> Hi Cormac, >>>>> >>>>> Do you know anything on "git"? How did you guys build git on the >>>>> falcon1? The default git on Falcon1 does not work with petsc any more. >>>>> >>>>> >>>>> Fande, >>>>> >>>>> >>>>> >>>>> On Wed, Apr 4, 2018 at 4:20 PM, Satish Balay >>>>> wrote: >>>>> >>>>>> Ok - I don't think I have access to this OS. >>>>>> >>>>>> And I see its from 2009 [sure its enterprise os - with regular >>>>>> backport updates] >>>>>> >>>>>> But its wierd that you have such a new version of git at /usr/bin/git >>>>>> >>>>>> From what we know so far - the problem appears to be some bad >>>>>> interaction of python-2.6 with this old OS [i.e old glibc etc..] - and >>>>>> this new git version [binary built locally or on a different OS and >>>>>> installed locally ?]. >>>>>> >>>>>> Satish >>>>>> >>>>>> On Wed, 4 Apr 2018, Kong, Fande wrote: >>>>>> >>>>>> > moose]$ uname -a >>>>>> > Linux falcon1 3.0.101-108.13-default #1 SMP Wed Oct 11 12:30:40 UTC >>>>>> 2017 >>>>>> > (294ccfe) x86_64 x86_64 x86_64 GNU/Linux >>>>>> > >>>>>> > >>>>>> > moose]$ lsb_release -a >>>>>> > LSB Version: >>>>>> > core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86 >>>>>> _64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:deskto >>>>>> p-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics >>>>>> -3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch >>>>>> > Distributor ID: SUSE LINUX >>>>>> > Description: SUSE Linux Enterprise Server 11 (x86_64) >>>>>> > Release: 11 >>>>>> > Codename: n/a >>>>>> > >>>>>> > >>>>>> > >>>>>> > On Wed, Apr 4, 2018 at 4:08 PM, Satish Balay >>>>>> wrote: >>>>>> > >>>>>> > > On Wed, 4 Apr 2018, Satish Balay wrote: >>>>>> > > >>>>>> > > > On Wed, 4 Apr 2018, Satish Balay wrote: >>>>>> > > > >>>>>> > > > > was your '2.16.2' version installed from source? >>>>>> > > > >>>>>> > > > >>> >>>>>> > > > Checking for program /usr/bin/git...found >>>>>> > > > Defined make macro "GIT" to "git" >>>>>> > > > Executing: git --version >>>>>> > > > stdout: git version 2.16.2 >>>>>> > > > <<<< >>>>>> > > > >>>>>> > > > I gues its the OS default package >>>>>> > > > >>>>>> > > > >>>>> >>>>>> > > > Machine platform: >>>>>> > > > ('Linux', 'falcon1', '3.0.101-108.13-default', '#1 SMP Wed Oct >>>>>> 11 >>>>>> > > 12:30:40 UTC 2017 (294ccfe)', 'x86_64', 'x86_64') >>>>>> > > > Python version: >>>>>> > > > 2.6.9 (unknown, Aug 5 2016, 11:15:31) >>>>>> > > > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] >>>>>> > > > <<<< >>>>>> > > > >>>>>> > > > What OS/version is on this machine? I can try reproducing in a >>>>>> VM >>>>>> > > >>>>>> > > It is strange that the kernel is old [3.0 - perhaps LTS OS] , >>>>>> python is >>>>>> > > old [2.6] - but git is new? [2.16?] >>>>>> > > >>>>>> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ >>>>>> > > mirrors.edge.kernel.org_pub_software_scm_git_&d=DwIBAg&c= >>>>>> > > 54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_ >>>>>> > > JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=WF6ZxIh9Z9Hh2kYY0w70aNrgfPicp6 >>>>>> > > kgIh5BvezPiEY&s=KWMsO7XC0-pQuQ_mD03tNJWEpxSTATlZW_DmX0QofGw&e= >>>>>> > > git-2.16.2.tar.gz 16-Feb-2018 >>>>>> 17:48 >>>>>> > > 7M >>>>>> > > >>>>>> > > Satish >>>>>> > > >>>>>> > >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Cormac Garvey >>>> HPC Software Consultant >>>> Scientific Computing >>>> Idaho National Laboratory >>>> Ph: 208-526-6294 >>>> >>>> >>> >> > > > -- > Cormac Garvey > HPC Software Consultant > Scientific Computing > Idaho National Laboratory > Ph: 208-526-6294 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Thu Apr 5 04:09:19 2018 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Thu, 5 Apr 2018 09:09:19 +0000 Subject: [petsc-users] --download-hdf5 and then compiling cgns In-Reply-To: References: <1522329528872.65565@marin.nl> , Message-ID: <1522919359888.15860@marin.nl> Satish, Gaetan, thanks for your suggestions. I've decided not to use cmake for cgns, and that works fine with petsc's build of hdf5. Chris dr. ir. Christiaan Klaij | Senior Researcher | Research & Development MARIN | T +31 317 49 33 44 | C.Klaij at marin.nl | www.marin.nl [LinkedIn] [YouTube] [Twitter] [Facebook] MARIN news: Kom zaterdag 10 november naar de open dag in Wageningen! ________________________________ From: Gaetan Kenway Sent: Thursday, March 29, 2018 6:43 PM To: petsc-users Cc: Klaij, Christiaan Subject: Re: [petsc-users] --download-hdf5 and then compiling cgns I have compiled CGNS3.3 with hdf from PETSc but not using cmake from CGNS. Here is what I used: git clone https://github.com/CGNS/CGNS.git cd CGNS/src export CC=mpicc export FC=mpif90 export FCFLAGS='-fPIC -O3' export CFLAGS='-fPIC -O3' export LIBS='-lz -ldl' ./configure --with-hdf5=$PETSC_DIR/$PETSC_ARCH --enable-64bit --prefix=/u/wk/gkenway/.local --enable-parallel --enable-cgnstools=yes --enable-shared make -j 10 Hope that helps Gaetan On Thu, Mar 29, 2018 at 9:37 AM, Satish Balay > wrote: --download-hdf5 uses 'autoconf' build of hdf5 - and not cmake build. [hdf5 provides both build tools]. This must be the reason why hdf5Config.cmake and hdf5-config.cmake files might be missing. You might be able to install hdf5 manually with cmake - and then build cgns-3.3.1 with it. Satish On Thu, 29 Mar 2018, Klaij, Christiaan wrote: > Satish, > > I've build petsc-3.8.3 with --download-hdf5 and this works > fine. Now, I'm trying to compile cgns-3.3.1 and have it use the > hdf5 library from petsc. Therefore, in the cmake configure I have > set this option > > -D HDF5_DIR=/path/to/petsc/hdf5 > > but cmake gives this message: > > CMake Warning at CMakeLists.txt:207 (find_package): > Could not find a package configuration file provided by "HDF5" with any of > the following names: > > hdf5Config.cmake > hdf5-config.cmake > > and then falls back to the system's hdf5. Indeed there are no > such files in /path/to/petsc/hdf5. Any idea on how to proceed > from here? > > Chris > > > dr. ir. Christiaan Klaij | Senior Researcher | Research & Development > MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl > > MARIN news: http://www.marin.nl/web/News/News-items/Comfort-draft-for-semisubmersible-yachts.htm > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: imagee7f06f.PNG Type: image/png Size: 293 bytes Desc: imagee7f06f.PNG URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image720aad.PNG Type: image/png Size: 331 bytes Desc: image720aad.PNG URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: imageeef69a.PNG Type: image/png Size: 333 bytes Desc: imageeef69a.PNG URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image92a631.PNG Type: image/png Size: 253 bytes Desc: image92a631.PNG URL: From r.speck at fz-juelich.de Thu Apr 5 05:39:01 2018 From: r.speck at fz-juelich.de (Robert Speck) Date: Thu, 5 Apr 2018 12:39:01 +0200 Subject: [petsc-users] petsc4py: reuse setup for multiple solver calls? Message-ID: Hi! I would like to use petsc4py for my own Python library. Installation went well, first tests (serial and parallel) look good. Here is what I want to do: I have my own time-stepping playground and I want petsc4py to be one of my backbones for the data types and (linear or non-linear, serial or parallel) solvers. I don't want to use PETSc's time-steppers, at least not for now. So, I saw in some examples, in particular the ones shipped with petsc4py, that the standard way of running one of PETSc's solvers is a bunch of setup routines, then setting the right-hand side and solve. Now, I don't want to rerun the whole setup part each time I call the solver. I know that I can change the right-hand side without having to do so, but what if I change a parameter of my operator like, say, the time-step size or some material parameter? Take the simplest case: Say I have my own implicit Euler written in Python. I know the right-hand side F of my ODE, so in each step I want to solve "I - dt*F". But the time-step changes every now and then, so I cannot pre-assemble everything once and for all (or I don't want to). What do I need to rerun before I can use the solver again, what can I reuse? Could I just assemble F and combine it with the identity and the parameter dt right before I call the solver? How would that look like? I'm pretty new to PETSc and to petsc4py, so please forgive any stupidity or ignorance in these questions. I'm happy to take any advice, links to examples or previous questions. Thanks! Kind regards -Robert- -- Dr. Robert Speck Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Tel: +49 2461 61 1644 Fax: +49 2461 61 6656 Email: r.speck at fz-juelich.de Website: http://www.fz-juelich.de/ias/jsc/speck_r PinT: http://www.fz-juelich.de/ias/jsc/pint ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ From knepley at gmail.com Thu Apr 5 05:53:00 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 5 Apr 2018 06:53:00 -0400 Subject: [petsc-users] petsc4py: reuse setup for multiple solver calls? In-Reply-To: References: Message-ID: On Thu, Apr 5, 2018 at 6:39 AM, Robert Speck wrote: > Hi! > > I would like to use petsc4py for my own Python library. Installation > went well, first tests (serial and parallel) look good. > > Here is what I want to do: I have my own time-stepping playground and I > want petsc4py to be one of my backbones for the data types and (linear > or non-linear, serial or parallel) solvers. I don't want to use PETSc's > time-steppers, at least not for now. So, I saw in some examples, in > particular the ones shipped with petsc4py, that the standard way of > running one of PETSc's solvers is a bunch of setup routines, then > setting the right-hand side and solve. > > Now, I don't want to rerun the whole setup part each time I call the > solver. I know that I can change the right-hand side without having to > do so, but what if I change a parameter of my operator like, say, the > time-step size or some material parameter? > > Take the simplest case: Say I have my own implicit Euler written in > Python. I know the right-hand side F of my ODE, so in each step I want > to solve "I - dt*F". But the time-step changes every now and then, so I > cannot pre-assemble everything once and for all (or I don't want to). > What do I need to rerun before I can use the solver again, what can I > reuse? Could I just assemble F and combine it with the identity and the > parameter dt right before I call the solver? How would that look like? > > I'm pretty new to PETSc and to petsc4py, so please forgive any stupidity > or ignorance in these questions. I'm happy to take any advice, links to > examples or previous questions. Thanks! > For linear solves which stay the same size, you just have to call SetOperators again with the new operator. For nonlinear solves which stay the same size, you do nothing. If the system size changes, it generally better to create the object. If that is hard, then you can use KSPReset(). Matt > Kind regards > -Robert- > > -- > Dr. Robert Speck > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > 52425 Juelich, Germany > > Tel: +49 2461 61 1644 > Fax: +49 2461 61 6656 > > Email: r.speck at fz-juelich.de > Website: http://www.fz-juelich.de/ias/jsc/speck_r > PinT: http://www.fz-juelich.de/ias/jsc/pint > > > > ------------------------------------------------------------ > ------------------------------------ > ------------------------------------------------------------ > ------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------ > ------------------------------------ > ------------------------------------------------------------ > ------------------------------------ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Apr 5 08:31:47 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 5 Apr 2018 08:31:47 -0500 Subject: [petsc-users] --download-hdf5 and then compiling cgns In-Reply-To: <1522919359888.15860@marin.nl> References: <1522329528872.65565@marin.nl> , <1522919359888.15860@marin.nl> Message-ID: Glad it works.. Perhaps its is simple fix to add in --download-cgns option to petsc configure.. [I see its a direct dependency from PETSc sources - src/dm/impls/plex/plexcgns.c ] Satish On Thu, 5 Apr 2018, Klaij, Christiaan wrote: > Satish, Gaetan, thanks for your suggestions. I've decided not to > use cmake for cgns, and that works fine with petsc's build of > hdf5. > > > Chris > > dr. ir. Christiaan Klaij | Senior Researcher | Research & Development > MARIN | T +31 317 49 33 44 | C.Klaij at marin.nl | www.marin.nl > > [LinkedIn] [YouTube] [Twitter] [Facebook] > MARIN news: Kom zaterdag 10 november naar de open dag in Wageningen! > > ________________________________ > From: Gaetan Kenway > Sent: Thursday, March 29, 2018 6:43 PM > To: petsc-users > Cc: Klaij, Christiaan > Subject: Re: [petsc-users] --download-hdf5 and then compiling cgns > > I have compiled CGNS3.3 with hdf from PETSc but not using cmake from CGNS. Here is what I used: > > git clone https://github.com/CGNS/CGNS.git > > cd CGNS/src > export CC=mpicc > export FC=mpif90 > export FCFLAGS='-fPIC -O3' > export CFLAGS='-fPIC -O3' > export LIBS='-lz -ldl' > > ./configure --with-hdf5=$PETSC_DIR/$PETSC_ARCH --enable-64bit --prefix=/u/wk/gkenway/.local --enable-parallel --enable-cgnstools=yes --enable-shared > make -j 10 > > Hope that helps > Gaetan > > On Thu, Mar 29, 2018 at 9:37 AM, Satish Balay > wrote: > --download-hdf5 uses 'autoconf' build of hdf5 - and not cmake > build. [hdf5 provides both build tools]. > > This must be the reason why hdf5Config.cmake and hdf5-config.cmake > files might be missing. > > You might be able to install hdf5 manually with cmake - and then build > cgns-3.3.1 with it. > > Satish > > > On Thu, 29 Mar 2018, Klaij, Christiaan wrote: > > > Satish, > > > > I've build petsc-3.8.3 with --download-hdf5 and this works > > fine. Now, I'm trying to compile cgns-3.3.1 and have it use the > > hdf5 library from petsc. Therefore, in the cmake configure I have > > set this option > > > > -D HDF5_DIR=/path/to/petsc/hdf5 > > > > but cmake gives this message: > > > > CMake Warning at CMakeLists.txt:207 (find_package): > > Could not find a package configuration file provided by "HDF5" with any of > > the following names: > > > > hdf5Config.cmake > > hdf5-config.cmake > > > > and then falls back to the system's hdf5. Indeed there are no > > such files in /path/to/petsc/hdf5. Any idea on how to proceed > > from here? > > > > Chris > > > > > > dr. ir. Christiaan Klaij | Senior Researcher | Research & Development > > MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl > > > > MARIN news: http://www.marin.nl/web/News/News-items/Comfort-draft-for-semisubmersible-yachts.htm > > > > > > > > > From C.Klaij at marin.nl Thu Apr 5 09:06:59 2018 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Thu, 5 Apr 2018 14:06:59 +0000 Subject: [petsc-users] --download-hdf5 and then compiling cgns In-Reply-To: References: <1522329528872.65565@marin.nl> , <1522919359888.15860@marin.nl>, Message-ID: <1522937219492.87250@marin.nl> That would be nice. The minimum for me would be ./configure \ --prefix=/path/to/myprefix \ --with-hdf5=/path/to/petsc/hdf5 \ --with-fortran=yes but I sure like to have their tools as well ./configure \ --prefix=/path/to/myprefix \ --with-hdf5=/path/to/petsc/hdf5 \ --with-fortran=yes \ --enable-cgnstools \ --datarootdir=/path/to/myprefix/share This is more tricky as it requires tcl and tk libs. Also datarootdir is supposed to be PREFIX/share by default, but there's a bug so I had to specify it explicitly. Chris dr. ir. Christiaan Klaij | Senior Researcher | Research & Development MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl MARIN news: http://www.marin.nl/web/News/News-items/Kom-zaterdag-10-november-naar-de-open-dag-in-Wageningen.htm ________________________________________ From: Satish Balay Sent: Thursday, April 05, 2018 3:31 PM To: Klaij, Christiaan Cc: Gaetan Kenway; petsc-users Subject: Re: [petsc-users] --download-hdf5 and then compiling cgns Glad it works.. Perhaps its is simple fix to add in --download-cgns option to petsc configure.. [I see its a direct dependency from PETSc sources - src/dm/impls/plex/plexcgns.c ] Satish On Thu, 5 Apr 2018, Klaij, Christiaan wrote: > Satish, Gaetan, thanks for your suggestions. I've decided not to > use cmake for cgns, and that works fine with petsc's build of > hdf5. > > > Chris > > dr. ir. Christiaan Klaij | Senior Researcher | Research & Development > MARIN | T +31 317 49 33 44 | C.Klaij at marin.nl | www.marin.nl > > [LinkedIn] [YouTube] [Twitter] [Facebook] > MARIN news: Kom zaterdag 10 november naar de open dag in Wageningen! > > ________________________________ > From: Gaetan Kenway > Sent: Thursday, March 29, 2018 6:43 PM > To: petsc-users > Cc: Klaij, Christiaan > Subject: Re: [petsc-users] --download-hdf5 and then compiling cgns > > I have compiled CGNS3.3 with hdf from PETSc but not using cmake from CGNS. Here is what I used: > > git clone https://github.com/CGNS/CGNS.git > > cd CGNS/src > export CC=mpicc > export FC=mpif90 > export FCFLAGS='-fPIC -O3' > export CFLAGS='-fPIC -O3' > export LIBS='-lz -ldl' > > ./configure --with-hdf5=$PETSC_DIR/$PETSC_ARCH --enable-64bit --prefix=/u/wk/gkenway/.local --enable-parallel --enable-cgnstools=yes --enable-shared > make -j 10 > > Hope that helps > Gaetan > > On Thu, Mar 29, 2018 at 9:37 AM, Satish Balay > wrote: > --download-hdf5 uses 'autoconf' build of hdf5 - and not cmake > build. [hdf5 provides both build tools]. > > This must be the reason why hdf5Config.cmake and hdf5-config.cmake > files might be missing. > > You might be able to install hdf5 manually with cmake - and then build > cgns-3.3.1 with it. > > Satish > > > On Thu, 29 Mar 2018, Klaij, Christiaan wrote: > > > Satish, > > > > I've build petsc-3.8.3 with --download-hdf5 and this works > > fine. Now, I'm trying to compile cgns-3.3.1 and have it use the > > hdf5 library from petsc. Therefore, in the cmake configure I have > > set this option > > > > -D HDF5_DIR=/path/to/petsc/hdf5 > > > > but cmake gives this message: > > > > CMake Warning at CMakeLists.txt:207 (find_package): > > Could not find a package configuration file provided by "HDF5" with any of > > the following names: > > > > hdf5Config.cmake > > hdf5-config.cmake > > > > and then falls back to the system's hdf5. Indeed there are no > > such files in /path/to/petsc/hdf5. Any idea on how to proceed > > from here? > > > > Chris > > > > > > dr. ir. Christiaan Klaij | Senior Researcher | Research & Development > > MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl > > > > MARIN news: http://www.marin.nl/web/News/News-items/Comfort-draft-for-semisubmersible-yachts.htm > > > > > > > > > From rlmackie862 at gmail.com Thu Apr 5 10:03:13 2018 From: rlmackie862 at gmail.com (Randall Mackie) Date: Thu, 5 Apr 2018 17:03:13 +0200 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel Message-ID: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> Dear PETSc users, I?m curious if anyone else experiences problems using DMDAVecGetArrayF90 in conjunction with Intel compilers? We have had many problems (typically 11 SEGV segmentation violations) when PETSc is compiled in optimize mode (with various combinations of options). These same codes run valgrind clean with gfortran, so I assume this is an Intel bug, but before we submit a bug report I wanted to see if anyone else had similar experiences? We have basically gone back and replaced our calls to DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers into a ?local? subroutine that works fine. In case anyone is curious, the attached test code shows this behavior when PETSc is compiled with the following options: ./configure \ --with-clean=1 \ --with-debugging=0 \ --with-fortran=1 \ --with-64-bit-indices \ --download-mpich=../mpich-3.3a2.tar.gz \ --with-blas-lapack-dir=/opt/intel/mkl \ --with-cc=icc \ --with-fc=ifort \ --with-cxx=icc \ --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2? \ Thanks, Randy M. -------------- next part -------------- A non-text attachment was scrubbed... Name: cmd_test Type: application/octet-stream Size: 63 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: makefile Type: application/octet-stream Size: 370 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.F90 Type: application/octet-stream Size: 2507 bytes Desc: not available URL: From Fabian.Jakub at physik.uni-muenchen.de Thu Apr 5 11:49:15 2018 From: Fabian.Jakub at physik.uni-muenchen.de (Fabian.Jakub) Date: Thu, 5 Apr 2018 18:49:15 +0200 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: <54E1989A.1060507@dkrz.de> References: <54C229DF.1090703@physik.uni-muenchen.de> <54C249CF.8090303@dkrz.de> <54C763C3.7010004@dkrz.de> <54C8A451.4080503@dkrz.de> <54C8E852.7000103@dkrz.de> <54D21632.1070203@physik.uni-muenchen.de> <54D317AB.1000502@dkrz.de> <54D341D8.6090906@physik.uni-muenchen.de> <54D36AC6.60303@dkrz.de> <54DE22D4.6020809@physik.uni-muenchen.de> <54E1989A.1060507@dkrz.de> Message-ID: Hi, Not sure if this is related to your current issues but I once had issues with this particular function with IBM XLF compilers but never got to a solution because... a) VecGetArrayF90 also worked for me b) and the IBM Machine got switched off Anyway, I did not have any issues since then... not with intel nor with gfortran compilers. The helpdesk back then created a petsc bug report but I think the issue was never resolved... For the sake of completeness, I am attaching the messages we had then, maybe the debugging efforts are related to your problems? Good Luck, Fabian On 04/05/2018 05:03 PM, Randall Mackie wrote: > Dear PETSc users, > > I?m curious if anyone else experiences problems using DMDAVecGetArrayF90 in conjunction with Intel compilers? > We have had many problems (typically 11 SEGV segmentation violations) when PETSc is compiled in optimize mode (with various combinations of options). > These same codes run valgrind clean with gfortran, so I assume this is an Intel bug, but before we submit a bug report I wanted to see if anyone else had similar experiences? > We have basically gone back and replaced our calls to DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers into a ?local? subroutine that works fine. > > In case anyone is curious, the attached test code shows this behavior when PETSc is compiled with the following options: > > ./configure \ > --with-clean=1 \ > --with-debugging=0 \ > --with-fortran=1 \ > --with-64-bit-indices \ > --download-mpich=../mpich-3.3a2.tar.gz \ > --with-blas-lapack-dir=/opt/intel/mkl \ > --with-cc=icc \ > --with-fc=ifort \ > --with-cxx=icc \ > --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2? \ > > > > Thanks, Randy M. > -------------- next part -------------- An embedded message was scrubbed... From: "Fabian.Jakub" Subject: Problems Using Petsc at Blizzard Date: Fri, 23 Jan 2015 12:00:47 +0100 Size: 18589 URL: -------------- next part -------------- An embedded message was scrubbed... From: Hendryk Bockelmann Subject: Re: Problems Using Petsc at Blizzard Date: Tue, 27 Jan 2015 11:09:07 +0100 Size: 4151 URL: -------------- next part -------------- An embedded message was scrubbed... From: Hendryk Bockelmann Subject: Re: Problems Using Petsc at Blizzard Date: Wed, 28 Jan 2015 09:56:49 +0100 Size: 3691 URL: -------------- next part -------------- An embedded message was scrubbed... From: Hendryk Bockelmann Subject: Re: Problems Using Petsc at Blizzard Date: Thu, 05 Feb 2015 14:06:14 +0100 Size: 4509 URL: -------------- next part -------------- An embedded message was scrubbed... From: "Fabian.Jakub" Subject: Re: Problems Using Petsc at Blizzard Date: Fri, 13 Feb 2015 17:14:12 +0100 Size: 3095 URL: -------------- next part -------------- An embedded message was scrubbed... From: Hendryk Bockelmann Subject: Re: Problems Using Petsc at Blizzard Date: Mon, 16 Feb 2015 08:13:30 +0100 Size: 5059 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From cormac.garvey at inl.gov Thu Apr 5 12:21:35 2018 From: cormac.garvey at inl.gov (Garvey, Cormac T) Date: Thu, 5 Apr 2018 11:21:35 -0600 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: I made some changes to the falcon cluster environment. Please try the default git (i.e Without loading a module) and see if petsc will install correctly. Thanks, Cormac. On Wed, Apr 4, 2018 at 5:09 PM, Fande Kong wrote: > The default git gives me: > > *Could not execute "['git', 'rev-parse', '--git-dir']"* > > when I am configuring PETSc. > > The manually loaded *gits* work just fine. > > > Fande, > > > On Wed, Apr 4, 2018 at 5:04 PM, Garvey, Cormac T > wrote: > >> I though it was fixed, yes I will look into it again. >> >> Do you get an error just doing a git clone on falcon1 and falcon2? >> >> On Wed, Apr 4, 2018 at 4:48 PM, Kong, Fande wrote: >> >>> module load git/2.16.2-GCCcore-5.4.0" also works. >>> >>> Could you somehow make the default git work as well? Hence we do not >>> need to have this extra "module load for git" >>> >>> Fande, >>> >>> On Wed, Apr 4, 2018 at 4:43 PM, Kong, Fande wrote: >>> >>>> Thanks, Cormac, >>>> >>>> *module load git/1.8.5.2-GCC-4.8.3 * >>>> >>>> works for me. >>>> >>>> Did not try "module load git/2.16.2-GCCcore-5.4.0" yet. >>>> >>>> I will try, and get it back here. >>>> >>>> >>>> >>>> Fande >>>> >>>> On Wed, Apr 4, 2018 at 4:39 PM, Garvey, Cormac T >>> > wrote: >>>> >>>>> >>>>> We needed to rebuilt git on the INL falcon cluster because github >>>>> server changed such that it no longer accepted TLSv1. >>>>> >>>>> The default git on the falcon cluster /usr/bin/git is just a wrapper >>>>> script, so users would not need to load any modules to >>>>> use git. >>>>> >>>>> When you load load git on falcon1 or falcon2 does it still fail? >>>>> >>>>> module load git/2.16.2-GCCcore-5.4.0 >>>>> >>>>> Thanks, >>>>> Cormac. >>>>> >>>>> >>>>> >>>>> On Wed, Apr 4, 2018 at 4:28 PM, Kong, Fande >>>>> wrote: >>>>> >>>>>> Hi Cormac, >>>>>> >>>>>> Do you know anything on "git"? How did you guys build git on the >>>>>> falcon1? The default git on Falcon1 does not work with petsc any more. >>>>>> >>>>>> >>>>>> Fande, >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Apr 4, 2018 at 4:20 PM, Satish Balay >>>>>> wrote: >>>>>> >>>>>>> Ok - I don't think I have access to this OS. >>>>>>> >>>>>>> And I see its from 2009 [sure its enterprise os - with regular >>>>>>> backport updates] >>>>>>> >>>>>>> But its wierd that you have such a new version of git at /usr/bin/git >>>>>>> >>>>>>> From what we know so far - the problem appears to be some bad >>>>>>> interaction of python-2.6 with this old OS [i.e old glibc etc..] - >>>>>>> and >>>>>>> this new git version [binary built locally or on a different OS and >>>>>>> installed locally ?]. >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> On Wed, 4 Apr 2018, Kong, Fande wrote: >>>>>>> >>>>>>> > moose]$ uname -a >>>>>>> > Linux falcon1 3.0.101-108.13-default #1 SMP Wed Oct 11 12:30:40 >>>>>>> UTC 2017 >>>>>>> > (294ccfe) x86_64 x86_64 x86_64 GNU/Linux >>>>>>> > >>>>>>> > >>>>>>> > moose]$ lsb_release -a >>>>>>> > LSB Version: >>>>>>> > core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86 >>>>>>> _64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:deskto >>>>>>> p-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics >>>>>>> -3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4 >>>>>>> .0-noarch >>>>>>> > Distributor ID: SUSE LINUX >>>>>>> > Description: SUSE Linux Enterprise Server 11 (x86_64) >>>>>>> > Release: 11 >>>>>>> > Codename: n/a >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > On Wed, Apr 4, 2018 at 4:08 PM, Satish Balay >>>>>>> wrote: >>>>>>> > >>>>>>> > > On Wed, 4 Apr 2018, Satish Balay wrote: >>>>>>> > > >>>>>>> > > > On Wed, 4 Apr 2018, Satish Balay wrote: >>>>>>> > > > >>>>>>> > > > > was your '2.16.2' version installed from source? >>>>>>> > > > >>>>>>> > > > >>> >>>>>>> > > > Checking for program /usr/bin/git...found >>>>>>> > > > Defined make macro "GIT" to "git" >>>>>>> > > > Executing: git --version >>>>>>> > > > stdout: git version 2.16.2 >>>>>>> > > > <<<< >>>>>>> > > > >>>>>>> > > > I gues its the OS default package >>>>>>> > > > >>>>>>> > > > >>>>> >>>>>>> > > > Machine platform: >>>>>>> > > > ('Linux', 'falcon1', '3.0.101-108.13-default', '#1 SMP Wed Oct >>>>>>> 11 >>>>>>> > > 12:30:40 UTC 2017 (294ccfe)', 'x86_64', 'x86_64') >>>>>>> > > > Python version: >>>>>>> > > > 2.6.9 (unknown, Aug 5 2016, 11:15:31) >>>>>>> > > > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] >>>>>>> > > > <<<< >>>>>>> > > > >>>>>>> > > > What OS/version is on this machine? I can try reproducing in a >>>>>>> VM >>>>>>> > > >>>>>>> > > It is strange that the kernel is old [3.0 - perhaps LTS OS] , >>>>>>> python is >>>>>>> > > old [2.6] - but git is new? [2.16?] >>>>>>> > > >>>>>>> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ >>>>>>> > > mirrors.edge.kernel.org_pub_software_scm_git_&d=DwIBAg&c= >>>>>>> > > 54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_ >>>>>>> > > JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=WF6ZxIh9Z9Hh2kYY0w70aNrgf >>>>>>> Picp6 >>>>>>> > > kgIh5BvezPiEY&s=KWMsO7XC0-pQuQ_mD03tNJWEpxSTATlZW_DmX0QofGw&e= >>>>>>> > > git-2.16.2.tar.gz 16-Feb-2018 >>>>>>> 17:48 >>>>>>> > > 7M >>>>>>> > > >>>>>>> > > Satish >>>>>>> > > >>>>>>> > >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Cormac Garvey >>>>> HPC Software Consultant >>>>> Scientific Computing >>>>> Idaho National Laboratory >>>>> Ph: 208-526-6294 >>>>> >>>>> >>>> >>> >> >> >> -- >> Cormac Garvey >> HPC Software Consultant >> Scientific Computing >> Idaho National Laboratory >> Ph: 208-526-6294 >> >> > -- Cormac Garvey HPC Software Consultant Scientific Computing Idaho National Laboratory Ph: 208-526-6294 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Thu Apr 5 13:47:04 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Thu, 5 Apr 2018 12:47:04 -0600 Subject: [petsc-users] Could not execute "['git', 'rev-parse', '--git-dir']" In-Reply-To: References: Message-ID: Hi Cormac, Thanks so much! It is working now. Fande, On Thu, Apr 5, 2018 at 11:21 AM, Garvey, Cormac T wrote: > I made some changes to the falcon cluster environment. Please try the > default git (i.e Without loading a module) and > see if petsc will install correctly. > > Thanks, > Cormac. > > On Wed, Apr 4, 2018 at 5:09 PM, Fande Kong wrote: > >> The default git gives me: >> >> *Could not execute "['git', 'rev-parse', '--git-dir']"* >> >> when I am configuring PETSc. >> >> The manually loaded *gits* work just fine. >> >> >> Fande, >> >> >> On Wed, Apr 4, 2018 at 5:04 PM, Garvey, Cormac T >> wrote: >> >>> I though it was fixed, yes I will look into it again. >>> >>> Do you get an error just doing a git clone on falcon1 and falcon2? >>> >>> On Wed, Apr 4, 2018 at 4:48 PM, Kong, Fande wrote: >>> >>>> module load git/2.16.2-GCCcore-5.4.0" also works. >>>> >>>> Could you somehow make the default git work as well? Hence we do not >>>> need to have this extra "module load for git" >>>> >>>> Fande, >>>> >>>> On Wed, Apr 4, 2018 at 4:43 PM, Kong, Fande wrote: >>>> >>>>> Thanks, Cormac, >>>>> >>>>> *module load git/1.8.5.2-GCC-4.8.3 * >>>>> >>>>> works for me. >>>>> >>>>> Did not try "module load git/2.16.2-GCCcore-5.4.0" yet. >>>>> >>>>> I will try, and get it back here. >>>>> >>>>> >>>>> >>>>> Fande >>>>> >>>>> On Wed, Apr 4, 2018 at 4:39 PM, Garvey, Cormac T < >>>>> cormac.garvey at inl.gov> wrote: >>>>> >>>>>> >>>>>> We needed to rebuilt git on the INL falcon cluster because github >>>>>> server changed such that it no longer accepted TLSv1. >>>>>> >>>>>> The default git on the falcon cluster /usr/bin/git is just a wrapper >>>>>> script, so users would not need to load any modules to >>>>>> use git. >>>>>> >>>>>> When you load load git on falcon1 or falcon2 does it still fail? >>>>>> >>>>>> module load git/2.16.2-GCCcore-5.4.0 >>>>>> >>>>>> Thanks, >>>>>> Cormac. >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Apr 4, 2018 at 4:28 PM, Kong, Fande >>>>>> wrote: >>>>>> >>>>>>> Hi Cormac, >>>>>>> >>>>>>> Do you know anything on "git"? How did you guys build git on the >>>>>>> falcon1? The default git on Falcon1 does not work with petsc any more. >>>>>>> >>>>>>> >>>>>>> Fande, >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Apr 4, 2018 at 4:20 PM, Satish Balay >>>>>>> wrote: >>>>>>> >>>>>>>> Ok - I don't think I have access to this OS. >>>>>>>> >>>>>>>> And I see its from 2009 [sure its enterprise os - with regular >>>>>>>> backport updates] >>>>>>>> >>>>>>>> But its wierd that you have such a new version of git at >>>>>>>> /usr/bin/git >>>>>>>> >>>>>>>> From what we know so far - the problem appears to be some bad >>>>>>>> interaction of python-2.6 with this old OS [i.e old glibc etc..] - >>>>>>>> and >>>>>>>> this new git version [binary built locally or on a different OS and >>>>>>>> installed locally ?]. >>>>>>>> >>>>>>>> Satish >>>>>>>> >>>>>>>> On Wed, 4 Apr 2018, Kong, Fande wrote: >>>>>>>> >>>>>>>> > moose]$ uname -a >>>>>>>> > Linux falcon1 3.0.101-108.13-default #1 SMP Wed Oct 11 12:30:40 >>>>>>>> UTC 2017 >>>>>>>> > (294ccfe) x86_64 x86_64 x86_64 GNU/Linux >>>>>>>> > >>>>>>>> > >>>>>>>> > moose]$ lsb_release -a >>>>>>>> > LSB Version: >>>>>>>> > core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86 >>>>>>>> _64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:deskto >>>>>>>> p-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics >>>>>>>> -3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4 >>>>>>>> .0-noarch >>>>>>>> > Distributor ID: SUSE LINUX >>>>>>>> > Description: SUSE Linux Enterprise Server 11 (x86_64) >>>>>>>> > Release: 11 >>>>>>>> > Codename: n/a >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > On Wed, Apr 4, 2018 at 4:08 PM, Satish Balay >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > > On Wed, 4 Apr 2018, Satish Balay wrote: >>>>>>>> > > >>>>>>>> > > > On Wed, 4 Apr 2018, Satish Balay wrote: >>>>>>>> > > > >>>>>>>> > > > > was your '2.16.2' version installed from source? >>>>>>>> > > > >>>>>>>> > > > >>> >>>>>>>> > > > Checking for program /usr/bin/git...found >>>>>>>> > > > Defined make macro "GIT" to "git" >>>>>>>> > > > Executing: git --version >>>>>>>> > > > stdout: git version 2.16.2 >>>>>>>> > > > <<<< >>>>>>>> > > > >>>>>>>> > > > I gues its the OS default package >>>>>>>> > > > >>>>>>>> > > > >>>>> >>>>>>>> > > > Machine platform: >>>>>>>> > > > ('Linux', 'falcon1', '3.0.101-108.13-default', '#1 SMP Wed >>>>>>>> Oct 11 >>>>>>>> > > 12:30:40 UTC 2017 (294ccfe)', 'x86_64', 'x86_64') >>>>>>>> > > > Python version: >>>>>>>> > > > 2.6.9 (unknown, Aug 5 2016, 11:15:31) >>>>>>>> > > > [GCC 4.3.4 [gcc-4_3-branch revision 152973]] >>>>>>>> > > > <<<< >>>>>>>> > > > >>>>>>>> > > > What OS/version is on this machine? I can try reproducing in >>>>>>>> a VM >>>>>>>> > > >>>>>>>> > > It is strange that the kernel is old [3.0 - perhaps LTS OS] , >>>>>>>> python is >>>>>>>> > > old [2.6] - but git is new? [2.16?] >>>>>>>> > > >>>>>>>> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ >>>>>>>> > > mirrors.edge.kernel.org_pub_software_scm_git_&d=DwIBAg&c= >>>>>>>> > > 54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_ >>>>>>>> > > JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=WF6ZxIh9Z9Hh2kYY0w70aNrgf >>>>>>>> Picp6 >>>>>>>> > > kgIh5BvezPiEY&s=KWMsO7XC0-pQuQ_mD03tNJWEpxSTATlZW_DmX0QofGw&e= >>>>>>>> > > git-2.16.2.tar.gz 16-Feb-2018 >>>>>>>> 17:48 >>>>>>>> > > 7M >>>>>>>> > > >>>>>>>> > > Satish >>>>>>>> > > >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Cormac Garvey >>>>>> HPC Software Consultant >>>>>> Scientific Computing >>>>>> Idaho National Laboratory >>>>>> Ph: 208-526-6294 >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Cormac Garvey >>> HPC Software Consultant >>> Scientific Computing >>> Idaho National Laboratory >>> Ph: 208-526-6294 >>> >>> >> > > > -- > Cormac Garvey > HPC Software Consultant > Scientific Computing > Idaho National Laboratory > Ph: 208-526-6294 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedmud at gmail.com Thu Apr 5 17:45:07 2018 From: friedmud at gmail.com (Derek Gaston) Date: Thu, 05 Apr 2018 22:45:07 +0000 Subject: [petsc-users] [petsc-dev] ftp.mcs.anl.gov is down - breaking petsc builds with --download-package In-Reply-To: References: Message-ID: Is ftp access down again? The download links on the PETSc website appear to just stall... Derek On Thu, Mar 29, 2018 at 9:52 AM Satish Balay wrote: > ftp.mcs.anl.gov is now back online [serving ftp:// - http:// is not yet > enabled] > > petsc configure builds with --download-package option should now work. > > For direct tarball download - switching the URL from > http://ftp.mcs.anl.gov to ftp://ftp.mcs.anl.gov notation is the > temporary workaround. For eg: > > change: > http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.8.4.tar.gz > to: > ftp://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.8.4.tar.gz > > Satish > > On Thu, 29 Mar 2018, Satish Balay wrote: > > > > > We've been informed by the site admin that ftp.mcs.anl.gov is taken > > down due to 'cyber incident' - there is no ETA on when it will be > > back. > > > > We've been using it the primary host for distributing petsc tarballs > > and hosting some externalpackage tarballs - so some of the petsc buils > > with '--download-package' option are failing. > > > > Obtaining petsc [and some most externalpackages] via git should work. > > > > I will update when ftp.mcs.anl.gov is brought back up. > > > > Satish > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Apr 5 17:54:46 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 5 Apr 2018 17:54:46 -0500 Subject: [petsc-users] [petsc-dev] ftp.mcs.anl.gov is down - breaking petsc builds with --download-package In-Reply-To: References: Message-ID: Both ftp and http work fine for me.. balay at asterix /home/balay/petsc.x/petsc (balay/test=) $ lftp ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages cd ok, cwd=/pub/petsc/externalpackages lftp ftp.mcs.anl.gov:/pub/petsc/externalpackages> ls |head -rw-r----- 1 ftp ftp 0 Nov 18 2016 ==33186== -rw-r----- 1 ftp ftp 828346 May 2 2013 Chaco-2.2-p1.tar.gz -rw-r----- 1 ftp ftp 1981220 Sep 23 2014 Chaco-2.2-p2.tar.gz -rw-r----- 1 ftp ftp 828295 Oct 23 2009 Chaco-2.2.tar.gz -rw-r----- 1 ftp ftp 1685576 Feb 16 2015 Elemental-0.85-p1.tar.gz -rw-r----- 1 ftp ftp 112597 Mar 28 2012 Generator.tar.gz -rw-r----- 1 ftp ftp 2471411 Nov 18 2011 MUMPS_4.10.0-p1.tar.gz -rw-r----- 1 ftp ftp 2471465 Feb 26 2012 MUMPS_4.10.0-p2.tar.gz -rw-r----- 1 ftp ftp 2440565 Mar 15 2012 MUMPS_4.10.0-p3.tar.gz -rw-r----- 1 ftp ftp 2439922 Aug 17 2011 MUMPS_4.10.0.tar.gz lftp ftp.mcs.anl.gov:/pub/petsc/externalpackages> exit balay at asterix /home/balay/petsc.x/petsc (balay/test=) $ lftp http://ftp.mcs.anl.gov/pub/petsc/externalpackages cd: received redirection to `http://ftp.mcs.anl.gov/pub/petsc/externalpackages/' cd ok, cwd=/pub/petsc/externalpackages lftp ftp.mcs.anl.gov:/pub/petsc/externalpackages> ls |head drwxr-xr-x -- .. -rw-r--r-- 0 2016-11-18 11:03 ==33186== -rw-r--r-- 809K 2013-05-01 22:00 Chaco-2.2-p1.tar.gz -rw-r--r-- 1.9M 2014-09-23 13:52 Chaco-2.2-p2.tar.gz -rw-r--r-- 809K 2009-10-22 19:53 Chaco-2.2.tar.gz -rw-r--r-- 1.6M 2015-02-16 14:21 Elemental-0.85-p1.tar.gz -rw-r--r-- 110K 2012-03-27 19:22 Generator.tar.gz -rw-r--r-- 2.4M 2011-11-18 13:59 MUMPS_4.10.0-p1.tar.gz -rw-r--r-- 2.4M 2012-02-25 21:05 MUMPS_4.10.0-p2.tar.gz -rw-r--r-- 2.3M 2012-03-14 22:57 MUMPS_4.10.0-p3.tar.gz ls: Broken pipe lftp ftp.mcs.anl.gov:/pub/petsc/externalpackages> exit Satish On Thu, 5 Apr 2018, Derek Gaston wrote: > Is ftp access down again? The download links on the PETSc website appear > to just stall... > > Derek > > On Thu, Mar 29, 2018 at 9:52 AM Satish Balay wrote: > > > ftp.mcs.anl.gov is now back online [serving ftp:// - http:// is not yet > > enabled] > > > > petsc configure builds with --download-package option should now work. > > > > For direct tarball download - switching the URL from > > http://ftp.mcs.anl.gov to ftp://ftp.mcs.anl.gov notation is the > > temporary workaround. For eg: > > > > change: > > http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.8.4.tar.gz > > to: > > ftp://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.8.4.tar.gz > > > > Satish > > > > On Thu, 29 Mar 2018, Satish Balay wrote: > > > > > > > > We've been informed by the site admin that ftp.mcs.anl.gov is taken > > > down due to 'cyber incident' - there is no ETA on when it will be > > > back. > > > > > > We've been using it the primary host for distributing petsc tarballs > > > and hosting some externalpackage tarballs - so some of the petsc buils > > > with '--download-package' option are failing. > > > > > > Obtaining petsc [and some most externalpackages] via git should work. > > > > > > I will update when ftp.mcs.anl.gov is brought back up. > > > > > > Satish > > > > > > > > > > > From friedmud at gmail.com Thu Apr 5 17:57:57 2018 From: friedmud at gmail.com (Derek Gaston) Date: Thu, 05 Apr 2018 22:57:57 +0000 Subject: [petsc-users] [petsc-dev] ftp.mcs.anl.gov is down - breaking petsc builds with --download-package In-Reply-To: References: Message-ID: Thanks for getting back to me - must be something about my environment here (ftp might even be blocked currently - although I did try from two different networks). I was able to get what I needed by using "http" for now. Sorry for the traffic. Derek On Thu, Apr 5, 2018 at 4:54 PM Satish Balay wrote: > Both ftp and http work fine for me.. > > balay at asterix /home/balay/petsc.x/petsc (balay/test=) > $ lftp ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages > cd ok, cwd=/pub/petsc/externalpackages > lftp ftp.mcs.anl.gov:/pub/petsc/externalpackages> ls |head > -rw-r----- 1 ftp ftp 0 Nov 18 2016 ==33186== > -rw-r----- 1 ftp ftp 828346 May 2 2013 Chaco-2.2-p1.tar.gz > -rw-r----- 1 ftp ftp 1981220 Sep 23 2014 Chaco-2.2-p2.tar.gz > -rw-r----- 1 ftp ftp 828295 Oct 23 2009 Chaco-2.2.tar.gz > -rw-r----- 1 ftp ftp 1685576 Feb 16 2015 > Elemental-0.85-p1.tar.gz > -rw-r----- 1 ftp ftp 112597 Mar 28 2012 Generator.tar.gz > -rw-r----- 1 ftp ftp 2471411 Nov 18 2011 > MUMPS_4.10.0-p1.tar.gz > -rw-r----- 1 ftp ftp 2471465 Feb 26 2012 > MUMPS_4.10.0-p2.tar.gz > -rw-r----- 1 ftp ftp 2440565 Mar 15 2012 > MUMPS_4.10.0-p3.tar.gz > -rw-r----- 1 ftp ftp 2439922 Aug 17 2011 MUMPS_4.10.0.tar.gz > lftp ftp.mcs.anl.gov:/pub/petsc/externalpackages> exit > balay at asterix /home/balay/petsc.x/petsc (balay/test=) > $ lftp http://ftp.mcs.anl.gov/pub/petsc/externalpackages > cd: received redirection to ` > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/' > cd ok, cwd=/pub/petsc/externalpackages > lftp ftp.mcs.anl.gov:/pub/petsc/externalpackages> ls |head > drwxr-xr-x -- .. > -rw-r--r-- 0 2016-11-18 11:03 ==33186== > -rw-r--r-- 809K 2013-05-01 22:00 Chaco-2.2-p1.tar.gz > -rw-r--r-- 1.9M 2014-09-23 13:52 Chaco-2.2-p2.tar.gz > -rw-r--r-- 809K 2009-10-22 19:53 Chaco-2.2.tar.gz > -rw-r--r-- 1.6M 2015-02-16 14:21 Elemental-0.85-p1.tar.gz > -rw-r--r-- 110K 2012-03-27 19:22 Generator.tar.gz > -rw-r--r-- 2.4M 2011-11-18 13:59 MUMPS_4.10.0-p1.tar.gz > -rw-r--r-- 2.4M 2012-02-25 21:05 MUMPS_4.10.0-p2.tar.gz > -rw-r--r-- 2.3M 2012-03-14 22:57 MUMPS_4.10.0-p3.tar.gz > ls: Broken pipe > lftp ftp.mcs.anl.gov:/pub/petsc/externalpackages> exit > > > Satish > > On Thu, 5 Apr 2018, Derek Gaston wrote: > > > Is ftp access down again? The download links on the PETSc website appear > > to just stall... > > > > Derek > > > > On Thu, Mar 29, 2018 at 9:52 AM Satish Balay wrote: > > > > > ftp.mcs.anl.gov is now back online [serving ftp:// - http:// is not > yet > > > enabled] > > > > > > petsc configure builds with --download-package option should now work. > > > > > > For direct tarball download - switching the URL from > > > http://ftp.mcs.anl.gov to ftp://ftp.mcs.anl.gov notation is the > > > temporary workaround. For eg: > > > > > > change: > > > http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.8.4.tar.gz > > > to: > > > ftp://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.8.4.tar.gz > > > > > > Satish > > > > > > On Thu, 29 Mar 2018, Satish Balay wrote: > > > > > > > > > > > We've been informed by the site admin that ftp.mcs.anl.gov is taken > > > > down due to 'cyber incident' - there is no ETA on when it will be > > > > back. > > > > > > > > We've been using it the primary host for distributing petsc tarballs > > > > and hosting some externalpackage tarballs - so some of the petsc > buils > > > > with '--download-package' option are failing. > > > > > > > > Obtaining petsc [and some most externalpackages] via git should work. > > > > > > > > I will update when ftp.mcs.anl.gov is brought back up. > > > > > > > > Satish > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernardomartinsrocha at gmail.com Thu Apr 5 18:16:36 2018 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Thu, 5 Apr 2018 20:16:36 -0300 Subject: [petsc-users] DMPlexGetCone in Fortran Message-ID: Hello everyone, I've been trying to use DMPlex in a Fortran 77 application (unfortunately this is legacy code and I can't move to F90). Is there a way to use DMPlexGetCone on it? Although the documentation online says it is only available for F90, I'm wondering if there is a trick like the ones in the users manual for routines that return an array. I tried, but I'm not sure it it is right. Best regards, Bernardo M. Rocha -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Apr 5 18:21:53 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 5 Apr 2018 19:21:53 -0400 Subject: [petsc-users] DMPlexGetCone in Fortran In-Reply-To: References: Message-ID: On Thu, Apr 5, 2018 at 7:16 PM, Bernardo Rocha < bernardomartinsrocha at gmail.com> wrote: > Hello everyone, > > I've been trying to use DMPlex in a Fortran 77 application > (unfortunately this is legacy code and I can't move to F90). > > Is there a way to use DMPlexGetCone on it? > Although the documentation online says it is only available for F90, > I'm wondering if there is a trick like the ones in the users manual for > routines > that return an array. I tried, but I'm not sure it it is right. > We have standardized our array handling with F90, and I don't think we will go back to supporting new things for F77-style arrays. That being said, I think all you have to do is compile that source with an F90 compiler. You should not have to change anything. If you want the F90 array to look like an F77 array, just pass it as a function argument (I think this works). Thanks, Matt > Best regards, > Bernardo M. Rocha > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernardomartinsrocha at gmail.com Thu Apr 5 20:57:04 2018 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Thu, 5 Apr 2018 22:57:04 -0300 Subject: [petsc-users] DMPlexGetCone in Fortran In-Reply-To: References: Message-ID: OK, thanks a lot. I had to rename the files and add -ffixed-form as argument to the compiler. Best regards, Bernardo On Thu, Apr 5, 2018 at 8:21 PM, Matthew Knepley wrote: > On Thu, Apr 5, 2018 at 7:16 PM, Bernardo Rocha < > bernardomartinsrocha at gmail.com> wrote: > >> Hello everyone, >> >> I've been trying to use DMPlex in a Fortran 77 application >> (unfortunately this is legacy code and I can't move to F90). >> >> Is there a way to use DMPlexGetCone on it? >> Although the documentation online says it is only available for F90, >> I'm wondering if there is a trick like the ones in the users manual for >> routines >> that return an array. I tried, but I'm not sure it it is right. >> > > We have standardized our array handling with F90, and I don't think we > will go back > to supporting new things for F77-style arrays. > > That being said, I think all you have to do is compile that source with an > F90 compiler. You > should not have to change anything. If you want the F90 array to look like > an F77 array, > just pass it as a function argument (I think this works). > > Thanks, > > Matt > > >> Best regards, >> Bernardo M. Rocha >> > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.speck at fz-juelich.de Fri Apr 6 00:48:33 2018 From: r.speck at fz-juelich.de (Robert Speck) Date: Fri, 6 Apr 2018 07:48:33 +0200 Subject: [petsc-users] petsc4py: reuse setup for multiple solver calls? In-Reply-To: References: Message-ID: Thank you for your answer! Please see below for comments/questions. On 05.04.18 12:53, Matthew Knepley wrote: > On Thu, Apr 5, 2018 at 6:39 AM, Robert Speck > wrote: > > Hi! > > I would like to use petsc4py for my own Python library. Installation > went well, first tests (serial and parallel) look good. > > Here is what I want to do: I have my own time-stepping playground and I > want petsc4py to be one of my backbones for the data types and (linear > or non-linear, serial or parallel) solvers. I don't want to use PETSc's > time-steppers, at least not for now. So, I saw in some examples, in > particular the ones shipped with petsc4py, that the standard way of > running one of PETSc's solvers is a bunch of setup routines, then > setting the right-hand side and solve. > > Now, I don't want to rerun the whole setup part each time I call the > solver. I know that I can change the right-hand side without having to > do so, but what if I change a parameter of my operator like, say, the > time-step size or some material parameter? > > Take the simplest case: Say I have my own implicit Euler written in > Python. I know the right-hand side F of my ODE, so in each step I want > to solve "I - dt*F". But the time-step changes every now and then, so I > cannot pre-assemble everything once and for all (or I don't want to). > What do I need to rerun before I can use the solver again, what can I > reuse? Could I just assemble F and combine it with the identity and the > parameter dt right before I call the solver? How would that look like? > > I'm pretty new to PETSc and to petsc4py, so please forgive any stupidity > or ignorance in these questions. I'm happy to take any advice, links to > examples or previous questions. Thanks! > > > For linear solves which stay the same size, you just have to call > SetOperators > again with the new operator. OK, this sounds straightforward. Thanks! > > For nonlinear solves which stay the same size, you do nothing. "nothing" in terms of "nothing you can do" or "nothing you have to do"? > > If the system size changes, it generally better to create the object. What does this mean? Thanks again! -Robert- ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ From dave.mayhem23 at gmail.com Fri Apr 6 01:07:57 2018 From: dave.mayhem23 at gmail.com (Dave May) Date: Fri, 06 Apr 2018 06:07:57 +0000 Subject: [petsc-users] petsc4py: reuse setup for multiple solver calls? In-Reply-To: References: Message-ID: On Fri, 6 Apr 2018 at 07:48, Robert Speck wrote: > Thank you for your answer! Please see below for comments/questions. > > On 05.04.18 12:53, Matthew Knepley wrote: > > On Thu, Apr 5, 2018 at 6:39 AM, Robert Speck > > wrote: > > > > Hi! > > > > I would like to use petsc4py for my own Python library. Installation > > went well, first tests (serial and parallel) look good. > > > > Here is what I want to do: I have my own time-stepping playground > and I > > want petsc4py to be one of my backbones for the data types and > (linear > > or non-linear, serial or parallel) solvers. I don't want to use > PETSc's > > time-steppers, at least not for now. So, I saw in some examples, in > > particular the ones shipped with petsc4py, that the standard way of > > running one of PETSc's solvers is a bunch of setup routines, then > > setting the right-hand side and solve. > > > > Now, I don't want to rerun the whole setup part each time I call the > > solver. I know that I can change the right-hand side without having > to > > do so, but what if I change a parameter of my operator like, say, the > > time-step size or some material parameter? > > > > Take the simplest case: Say I have my own implicit Euler written in > > Python. I know the right-hand side F of my ODE, so in each step I > want > > to solve "I - dt*F". But the time-step changes every now and then, > so I > > cannot pre-assemble everything once and for all (or I don't want to). > > What do I need to rerun before I can use the solver again, what can I > > reuse? Could I just assemble F and combine it with the identity and > the > > parameter dt right before I call the solver? How would that look > like? > > > > I'm pretty new to PETSc and to petsc4py, so please forgive any > stupidity > > or ignorance in these questions. I'm happy to take any advice, links > to > > examples or previous questions. Thanks! > > > > > > For linear solves which stay the same size, you just have to call > > SetOperators > > again with the new operator. > > OK, this sounds straightforward. Thanks! > > > > > For nonlinear solves which stay the same size, you do nothing. > > "nothing" in terms of "nothing you can do" or "nothing you have to do"? Nothing you have to do. > > > > > If the system size changes, it generally better to create the object. > > What does this mean? Possibly a typo - Matt is advising to "re-create" the objects if the system sizes change. Petsc objects are typically not dynamic with respect to their size (e.g. Mat, Vec), however creating new instances is generally cheap (matrices can be the exception if you don't preallocate) > > Thanks again! > -Robert- > > > > > > > ------------------------------------------------------------------------------------------------ > > ------------------------------------------------------------------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > > ------------------------------------------------------------------------------------------------ > > ------------------------------------------------------------------------------------------------ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yongxiang27 at gmail.com Fri Apr 6 07:44:50 2018 From: yongxiang27 at gmail.com (Yongxiang Wu) Date: Fri, 6 Apr 2018 14:44:50 +0200 Subject: [petsc-users] Scatter scipy Sparse matrix to petsc local processor Message-ID: Hello, everyone, I already have a scipy sparse square matrix L0 . Since my problem is large, parallel run is preferred. My Question is, how can I scatter my L0 to each of the processors? In the following code, I can get the indices of the localized part of the matrix. In the tutorial, the matrix element are directly assign with value, but in my case, the matrix are so large, assign each element in loop (commented code) is not efficient. So if any function would do the mpi scatter work? With regards and Thanks. import sys, slepc4py slepc4py.init(sys.argv) from petsc4py import PETSc from slepc4py import SLEPc opts = PETSc.Options() opts.setValue('-st_pc_factor_mat_solver_package','mumps') A = PETSc.Mat().createAIJ(size=L0.shape,comm=PETSc.COMM_WORLD) A.setUp() Istart, Iend = A.getOwnershipRange()# for I in range(Istart,Iend):# for J in range(0,L0.shape[0]):# A[I,J] = L0[I,J] The flowing code, would make the assignment from the scipy sparse matrix L0 to PETSc matrix A. But this would only work for one process. A = PETSc.Mat().createAIJ(size=L0.shape, csr=(L0.indptr, L0.indices, L0.data), comm=PETSc.COMM_WORLD) -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 6 08:00:43 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 6 Apr 2018 09:00:43 -0400 Subject: [petsc-users] Scatter scipy Sparse matrix to petsc local processor In-Reply-To: References: Message-ID: On Fri, Apr 6, 2018 at 8:44 AM, Yongxiang Wu wrote: > Hello, everyone, > > I already have a scipy sparse square matrix L0 . Since my problem is > large, parallel run is preferred. My Question is, how can I scatter my L0 to > each of the processors? In the following code, I can get the indices of the > localized part of the matrix. In the tutorial, the matrix element are > directly assign with value, but in my case, the matrix are so large, assign > each element in loop (commented code) is not efficient. So if any function > would do the mpi scatter work? > Hi Yongxiang, It would be really anomalous for what you propose to results in any speedup at all. If the matrix is large, it will not fit on one process. Any speedup from using more processes will be eaten up by the time to communicate the matrix. I would focus on generating the matrix in parallel. Thanks, Matt > With regards and Thanks. > > import sys, slepc4py > slepc4py.init(sys.argv) > from petsc4py import PETSc > from slepc4py import SLEPc > > opts = PETSc.Options() > opts.setValue('-st_pc_factor_mat_solver_package','mumps') > > A = PETSc.Mat().createAIJ(size=L0.shape,comm=PETSc.COMM_WORLD) > A.setUp() > > Istart, Iend = A.getOwnershipRange()# for I in range(Istart,Iend):# for J in range(0,L0.shape[0]):# A[I,J] = L0[I,J] > > The flowing code, would make the assignment from the scipy sparse matrix > L0 to PETSc matrix A. But this would only work for one process. > > A = PETSc.Mat().createAIJ(size=L0.shape, > csr=(L0.indptr, L0.indices, > L0.data), comm=PETSc.COMM_WORLD) > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yongxiang27 at gmail.com Fri Apr 6 08:17:47 2018 From: yongxiang27 at gmail.com (Yongxiang Wu) Date: Fri, 6 Apr 2018 15:17:47 +0200 Subject: [petsc-users] Scatter scipy Sparse matrix to petsc local processor In-Reply-To: References: Message-ID: Hello Matt, thanks for your quick reply. My situation is a bit different. What I am solving is a generalized eigenvalue problem. I only recently switched from scipy.eigs to slepc, because slepc support parallel eigenvalue problem. In other word, I am adapting my old code to utilizing slepc/petsc parallel support. To have a first test, I saved the matrix from eigenvalue problem with old code. Then read in this matrix for slepc/petsc validation. That's why I need to scatter it to processors. Do you have any idea how can I achieve this? with regards Yongxiang On 6 April 2018 at 15:00, Matthew Knepley wrote: > On Fri, Apr 6, 2018 at 8:44 AM, Yongxiang Wu > wrote: > >> Hello, everyone, >> >> I already have a scipy sparse square matrix L0 . Since my problem is >> large, parallel run is preferred. My Question is, how can I scatter my L0 >> to each of the processors? In the following code, I can get the indices >> of the localized part of the matrix. In the tutorial, the matrix element >> are directly assign with value, but in my case, the matrix are so large, >> assign each element in loop (commented code) is not efficient. So if any >> function would do the mpi scatter work? >> > > Hi Yongxiang, > > It would be really anomalous for what you propose to results in any > speedup at all. If the matrix is large, > it will not fit on one process. Any speedup from using more processes will > be eaten up by the time to > communicate the matrix. I would focus on generating the matrix in parallel. > > Thanks, > > Matt > > >> With regards and Thanks. >> >> import sys, slepc4py >> slepc4py.init(sys.argv) >> from petsc4py import PETSc >> from slepc4py import SLEPc >> >> opts = PETSc.Options() >> opts.setValue('-st_pc_factor_mat_solver_package','mumps') >> >> A = PETSc.Mat().createAIJ(size=L0.shape,comm=PETSc.COMM_WORLD) >> A.setUp() >> >> Istart, Iend = A.getOwnershipRange()# for I in range(Istart,Iend):# for J in range(0,L0.shape[0]):# A[I,J] = L0[I,J] >> >> The flowing code, would make the assignment from the scipy sparse matrix >> L0 to PETSc matrix A. But this would only work for one process. >> >> A = PETSc.Mat().createAIJ(size=L0.shape, >> csr=(L0.indptr, L0.indices, >> L0.data), comm=PETSc.COMM_WORLD) >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 6 08:22:17 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 6 Apr 2018 09:22:17 -0400 Subject: [petsc-users] Scatter scipy Sparse matrix to petsc local processor In-Reply-To: References: Message-ID: On Fri, Apr 6, 2018 at 9:17 AM, Yongxiang Wu wrote: > Hello Matt, > thanks for your quick reply. My situation is a bit different. What I am > solving is a generalized eigenvalue problem. I only recently switched from > scipy.eigs to slepc, because slepc support parallel eigenvalue problem. In > other word, I am adapting my old code to utilizing slepc/petsc parallel > support. To have a first test, I saved the matrix from eigenvalue problem > with old code. Then read in this matrix for slepc/petsc validation. That's > why I need to scatter it to processors. Do you have any idea how can I > achieve this? > Ah, the best way to do this is to first convert your matrix from the format it is in to PETSc binary format. Then you can load it in parallel automatically. So, I would a) Load my matrix b) Create a serial PETSc matrix from that using petc4py c) Save that matrix using a binary viewer Then in your parallel code you can jsut MatLoad() that matrix, and it will automatically be distributed. Thanks, Matt > with regards > Yongxiang > > On 6 April 2018 at 15:00, Matthew Knepley wrote: > >> On Fri, Apr 6, 2018 at 8:44 AM, Yongxiang Wu >> wrote: >> >>> Hello, everyone, >>> >>> I already have a scipy sparse square matrix L0 . Since my problem is >>> large, parallel run is preferred. My Question is, how can I scatter my >>> L0 to each of the processors? In the following code, I can get the >>> indices of the localized part of the matrix. In the tutorial, the matrix >>> element are directly assign with value, but in my case, the matrix are so >>> large, assign each element in loop (commented code) is not efficient. So if >>> any function would do the mpi scatter work? >>> >> >> Hi Yongxiang, >> >> It would be really anomalous for what you propose to results in any >> speedup at all. If the matrix is large, >> it will not fit on one process. Any speedup from using more processes >> will be eaten up by the time to >> communicate the matrix. I would focus on generating the matrix in >> parallel. >> >> Thanks, >> >> Matt >> >> >>> With regards and Thanks. >>> >>> import sys, slepc4py >>> slepc4py.init(sys.argv) >>> from petsc4py import PETSc >>> from slepc4py import SLEPc >>> >>> opts = PETSc.Options() >>> opts.setValue('-st_pc_factor_mat_solver_package','mumps') >>> >>> A = PETSc.Mat().createAIJ(size=L0.shape,comm=PETSc.COMM_WORLD) >>> A.setUp() >>> >>> Istart, Iend = A.getOwnershipRange()# for I in range(Istart,Iend):# for J in range(0,L0.shape[0]):# A[I,J] = L0[I,J] >>> >>> The flowing code, would make the assignment from the scipy sparse matrix >>> L0 to PETSc matrix A. But this would only work for one process. >>> >>> A = PETSc.Mat().createAIJ(size=L0.shape, >>> csr=(L0.indptr, L0.indices, >>> L0.data), comm=PETSc.COMM_WORLD) >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernardomartinsrocha at gmail.com Fri Apr 6 16:47:26 2018 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Fri, 6 Apr 2018 18:47:26 -0300 Subject: [petsc-users] DMPlexGetCone in Fortran In-Reply-To: References: Message-ID: Hello, I can't get a a simple code to work for DMPlexGetCone. I have tried different ways of handling the array/pointer which is the third argument of DMPlexGetCone, but neither of them worked for me. The last version that I got compiling is this one, which is very much based on the following example: .../petsc-3.7.6/src/dm/impls/plex/examples/tutorials/ex1f90.F PetscInt, target, dimension(6) :: icone PetscInt, pointer :: pIcone(:) call DMPlexCreateFromCellList(PETSC_COMM_WORLD, & ndim,numel,numnp,nen,PETSC_TRUE, & ielement,ndim,xnodes,dm,ierr) ... call DMPlexGetCone(dm,0,pIcone,ierr) write(*,*) icone(1),icone(2),icone(3),icone(4),icone(5),icone(6) call DMPlexRestoreCone(dm,0,pIcone,ierr) I always get a segmentation fault. I've created a simple cuboid mesh with 2x2x2 hexaedral elements. Many other routines are working and give reasonable results. Output of DMView DM Object: 1 MPI processes type: plex DM_0x84000000_0 in 3 dimensions: 0-cells: 27 1-cells: 66 2-cells: 44 3-cells: 8 Labels: depth: 4 strata of sizes (27, 66, 44, 8) On Thu, Apr 5, 2018 at 10:57 PM, Bernardo Rocha < bernardomartinsrocha at gmail.com> wrote: > OK, thanks a lot. > I had to rename the files and add -ffixed-form as argument to the compiler. > Best regards, > Bernardo > > On Thu, Apr 5, 2018 at 8:21 PM, Matthew Knepley wrote: > >> On Thu, Apr 5, 2018 at 7:16 PM, Bernardo Rocha < >> bernardomartinsrocha at gmail.com> wrote: >> >>> Hello everyone, >>> >>> I've been trying to use DMPlex in a Fortran 77 application >>> (unfortunately this is legacy code and I can't move to F90). >>> >>> Is there a way to use DMPlexGetCone on it? >>> Although the documentation online says it is only available for F90, >>> I'm wondering if there is a trick like the ones in the users manual for >>> routines >>> that return an array. I tried, but I'm not sure it it is right. >>> >> >> We have standardized our array handling with F90, and I don't think we >> will go back >> to supporting new things for F77-style arrays. >> >> That being said, I think all you have to do is compile that source with >> an F90 compiler. You >> should not have to change anything. If you want the F90 array to look >> like an F77 array, >> just pass it as a function argument (I think this works). >> >> Thanks, >> >> Matt >> >> >>> Best regards, >>> Bernardo M. Rocha >>> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Fri Apr 6 19:50:05 2018 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 7 Apr 2018 03:50:05 +0300 Subject: [petsc-users] petsc4py: reuse setup for multiple solver calls? In-Reply-To: References: Message-ID: On 6 April 2018 at 08:48, Robert Speck wrote: > Thank you for your answer! Please see below for comments/questions. > > On 05.04.18 12:53, Matthew Knepley wrote: >> On Thu, Apr 5, 2018 at 6:39 AM, Robert Speck > >> For nonlinear solves which stay the same size, you do nothing. > > "nothing" in terms of "nothing you can do" or "nothing you have to do"? > I would say "nothing you have to do". Nonlinear solvers work by just setting two callback routines to compute the residual and the Jacobian. The user-defined residual routine receives a vector you have to put values in, same for the Jacobian routine, you have to put values in the (already preallocated) matrix that is reused on each newton step, and on each successive solve. SNES will take care of everything else (in particular, updating the KSP linear solver, and refreshing the preconditioner, always trying to reuse already-preallocated internal data structures as much as possible). All what what I said for SNES applies to TS, in case you ever want to give it a chance. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From knepley at gmail.com Fri Apr 6 21:48:25 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 6 Apr 2018 22:48:25 -0400 Subject: [petsc-users] DMPlexGetCone in Fortran In-Reply-To: References: Message-ID: If you send the whole code, we can try it here. I am assuming that the examples run for you. Matt On Fri, Apr 6, 2018 at 5:47 PM, Bernardo Rocha < bernardomartinsrocha at gmail.com> wrote: > Hello, > > I can't get a a simple code to work for DMPlexGetCone. > I have tried different ways of handling the array/pointer which is the > third argument of DMPlexGetCone, > but neither of them worked for me. > > The last version that I got compiling is this one, which is very much > based on the > following example: .../petsc-3.7.6/src/dm/impls/plex/examples/tutorials/ > ex1f90.F > > PetscInt, target, dimension(6) :: icone > PetscInt, pointer :: pIcone(:) > > call DMPlexCreateFromCellList(PETSC_COMM_WORLD, > & ndim,numel,numnp,nen,PETSC_TRUE, > & ielement,ndim,xnodes,dm,ierr) > ... > call DMPlexGetCone(dm,0,pIcone,ierr) > write(*,*) icone(1),icone(2),icone(3),icone(4),icone(5),icone(6) > call DMPlexRestoreCone(dm,0,pIcone,ierr) > > I always get a segmentation fault. > > I've created a simple cuboid mesh with 2x2x2 hexaedral elements. > Many other routines are working and give reasonable results. > > Output of DMView > > DM Object: 1 MPI processes > type: plex > DM_0x84000000_0 in 3 dimensions: > 0-cells: 27 > 1-cells: 66 > 2-cells: 44 > 3-cells: 8 > Labels: > depth: 4 strata of sizes (27, 66, 44, 8) > > > > > On Thu, Apr 5, 2018 at 10:57 PM, Bernardo Rocha < > bernardomartinsrocha at gmail.com> wrote: > >> OK, thanks a lot. >> I had to rename the files and add -ffixed-form as argument to the >> compiler. >> Best regards, >> Bernardo >> >> On Thu, Apr 5, 2018 at 8:21 PM, Matthew Knepley >> wrote: >> >>> On Thu, Apr 5, 2018 at 7:16 PM, Bernardo Rocha < >>> bernardomartinsrocha at gmail.com> wrote: >>> >>>> Hello everyone, >>>> >>>> I've been trying to use DMPlex in a Fortran 77 application >>>> (unfortunately this is legacy code and I can't move to F90). >>>> >>>> Is there a way to use DMPlexGetCone on it? >>>> Although the documentation online says it is only available for F90, >>>> I'm wondering if there is a trick like the ones in the users manual for >>>> routines >>>> that return an array. I tried, but I'm not sure it it is right. >>>> >>> >>> We have standardized our array handling with F90, and I don't think we >>> will go back >>> to supporting new things for F77-style arrays. >>> >>> That being said, I think all you have to do is compile that source with >>> an F90 compiler. You >>> should not have to change anything. If you want the F90 array to look >>> like an F77 array, >>> just pass it as a function argument (I think this works). >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Best regards, >>>> Bernardo M. Rocha >>>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rongliang.chan at gmail.com Sat Apr 7 00:24:34 2018 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Sat, 7 Apr 2018 13:24:34 +0800 Subject: [petsc-users] Index set does not match blocks Message-ID: <09b11f09-9cf7-c081-cdaf-1ad2a49dc621@gmail.com> Hello everyone, I updated my code from petsc-3.6 to petsc-3.8 and got the following error. But it works well if I set the block size to be 1. I am solving an unstructured mesh problem. Any suggestions for debugging this are welcome. Thanks. Best, Rongliang [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Index set does not match blocks [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 [0]PETSC ERROR: ./HiBflow on a 64bit-debug named rongliang-chen by rlchen Sat Apr 7 13:14:20 2018 [0]PETSC ERROR: Configure options --download-mpich --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-debugging=1 --download-fblaslapack --download-ctetgen --with-64-bit-indices [0]PETSC ERROR: #1 MatCreateSubMatrix_SeqBAIJ() line 184 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c [0]PETSC ERROR: #2 MatCreateSubMatrices_SeqBAIJ() line 264 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c [0]PETSC ERROR: #3 MatCreateSubMatrices() line 6764 in /home/rlchen/soft/petsc-3.8.4/src/mat/interface/matrix.c [0]PETSC ERROR: #4 PCSetUp_ASM() line 326 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #6 KSPSetUp() line 381 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #7 KSPSolve() line 612 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /home/rlchen/soft/petsc-3.8.4/src/snes/impls/ls/ls.c [0]PETSC ERROR: #9 SNESSolve() line 4131 in /home/rlchen/soft/petsc-3.8.4/src/snes/interface/snes.c [0]PETSC ERROR: #10 TimeStep() line 1751 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c [0]PETSC ERROR: #11 SolveTimeDependent() line 1315 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c [0]PETSC ERROR: #12 JoabCtxSolve() line 263 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c [0]PETSC ERROR: #13 main() line 158 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c From bsmith at mcs.anl.gov Sat Apr 7 10:20:59 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sat, 7 Apr 2018 15:20:59 +0000 Subject: [petsc-users] Index set does not match blocks In-Reply-To: <09b11f09-9cf7-c081-cdaf-1ad2a49dc621@gmail.com> References: <09b11f09-9cf7-c081-cdaf-1ad2a49dc621@gmail.com> Message-ID: <86823E90-74A6-4CAF-AAF5-A474E6D88D38@anl.gov> For BAIJ matrices with ASM preconditioner the index sets that define the ASM blocks must contain all the points in a block or none of them. The code is detecting this is not the case; I don't know why this would be different between 3.8 and 3.6 You can use the option -start_in_debugger and put a break point in MatCreateSubMatrix_SeqBAIJ() then do call ISView(isrow,0) and look at the indices printed. Do the sets of indices match the block size? How do you set the ASM blocks for your code? They must be not set correctly when bs > 1. Barry Perhaps in 3.6 we did less error checking and so did not detect the error in that case. > On Apr 7, 2018, at 12:24 AM, Rongliang Chen wrote: > > Hello everyone, > > > I updated my code from petsc-3.6 to petsc-3.8 and got the following error. But it works well if I set the block size to be 1. I am solving an unstructured mesh problem. Any suggestions for debugging this are welcome. Thanks. > > > Best, > > Rongliang > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Index set does not match blocks > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 > [0]PETSC ERROR: ./HiBflow on a 64bit-debug named rongliang-chen by rlchen Sat Apr 7 13:14:20 2018 > [0]PETSC ERROR: Configure options --download-mpich --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-debugging=1 --download-fblaslapack --download-ctetgen --with-64-bit-indices > [0]PETSC ERROR: #1 MatCreateSubMatrix_SeqBAIJ() line 184 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c > [0]PETSC ERROR: #2 MatCreateSubMatrices_SeqBAIJ() line 264 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c > [0]PETSC ERROR: #3 MatCreateSubMatrices() line 6764 in /home/rlchen/soft/petsc-3.8.4/src/mat/interface/matrix.c > [0]PETSC ERROR: #4 PCSetUp_ASM() line 326 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/impls/asm/asm.c > [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #6 KSPSetUp() line 381 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #7 KSPSolve() line 612 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /home/rlchen/soft/petsc-3.8.4/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #9 SNESSolve() line 4131 in /home/rlchen/soft/petsc-3.8.4/src/snes/interface/snes.c > [0]PETSC ERROR: #10 TimeStep() line 1751 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c > [0]PETSC ERROR: #11 SolveTimeDependent() line 1315 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c > [0]PETSC ERROR: #12 JoabCtxSolve() line 263 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c > [0]PETSC ERROR: #13 main() line 158 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c > From rongliang.chan at gmail.com Sat Apr 7 13:57:39 2018 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Sun, 8 Apr 2018 02:57:39 +0800 Subject: [petsc-users] Index set does not match blocks In-Reply-To: <86823E90-74A6-4CAF-AAF5-A474E6D88D38@anl.gov> References: <09b11f09-9cf7-c081-cdaf-1ad2a49dc621@gmail.com> <86823E90-74A6-4CAF-AAF5-A474E6D88D38@anl.gov> Message-ID: <3c371769-c585-2218-e7d7-092768f1e87f@gmail.com> Hi Barry, Thanks for your reply. I checked the isrow with ISView (see bellow for the output) and I found that the output of the 3.6 (output-3.6.txt) and 3.8 (output-3.8.txt) are totally different. I do not know why. By the way, the block size of my code should be 7 but the output of the isrow in 3.8 shows that it is 8. I did not set the ASM blocks by myself. I only created a local to global mapping and use it to create the matrix with MatCreateBAIJ. The ASM setup is done by petsc. Best, Rongliang On 04/07/2018 11:20 PM, Smith, Barry F. wrote: > For BAIJ matrices with ASM preconditioner the index sets that define the ASM blocks must contain all the points in a block or none of them. The code is detecting this is not the case; I don't know why this would be different between 3.8 and 3.6 > > You can use the option -start_in_debugger and put a break point in MatCreateSubMatrix_SeqBAIJ() then do > > call ISView(isrow,0) > > and look at the indices printed. Do the sets of indices match the block size? > > How do you set the ASM blocks for your code? They must be not set correctly when bs > 1. > > > Barry > > Perhaps in 3.6 we did less error checking and so did not detect the error in that case. > > > >> On Apr 7, 2018, at 12:24 AM, Rongliang Chen wrote: >> >> Hello everyone, >> >> >> I updated my code from petsc-3.6 to petsc-3.8 and got the following error. But it works well if I set the block size to be 1. I am solving an unstructured mesh problem. Any suggestions for debugging this are welcome. Thanks. >> >> >> Best, >> >> Rongliang >> >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Nonconforming object sizes >> [0]PETSC ERROR: Index set does not match blocks >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 >> [0]PETSC ERROR: ./HiBflow on a 64bit-debug named rongliang-chen by rlchen Sat Apr 7 13:14:20 2018 >> [0]PETSC ERROR: Configure options --download-mpich --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-debugging=1 --download-fblaslapack --download-ctetgen --with-64-bit-indices >> [0]PETSC ERROR: #1 MatCreateSubMatrix_SeqBAIJ() line 184 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >> [0]PETSC ERROR: #2 MatCreateSubMatrices_SeqBAIJ() line 264 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >> [0]PETSC ERROR: #3 MatCreateSubMatrices() line 6764 in /home/rlchen/soft/petsc-3.8.4/src/mat/interface/matrix.c >> [0]PETSC ERROR: #4 PCSetUp_ASM() line 326 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/impls/asm/asm.c >> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: #6 KSPSetUp() line 381 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: #7 KSPSolve() line 612 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /home/rlchen/soft/petsc-3.8.4/src/snes/impls/ls/ls.c >> [0]PETSC ERROR: #9 SNESSolve() line 4131 in /home/rlchen/soft/petsc-3.8.4/src/snes/interface/snes.c >> [0]PETSC ERROR: #10 TimeStep() line 1751 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >> [0]PETSC ERROR: #11 SolveTimeDependent() line 1315 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >> [0]PETSC ERROR: #12 JoabCtxSolve() line 263 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >> [0]PETSC ERROR: #13 main() line 158 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >> -------------- next part -------------- IS Object: 1 MPI processes type: general Number of indices in set 1729 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18 19 19 20 20 21 21 22 22 23 23 24 24 25 25 26 26 27 27 28 28 29 29 30 30 31 31 32 32 33 33 34 34 35 35 36 36 37 37 38 38 39 39 40 40 41 41 42 42 43 43 44 44 45 45 46 46 47 47 48 48 49 49 50 50 51 51 52 52 53 53 54 54 55 55 56 56 57 57 58 58 59 59 60 60 61 61 62 62 63 63 64 64 65 65 66 66 67 67 68 68 69 69 70 70 71 71 72 72 73 73 74 74 75 75 76 76 77 77 78 78 79 79 80 80 81 81 82 82 83 83 84 84 85 85 86 86 87 87 88 88 89 89 90 90 91 91 92 92 93 93 94 94 95 95 96 96 97 97 98 98 99 99 100 100 101 101 102 102 103 103 104 104 105 105 106 106 107 107 108 108 109 109 110 110 111 111 112 112 113 113 114 114 115 115 116 116 117 117 118 118 119 119 120 120 121 121 122 122 123 123 124 124 125 125 126 126 127 127 128 128 129 129 130 130 131 131 132 132 133 133 134 134 135 135 136 136 137 137 138 138 139 139 140 140 141 141 142 142 143 143 144 144 145 145 146 146 147 147 148 148 149 149 150 150 151 151 152 152 153 153 154 154 155 155 156 156 157 157 158 158 159 159 160 160 161 161 162 162 163 163 164 164 165 165 166 166 167 167 168 168 169 169 170 170 171 171 172 172 173 173 174 174 175 175 176 176 177 177 178 178 179 179 180 180 181 181 182 182 183 183 184 184 185 185 186 186 187 187 188 188 189 189 190 190 191 191 192 192 193 193 194 194 195 195 196 196 197 197 198 198 199 199 200 200 201 201 202 202 203 203 204 204 205 205 206 206 207 207 208 208 209 209 210 210 211 211 212 212 213 213 214 214 215 215 216 216 217 217 218 218 219 219 220 220 221 221 222 222 223 223 224 224 225 225 226 226 227 227 228 228 229 229 230 230 231 231 232 232 233 233 234 234 235 235 236 236 237 237 238 238 239 239 240 240 241 241 242 242 243 243 244 244 245 245 246 246 247 247 248 248 249 249 250 250 251 251 252 252 253 253 254 254 255 255 256 256 257 257 258 258 259 259 260 260 261 261 262 262 263 263 264 264 265 265 266 266 267 267 268 268 269 269 270 270 271 271 272 272 273 273 274 274 275 275 276 276 277 277 278 278 279 279 280 280 281 281 282 282 283 283 284 284 285 285 286 286 287 287 288 288 289 289 290 290 291 291 292 292 293 293 294 294 295 295 296 296 297 297 298 298 299 299 300 300 301 301 302 302 303 303 304 304 305 305 306 306 307 307 308 308 309 309 310 310 311 311 312 312 313 313 314 314 315 315 316 316 317 317 318 318 319 319 320 320 321 321 322 322 323 323 324 324 325 325 326 326 327 327 328 328 329 329 330 330 331 331 332 332 333 333 334 334 335 335 336 336 337 337 338 338 339 339 340 340 341 341 342 342 343 343 344 344 345 345 346 346 347 347 348 348 349 349 350 350 351 351 352 352 353 353 354 354 355 355 356 356 357 357 358 358 359 359 360 360 361 361 362 362 363 363 364 364 365 365 366 366 367 367 368 368 369 369 370 370 371 371 372 372 373 373 374 374 375 375 376 376 377 377 378 378 379 379 380 380 381 381 382 382 383 383 384 384 385 385 386 386 387 387 388 388 389 389 390 390 391 391 392 392 393 393 394 394 395 395 396 396 397 397 398 398 399 399 400 400 401 401 402 402 403 403 404 404 405 405 406 406 407 407 408 408 409 409 410 410 411 411 412 412 413 413 414 414 415 415 416 416 417 417 418 418 419 419 420 420 421 421 422 422 423 423 424 424 425 425 426 426 427 427 428 428 429 429 430 430 431 431 432 432 433 433 434 434 435 435 436 436 437 437 438 438 439 439 440 440 441 441 442 442 443 443 444 444 445 445 446 446 447 447 448 448 449 449 450 450 451 451 452 452 453 453 454 454 455 455 456 456 457 457 458 458 459 459 460 460 461 461 462 462 463 463 464 464 465 465 466 466 467 467 468 468 469 469 470 470 471 471 472 472 473 473 474 474 475 475 476 476 477 477 478 478 479 479 480 480 481 481 482 482 483 483 484 484 485 485 486 486 487 487 488 488 489 489 490 490 491 491 492 492 493 493 494 494 495 495 496 496 497 497 498 498 499 499 500 500 501 501 502 502 503 503 504 504 505 505 506 506 507 507 508 508 509 509 510 510 511 511 512 512 513 513 514 514 515 515 516 516 517 517 518 518 519 519 520 520 521 521 522 522 523 523 524 524 525 525 526 526 527 527 528 528 529 529 530 530 531 531 532 532 533 533 534 534 535 535 536 536 537 537 538 538 539 539 540 540 541 541 542 542 543 543 544 544 545 545 546 546 547 547 548 548 549 549 550 550 551 551 552 552 553 553 554 554 555 555 556 556 557 557 558 558 559 559 560 560 561 561 562 562 563 563 564 564 565 565 566 566 567 567 568 568 569 569 570 570 571 571 572 572 573 573 574 574 575 575 576 576 577 577 578 578 579 579 580 580 581 581 582 582 583 583 584 584 585 585 586 586 587 587 588 588 589 589 590 590 591 591 592 592 593 593 594 594 595 595 596 596 597 597 598 598 599 599 600 600 601 601 602 602 603 603 604 604 605 605 606 606 607 607 608 608 609 609 610 610 611 611 612 612 613 613 614 614 615 615 616 616 617 617 618 618 619 619 620 620 621 621 622 622 623 623 624 624 625 625 626 626 627 627 628 628 629 629 630 630 631 631 632 632 633 633 634 634 635 635 636 636 637 637 638 638 639 639 640 640 641 641 642 642 643 643 644 644 645 645 646 646 647 647 648 648 649 649 650 650 651 651 652 652 653 653 654 654 655 655 656 656 657 657 658 658 659 659 660 660 661 661 662 662 663 663 664 664 665 665 666 666 667 667 668 668 669 669 670 670 671 671 672 672 673 673 674 674 675 675 676 676 677 677 678 678 679 679 680 680 681 681 682 682 683 683 684 684 685 685 686 686 687 687 688 688 689 689 690 690 691 691 692 692 693 693 694 694 695 695 696 696 697 697 698 698 699 699 700 700 701 701 702 702 703 703 704 704 705 705 706 706 707 707 708 708 709 709 710 710 711 711 712 712 713 713 714 714 715 715 716 716 717 717 718 718 719 719 720 720 721 721 722 722 723 723 724 724 725 725 726 726 727 727 728 728 729 729 730 730 731 731 732 732 733 733 734 734 735 735 736 736 737 737 738 738 739 739 740 740 741 741 742 742 743 743 744 744 745 745 746 746 747 747 748 748 749 749 750 750 751 751 752 752 753 753 754 754 755 755 756 756 757 757 758 758 759 759 760 760 761 761 762 762 763 763 764 764 765 765 766 766 767 767 768 768 769 769 770 770 771 771 772 772 773 773 774 774 775 775 776 776 777 777 778 778 779 779 780 780 781 781 782 782 783 783 784 784 785 785 786 786 787 787 788 788 789 789 790 790 791 791 792 792 793 793 794 794 795 795 796 796 797 797 798 798 799 799 800 800 801 801 802 802 803 803 804 804 805 805 806 806 807 807 808 808 809 809 810 810 811 811 812 812 813 813 814 814 815 815 816 816 817 817 818 818 819 819 820 820 821 821 822 822 823 823 824 824 825 825 826 826 827 827 828 828 829 829 830 830 831 831 832 832 833 833 834 834 835 835 836 836 837 837 838 838 839 839 840 840 841 841 842 842 843 843 844 844 845 845 846 846 847 847 848 848 849 849 850 850 851 851 852 852 853 853 854 854 855 855 856 856 857 857 858 858 859 859 860 860 861 861 862 862 863 863 864 864 865 865 866 866 867 867 868 868 869 869 870 870 871 871 872 872 873 873 874 874 875 875 876 876 877 877 878 878 879 879 880 880 881 881 882 882 883 883 884 884 885 885 886 886 887 887 888 888 889 889 890 890 891 891 892 892 893 893 894 894 895 895 896 896 897 897 898 898 899 899 900 900 901 901 902 902 903 903 904 904 905 905 906 906 907 907 908 908 909 909 910 910 911 911 912 912 913 913 914 914 915 915 916 916 917 917 918 918 919 919 920 920 921 921 922 922 923 923 924 924 925 925 926 926 927 927 928 928 929 929 930 930 931 931 932 932 933 933 934 934 935 935 936 936 937 937 938 938 939 939 940 940 941 941 942 942 943 943 944 944 945 945 946 946 947 947 948 948 949 949 950 950 951 951 952 952 953 953 954 954 955 955 956 956 957 957 958 958 959 959 960 960 961 961 962 962 963 963 964 964 965 965 966 966 967 967 968 968 969 969 970 970 971 971 972 972 973 973 974 974 975 975 976 976 977 977 978 978 979 979 980 980 981 981 982 982 983 983 984 984 985 985 986 986 987 987 988 988 989 989 990 990 991 991 992 992 993 993 994 994 995 995 996 996 997 997 998 998 999 999 1000 1000 1001 1001 1002 1002 1003 1003 1004 1004 1005 1005 1006 1006 1007 1007 1008 1008 1009 1009 1010 1010 1011 1011 1012 1012 1013 1013 1014 1014 1015 1015 1016 1016 1017 1017 1018 1018 1019 1019 1020 1020 1021 1021 1022 1022 1023 1023 1024 1024 1025 1025 1026 1026 1027 1027 1028 1028 1029 1029 1030 1030 1031 1031 1032 1032 1033 1033 1034 1034 1035 1035 1036 1036 1037 1037 1038 1038 1039 1039 1040 1040 1041 1041 1042 1042 1043 1043 1044 1044 1045 1045 1046 1046 1047 1047 1048 1048 1049 1049 1050 1050 1051 1051 1052 1052 1053 1053 1054 1054 1055 1055 1056 1056 1057 1057 1058 1058 1059 1059 1060 1060 1061 1061 1062 1062 1063 1063 1064 1064 1065 1065 1066 1066 1067 1067 1068 1068 1069 1069 1070 1070 1071 1071 1072 1072 1073 1073 1074 1074 1075 1075 1076 1076 1077 1077 1078 1078 1079 1079 1080 1080 1081 1081 1082 1082 1083 1083 1084 1084 1085 1085 1086 1086 1087 1087 1088 1088 1089 1089 1090 1090 1091 1091 1092 1092 1093 1093 1094 1094 1095 1095 1096 1096 1097 1097 1098 1098 1099 1099 1100 1100 1101 1101 1102 1102 1103 1103 1104 1104 1105 1105 1106 1106 1107 1107 1108 1108 1109 1109 1110 1110 1111 1111 1112 1112 1113 1113 1114 1114 1115 1115 1116 1116 1117 1117 1118 1118 1119 1119 1120 1120 1121 1121 1122 1122 1123 1123 1124 1124 1125 1125 1126 1126 1127 1127 1128 1128 1129 1129 1130 1130 1131 1131 1132 1132 1133 1133 1134 1134 1135 1135 1136 1136 1137 1137 1138 1138 1139 1139 1140 1140 1141 1141 1142 1142 1143 1143 1144 1144 1145 1145 1146 1146 1147 1147 1148 1148 1149 1149 1150 1150 1151 1151 1152 1152 1153 1153 1154 1154 1155 1155 1156 1156 1157 1157 1158 1158 1159 1159 1160 1160 1161 1161 1162 1162 1163 1163 1164 1164 1165 1165 1166 1166 1167 1167 1168 1168 1169 1169 1170 1170 1171 1171 1172 1172 1173 1173 1174 1174 1175 1175 1176 1176 1177 1177 1178 1178 1179 1179 1180 1180 1181 1181 1182 1182 1183 1183 1184 1184 1185 1185 1186 1186 1187 1187 1188 1188 1189 1189 1190 1190 1191 1191 1192 1192 1193 1193 1194 1194 1195 1195 1196 1196 1197 1197 1198 1198 1199 1199 1200 1200 1201 1201 1202 1202 1203 1203 1204 1204 1205 1205 1206 1206 1207 1207 1208 1208 1209 1209 1210 1210 1211 1211 1212 1212 1213 1213 1214 1214 1215 1215 1216 1216 1217 1217 1218 1218 1219 1219 1220 1220 1221 1221 1222 1222 1223 1223 1224 1224 1225 1225 1226 1226 1227 1227 1228 1228 1229 1229 1230 1230 1231 1231 1232 1232 1233 1233 1234 1234 1235 1235 1236 1236 1237 1237 1238 1238 1239 1239 1240 1240 1241 1241 1242 1242 1243 1243 1244 1244 1245 1245 1246 1246 1247 1247 1248 1248 1249 1249 1250 1250 1251 1251 1252 1252 1253 1253 1254 1254 1255 1255 1256 1256 1257 1257 1258 1258 1259 1259 1260 1260 1261 1261 1262 1262 1263 1263 1264 1264 1265 1265 1266 1266 1267 1267 1268 1268 1269 1269 1270 1270 1271 1271 1272 1272 1273 1273 1274 1274 1275 1275 1276 1276 1277 1277 1278 1278 1279 1279 1280 1280 1281 1281 1282 1282 1283 1283 1284 1284 1285 1285 1286 1286 1287 1287 1288 1288 1289 1289 1290 1290 1291 1291 1292 1292 1293 1293 1294 1294 1295 1295 1296 1296 1297 1297 1298 1298 1299 1299 1300 1300 1301 1301 1302 1302 1303 1303 1304 1304 1305 1305 1306 1306 1307 1307 1308 1308 1309 1309 1310 1310 1311 1311 1312 1312 1313 1313 1314 1314 1315 1315 1316 1316 1317 1317 1318 1318 1319 1319 1320 1320 1321 1321 1322 1322 1323 1323 1324 1324 1325 1325 1326 1326 1327 1327 1328 1328 1329 1329 1330 1330 1331 1331 1332 1332 1333 1333 1334 1334 1335 1335 1336 1336 1337 1337 1338 1338 1339 1339 1340 1340 1341 1341 1342 1342 1343 1343 1344 1344 1345 1345 1346 1346 1347 1347 1348 1348 1349 1349 1350 1350 1351 1351 1352 1352 1353 1353 1354 1354 1355 1355 1356 1356 1357 1357 1358 1358 1359 1359 1360 1360 1361 1361 1362 1362 1363 1363 1364 1364 1365 1365 1366 1366 1367 1367 1368 1368 1369 1369 1370 1370 1371 1371 1372 1372 1373 1373 1374 1374 1375 1375 1376 1376 1377 1377 1378 1378 1379 1379 1380 1380 1381 1381 1382 1382 1383 1383 1384 1384 1385 1385 1386 1386 1387 1387 1388 1388 1389 1389 1390 1390 1391 1391 1392 1392 1393 1393 1394 1394 1395 1395 1396 1396 1397 1397 1398 1398 1399 1399 1400 1400 1401 1401 1402 1402 1403 1403 1404 1404 1405 1405 1406 1406 1407 1407 1408 1408 1409 1409 1410 1410 1411 1411 1412 1412 1413 1413 1414 1414 1415 1415 1416 1416 1417 1417 1418 1418 1419 1419 1420 1420 1421 1421 1422 1422 1423 1423 1424 1424 1425 1425 1426 1426 1427 1427 1428 1428 1429 1429 1430 1430 1431 1431 1432 1432 1433 1433 1434 1434 1435 1435 1436 1436 1437 1437 1438 1438 1439 1439 1440 1440 1441 1441 1442 1442 1443 1443 1444 1444 1445 1445 1446 1446 1447 1447 1448 1448 1449 1449 1450 1450 1451 1451 1452 1452 1453 1453 1454 1454 1455 1455 1456 1456 1457 1457 1458 1458 1459 1459 1460 1460 1461 1461 1462 1462 1463 1463 1464 1464 1465 1465 1466 1466 1467 1467 1468 1468 1469 1469 1470 1470 1471 1471 1472 1472 1473 1473 1474 1474 1475 1475 1476 1476 1477 1477 1478 1478 1479 1479 1480 1480 1481 1481 1482 1482 1483 1483 1484 1484 1485 1485 1486 1486 1487 1487 1488 1488 1489 1489 1490 1490 1491 1491 1492 1492 1493 1493 1494 1494 1495 1495 1496 1496 1497 1497 1498 1498 1499 1499 1500 1500 1501 1501 1502 1502 1503 1503 1504 1504 1505 1505 1506 1506 1507 1507 1508 1508 1509 1509 1510 1510 1511 1511 1512 1512 1513 1513 1514 1514 1515 1515 1516 1516 1517 1517 1518 1518 1519 1519 1520 1520 1521 1521 1522 1522 1523 1523 1524 1524 1525 1525 1526 1526 1527 1527 1528 1528 1529 1529 1530 1530 1531 1531 1532 1532 1533 1533 1534 1534 1535 1535 1536 1536 1537 1537 1538 1538 1539 1539 1540 1540 1541 1541 1542 1542 1543 1543 1544 1544 1545 1545 1546 1546 1547 1547 1548 1548 1549 1549 1550 1550 1551 1551 1552 1552 1553 1553 1554 1554 1555 1555 1556 1556 1557 1557 1558 1558 1559 1559 1560 1560 1561 1561 1562 1562 1563 1563 1564 1564 1565 1565 1566 1566 1567 1567 1568 1568 1569 1569 1570 1570 1571 1571 1572 1572 1573 1573 1574 1574 1575 1575 1576 1576 1577 1577 1578 1578 1579 1579 1580 1580 1581 1581 1582 1582 1583 1583 1584 1584 1585 1585 1586 1586 1587 1587 1588 1588 1589 1589 1590 1590 1591 1591 1592 1592 1593 1593 1594 1594 1595 1595 1596 1596 1597 1597 1598 1598 1599 1599 1600 1600 1601 1601 1602 1602 1603 1603 1604 1604 1605 1605 1606 1606 1607 1607 1608 1608 1609 1609 1610 1610 1611 1611 1612 1612 1613 1613 1614 1614 1615 1615 1616 1616 1617 1617 1618 1618 1619 1619 1620 1620 1621 1621 1622 1622 1623 1623 1624 1624 1625 1625 1626 1626 1627 1627 1628 1628 1629 1629 1630 1630 1631 1631 1632 1632 1633 1633 1634 1634 1635 1635 1636 1636 1637 1637 1638 1638 1639 1639 1640 1640 1641 1641 1642 1642 1643 1643 1644 1644 1645 1645 1646 1646 1647 1647 1648 1648 1649 1649 1650 1650 1651 1651 1652 1652 1653 1653 1654 1654 1655 1655 1656 1656 1657 1657 1658 1658 1659 1659 1660 1660 1661 1661 1662 1662 1663 1663 1664 1664 1665 1665 1666 1666 1667 1667 1668 1668 1669 1669 1670 1670 1671 1671 1672 1672 1673 1673 1674 1674 1675 1675 1676 1676 1677 1677 1678 1678 1679 1679 1680 1680 1681 1681 1682 1682 1683 1683 1684 1684 1685 1685 1686 1686 1687 1687 1688 1688 1689 1689 1690 1690 1691 1691 1692 1692 1693 1693 1694 1694 1695 1695 1696 1696 1697 1697 1698 1698 1699 1699 1700 1700 1701 1701 1702 1702 1703 1703 1704 1704 1705 1705 1706 1706 1707 1707 1708 1708 1709 1709 1710 1710 1711 1711 1712 1712 1713 1713 1714 1714 1715 1715 1716 1716 1717 1717 1718 1718 1719 1719 1720 1720 1721 1721 1722 1722 1723 1723 1724 1724 1725 1725 1726 1726 1727 1727 1728 1728 -------------- next part -------------- IS Object: 1 MPI processes type: general Number of indices in set 247 0 0 1 8 2 16 3 24 4 32 5 40 6 48 7 49 8 57 9 65 10 73 11 81 12 89 13 97 14 98 15 106 16 114 17 122 18 130 19 138 20 146 21 147 22 155 23 163 24 171 25 179 26 187 27 195 28 196 29 204 30 212 31 220 32 228 33 236 34 244 35 245 36 253 37 261 38 269 39 277 40 285 41 293 42 294 43 302 44 310 45 318 46 326 47 334 48 342 49 343 50 351 51 359 52 367 53 375 54 383 55 391 56 392 57 400 58 408 59 416 60 424 61 432 62 440 63 441 64 449 65 457 66 465 67 473 68 481 69 489 70 490 71 498 72 506 73 514 74 522 75 530 76 538 77 539 78 547 79 555 80 563 81 571 82 579 83 587 84 588 85 596 86 604 87 612 88 620 89 628 90 636 91 637 92 645 93 653 94 661 95 669 96 677 97 685 98 686 99 694 100 702 101 710 102 718 103 726 104 734 105 735 106 743 107 751 108 759 109 767 110 775 111 783 112 784 113 792 114 800 115 808 116 816 117 824 118 832 119 833 120 841 121 849 122 857 123 865 124 873 125 881 126 882 127 890 128 898 129 906 130 914 131 922 132 930 133 931 134 939 135 947 136 955 137 963 138 971 139 979 140 980 141 988 142 996 143 1004 144 1012 145 1020 146 1028 147 1029 148 1037 149 1045 150 1053 151 1061 152 1069 153 1077 154 1078 155 1086 156 1094 157 1102 158 1110 159 1118 160 1126 161 1127 162 1135 163 1143 164 1151 165 1159 166 1167 167 1175 168 1176 169 1184 170 1192 171 1200 172 1208 173 1216 174 1224 175 1225 176 1233 177 1241 178 1249 179 1257 180 1265 181 1273 182 1274 183 1282 184 1290 185 1298 186 1306 187 1314 188 1322 189 1323 190 1331 191 1339 192 1347 193 1355 194 1363 195 1371 196 1372 197 1380 198 1388 199 1396 200 1404 201 1412 202 1420 203 1421 204 1429 205 1437 206 1445 207 1453 208 1461 209 1469 210 1470 211 1478 212 1486 213 1494 214 1502 215 1510 216 1518 217 1519 218 1527 219 1535 220 1543 221 1551 222 1559 223 1567 224 1568 225 1576 226 1584 227 1592 228 1600 229 1608 230 1616 231 1617 232 1625 233 1633 234 1641 235 1649 236 1657 237 1665 238 1666 239 1674 240 1682 241 1690 242 1698 243 1706 244 1714 245 1715 246 1723 From bernardomartinsrocha at gmail.com Sat Apr 7 14:13:07 2018 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Sat, 7 Apr 2018 16:13:07 -0300 Subject: [petsc-users] DMPlexGetCone in Fortran In-Reply-To: References: Message-ID: OK, here is the entire code. I created a simple mesh to test DMPlex. As you can see, the GetCone is not working. As I said, I followed one of the examples for DMPlex using Fortran, I'm not an exper in F90. Best regards, Bernardo M. Rocha On Fri, Apr 6, 2018 at 11:48 PM, Matthew Knepley wrote: > If you send the whole code, we can try it here. I am assuming that the > examples run for you. > > Matt > > On Fri, Apr 6, 2018 at 5:47 PM, Bernardo Rocha < > bernardomartinsrocha at gmail.com> wrote: > >> Hello, >> >> I can't get a a simple code to work for DMPlexGetCone. >> I have tried different ways of handling the array/pointer which is the >> third argument of DMPlexGetCone, >> but neither of them worked for me. >> >> The last version that I got compiling is this one, which is very much >> based on the >> following example: .../petsc-3.7.6/src/dm/impls/p >> lex/examples/tutorials/ex1f90.F >> >> PetscInt, target, dimension(6) :: icone >> PetscInt, pointer :: pIcone(:) >> >> call DMPlexCreateFromCellList(PETSC_COMM_WORLD, >> & ndim,numel,numnp,nen,PETSC_TRUE, >> & ielement,ndim,xnodes,dm,ierr) >> ... >> call DMPlexGetCone(dm,0,pIcone,ierr) >> write(*,*) icone(1),icone(2),icone(3),icone(4),icone(5),icone(6) >> call DMPlexRestoreCone(dm,0,pIcone,ierr) >> >> I always get a segmentation fault. >> >> I've created a simple cuboid mesh with 2x2x2 hexaedral elements. >> Many other routines are working and give reasonable results. >> >> Output of DMView >> >> DM Object: 1 MPI processes >> type: plex >> DM_0x84000000_0 in 3 dimensions: >> 0-cells: 27 >> 1-cells: 66 >> 2-cells: 44 >> 3-cells: 8 >> Labels: >> depth: 4 strata of sizes (27, 66, 44, 8) >> >> >> >> >> On Thu, Apr 5, 2018 at 10:57 PM, Bernardo Rocha < >> bernardomartinsrocha at gmail.com> wrote: >> >>> OK, thanks a lot. >>> I had to rename the files and add -ffixed-form as argument to the >>> compiler. >>> Best regards, >>> Bernardo >>> >>> On Thu, Apr 5, 2018 at 8:21 PM, Matthew Knepley >>> wrote: >>> >>>> On Thu, Apr 5, 2018 at 7:16 PM, Bernardo Rocha < >>>> bernardomartinsrocha at gmail.com> wrote: >>>> >>>>> Hello everyone, >>>>> >>>>> I've been trying to use DMPlex in a Fortran 77 application >>>>> (unfortunately this is legacy code and I can't move to F90). >>>>> >>>>> Is there a way to use DMPlexGetCone on it? >>>>> Although the documentation online says it is only available for F90, >>>>> I'm wondering if there is a trick like the ones in the users manual >>>>> for routines >>>>> that return an array. I tried, but I'm not sure it it is right. >>>>> >>>> >>>> We have standardized our array handling with F90, and I don't think we >>>> will go back >>>> to supporting new things for F77-style arrays. >>>> >>>> That being said, I think all you have to do is compile that source with >>>> an F90 compiler. You >>>> should not have to change anything. If you want the F90 array to look >>>> like an F77 array, >>>> just pass it as a function argument (I think this works). >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Best regards, >>>>> Bernardo M. Rocha >>>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.F90 Type: text/x-fortran Size: 4762 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sat Apr 7 16:22:58 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sat, 7 Apr 2018 21:22:58 +0000 Subject: [petsc-users] Index set does not match blocks In-Reply-To: <3c371769-c585-2218-e7d7-092768f1e87f@gmail.com> References: <09b11f09-9cf7-c081-cdaf-1ad2a49dc621@gmail.com> <86823E90-74A6-4CAF-AAF5-A474E6D88D38@anl.gov> <3c371769-c585-2218-e7d7-092768f1e87f@gmail.com> Message-ID: <3E6D2524-423A-4A00-9062-4AE2DD1EFDE1@mcs.anl.gov> > On Apr 7, 2018, at 1:57 PM, Rongliang Chen wrote: > > Hi Barry, > > Thanks for your reply. I checked the isrow with ISView (see bellow for the output) and I found that the output of the 3.6 (output-3.6.txt) and 3.8 (output-3.8.txt) are totally different. I do not know why. By the way, the block size of my code should be 7 but the output of the isrow in 3.8 shows that it is 8. > > I did not set the ASM blocks by myself. I only created a local to global mapping and use it to create the matrix with MatCreateBAIJ. The ASM setup is done by petsc. There are some changes with the local to global mapping that is perhaps causing your difficulties. There use to be an MatSetLocalToGlobalMapping() and MatSetLocalToGlobalMappingBlock() they are gone and there is now a block size associated with the mapping. There is now one number per block instead of 1 number per index. Where do you get the ISLocalToGlobalMapping from() and what is its block size set too? Barry > > Best, > Rongliang > > > On 04/07/2018 11:20 PM, Smith, Barry F. wrote: >> For BAIJ matrices with ASM preconditioner the index sets that define the ASM blocks must contain all the points in a block or none of them. The code is detecting this is not the case; I don't know why this would be different between 3.8 and 3.6 >> >> You can use the option -start_in_debugger and put a break point in MatCreateSubMatrix_SeqBAIJ() then do >> >> call ISView(isrow,0) >> >> and look at the indices printed. Do the sets of indices match the block size? >> >> How do you set the ASM blocks for your code? They must be not set correctly when bs > 1. >> >> >> Barry >> >> Perhaps in 3.6 we did less error checking and so did not detect the error in that case. >> >> >> >>> On Apr 7, 2018, at 12:24 AM, Rongliang Chen wrote: >>> >>> Hello everyone, >>> >>> >>> I updated my code from petsc-3.6 to petsc-3.8 and got the following error. But it works well if I set the block size to be 1. I am solving an unstructured mesh problem. Any suggestions for debugging this are welcome. Thanks. >>> >>> >>> Best, >>> >>> Rongliang >>> >>> >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Nonconforming object sizes >>> [0]PETSC ERROR: Index set does not match blocks >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 >>> [0]PETSC ERROR: ./HiBflow on a 64bit-debug named rongliang-chen by rlchen Sat Apr 7 13:14:20 2018 >>> [0]PETSC ERROR: Configure options --download-mpich --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-debugging=1 --download-fblaslapack --download-ctetgen --with-64-bit-indices >>> [0]PETSC ERROR: #1 MatCreateSubMatrix_SeqBAIJ() line 184 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>> [0]PETSC ERROR: #2 MatCreateSubMatrices_SeqBAIJ() line 264 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>> [0]PETSC ERROR: #3 MatCreateSubMatrices() line 6764 in /home/rlchen/soft/petsc-3.8.4/src/mat/interface/matrix.c >>> [0]PETSC ERROR: #4 PCSetUp_ASM() line 326 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/impls/asm/asm.c >>> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: #6 KSPSetUp() line 381 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: #7 KSPSolve() line 612 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /home/rlchen/soft/petsc-3.8.4/src/snes/impls/ls/ls.c >>> [0]PETSC ERROR: #9 SNESSolve() line 4131 in /home/rlchen/soft/petsc-3.8.4/src/snes/interface/snes.c >>> [0]PETSC ERROR: #10 TimeStep() line 1751 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>> [0]PETSC ERROR: #11 SolveTimeDependent() line 1315 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>> [0]PETSC ERROR: #12 JoabCtxSolve() line 263 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>> [0]PETSC ERROR: #13 main() line 158 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>> > > From rongliang.chan at gmail.com Sat Apr 7 22:11:34 2018 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Sun, 8 Apr 2018 11:11:34 +0800 Subject: [petsc-users] Index set does not match blocks In-Reply-To: <3E6D2524-423A-4A00-9062-4AE2DD1EFDE1@mcs.anl.gov> References: <09b11f09-9cf7-c081-cdaf-1ad2a49dc621@gmail.com> <86823E90-74A6-4CAF-AAF5-A474E6D88D38@anl.gov> <3c371769-c585-2218-e7d7-092768f1e87f@gmail.com> <3E6D2524-423A-4A00-9062-4AE2DD1EFDE1@mcs.anl.gov> Message-ID: <291727b9-5e48-f36c-9ea0-8eb9c5364acd@gmail.com> Thanks Barry! I created the ISLocalToGlobalMapping after I loaded in the mesh and before the matrix creation. The block size for the ISLocalToGlobalMappingCreate is 7. In my code, the mapping is already in the form that one number per block since the petsc-3.6 requests this. So when I updated from 3.6 to 3.8, I did not make any changes for the ISLocalToGlobalMapping. The petsc-3.8 updated this mapping? And what should I do for this? Best, Rongliang On 04/08/2018 05:22 AM, Smith, Barry F. wrote: > >> On Apr 7, 2018, at 1:57 PM, Rongliang Chen wrote: >> >> Hi Barry, >> >> Thanks for your reply. I checked the isrow with ISView (see bellow for the output) and I found that the output of the 3.6 (output-3.6.txt) and 3.8 (output-3.8.txt) are totally different. I do not know why. By the way, the block size of my code should be 7 but the output of the isrow in 3.8 shows that it is 8. >> >> I did not set the ASM blocks by myself. I only created a local to global mapping and use it to create the matrix with MatCreateBAIJ. The ASM setup is done by petsc. > There are some changes with the local to global mapping that is perhaps causing your difficulties. There use to be an MatSetLocalToGlobalMapping() and MatSetLocalToGlobalMappingBlock() they are gone and there is now a block size associated with the mapping. There is now one number per block instead of 1 number per index. Where do you get the ISLocalToGlobalMapping from() and what is its block size set too? > > Barry > >> Best, >> Rongliang >> >> >> On 04/07/2018 11:20 PM, Smith, Barry F. wrote: >>> For BAIJ matrices with ASM preconditioner the index sets that define the ASM blocks must contain all the points in a block or none of them. The code is detecting this is not the case; I don't know why this would be different between 3.8 and 3.6 >>> >>> You can use the option -start_in_debugger and put a break point in MatCreateSubMatrix_SeqBAIJ() then do >>> >>> call ISView(isrow,0) >>> >>> and look at the indices printed. Do the sets of indices match the block size? >>> >>> How do you set the ASM blocks for your code? They must be not set correctly when bs > 1. >>> >>> >>> Barry >>> >>> Perhaps in 3.6 we did less error checking and so did not detect the error in that case. >>> >>> >>> >>>> On Apr 7, 2018, at 12:24 AM, Rongliang Chen wrote: >>>> >>>> Hello everyone, >>>> >>>> >>>> I updated my code from petsc-3.6 to petsc-3.8 and got the following error. But it works well if I set the block size to be 1. I am solving an unstructured mesh problem. Any suggestions for debugging this are welcome. Thanks. >>>> >>>> >>>> Best, >>>> >>>> Rongliang >>>> >>>> >>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>> [0]PETSC ERROR: Nonconforming object sizes >>>> [0]PETSC ERROR: Index set does not match blocks >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 >>>> [0]PETSC ERROR: ./HiBflow on a 64bit-debug named rongliang-chen by rlchen Sat Apr 7 13:14:20 2018 >>>> [0]PETSC ERROR: Configure options --download-mpich --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-debugging=1 --download-fblaslapack --download-ctetgen --with-64-bit-indices >>>> [0]PETSC ERROR: #1 MatCreateSubMatrix_SeqBAIJ() line 184 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>> [0]PETSC ERROR: #2 MatCreateSubMatrices_SeqBAIJ() line 264 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>> [0]PETSC ERROR: #3 MatCreateSubMatrices() line 6764 in /home/rlchen/soft/petsc-3.8.4/src/mat/interface/matrix.c >>>> [0]PETSC ERROR: #4 PCSetUp_ASM() line 326 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/impls/asm/asm.c >>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/interface/precon.c >>>> [0]PETSC ERROR: #6 KSPSetUp() line 381 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #7 KSPSolve() line 612 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /home/rlchen/soft/petsc-3.8.4/src/snes/impls/ls/ls.c >>>> [0]PETSC ERROR: #9 SNESSolve() line 4131 in /home/rlchen/soft/petsc-3.8.4/src/snes/interface/snes.c >>>> [0]PETSC ERROR: #10 TimeStep() line 1751 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>> [0]PETSC ERROR: #11 SolveTimeDependent() line 1315 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>> [0]PETSC ERROR: #12 JoabCtxSolve() line 263 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>> [0]PETSC ERROR: #13 main() line 158 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>> >> From bsmith at mcs.anl.gov Sat Apr 7 22:22:00 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 8 Apr 2018 03:22:00 +0000 Subject: [petsc-users] Index set does not match blocks In-Reply-To: <291727b9-5e48-f36c-9ea0-8eb9c5364acd@gmail.com> References: <09b11f09-9cf7-c081-cdaf-1ad2a49dc621@gmail.com> <86823E90-74A6-4CAF-AAF5-A474E6D88D38@anl.gov> <3c371769-c585-2218-e7d7-092768f1e87f@gmail.com> <3E6D2524-423A-4A00-9062-4AE2DD1EFDE1@mcs.anl.gov> <291727b9-5e48-f36c-9ea0-8eb9c5364acd@gmail.com> Message-ID: > On Apr 7, 2018, at 10:11 PM, Rongliang Chen wrote: > > Thanks Barry! I created the ISLocalToGlobalMapping after I loaded in the mesh and before the matrix creation. The block size for the ISLocalToGlobalMappingCreate is 7. In my code, the mapping is already in the form that one number per block since the petsc-3.6 requests this. So when I updated from 3.6 to 3.8, I did not make any changes for the ISLocalToGlobalMapping. The petsc-3.8 updated this mapping? And what should I do for this? The block change was from 3.5 to 3.6 so I am not sure what the problem is. is your code Fortran or C? Barry It's slower but you could try to see if you can get your code working with 3.7 first to determine if the problem comes up between 3.6 and 3.7 > > Best, > Rongliang > > On 04/08/2018 05:22 AM, Smith, Barry F. wrote: >> >>> On Apr 7, 2018, at 1:57 PM, Rongliang Chen wrote: >>> >>> Hi Barry, >>> >>> Thanks for your reply. I checked the isrow with ISView (see bellow for the output) and I found that the output of the 3.6 (output-3.6.txt) and 3.8 (output-3.8.txt) are totally different. I do not know why. By the way, the block size of my code should be 7 but the output of the isrow in 3.8 shows that it is 8. >>> >>> I did not set the ASM blocks by myself. I only created a local to global mapping and use it to create the matrix with MatCreateBAIJ. The ASM setup is done by petsc. >> There are some changes with the local to global mapping that is perhaps causing your difficulties. There use to be an MatSetLocalToGlobalMapping() and MatSetLocalToGlobalMappingBlock() they are gone and there is now a block size associated with the mapping. There is now one number per block instead of 1 number per index. Where do you get the ISLocalToGlobalMapping from() and what is its block size set too? >> >> Barry >> >>> Best, >>> Rongliang >>> >>> >>> On 04/07/2018 11:20 PM, Smith, Barry F. wrote: >>>> For BAIJ matrices with ASM preconditioner the index sets that define the ASM blocks must contain all the points in a block or none of them. The code is detecting this is not the case; I don't know why this would be different between 3.8 and 3.6 >>>> >>>> You can use the option -start_in_debugger and put a break point in MatCreateSubMatrix_SeqBAIJ() then do >>>> >>>> call ISView(isrow,0) >>>> >>>> and look at the indices printed. Do the sets of indices match the block size? >>>> >>>> How do you set the ASM blocks for your code? They must be not set correctly when bs > 1. >>>> >>>> >>>> Barry >>>> >>>> Perhaps in 3.6 we did less error checking and so did not detect the error in that case. >>>> >>>> >>>> >>>>> On Apr 7, 2018, at 12:24 AM, Rongliang Chen wrote: >>>>> >>>>> Hello everyone, >>>>> >>>>> >>>>> I updated my code from petsc-3.6 to petsc-3.8 and got the following error. But it works well if I set the block size to be 1. I am solving an unstructured mesh problem. Any suggestions for debugging this are welcome. Thanks. >>>>> >>>>> >>>>> Best, >>>>> >>>>> Rongliang >>>>> >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Nonconforming object sizes >>>>> [0]PETSC ERROR: Index set does not match blocks >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 >>>>> [0]PETSC ERROR: ./HiBflow on a 64bit-debug named rongliang-chen by rlchen Sat Apr 7 13:14:20 2018 >>>>> [0]PETSC ERROR: Configure options --download-mpich --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-debugging=1 --download-fblaslapack --download-ctetgen --with-64-bit-indices >>>>> [0]PETSC ERROR: #1 MatCreateSubMatrix_SeqBAIJ() line 184 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>>> [0]PETSC ERROR: #2 MatCreateSubMatrices_SeqBAIJ() line 264 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>>> [0]PETSC ERROR: #3 MatCreateSubMatrices() line 6764 in /home/rlchen/soft/petsc-3.8.4/src/mat/interface/matrix.c >>>>> [0]PETSC ERROR: #4 PCSetUp_ASM() line 326 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/impls/asm/asm.c >>>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: #6 KSPSetUp() line 381 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #7 KSPSolve() line 612 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /home/rlchen/soft/petsc-3.8.4/src/snes/impls/ls/ls.c >>>>> [0]PETSC ERROR: #9 SNESSolve() line 4131 in /home/rlchen/soft/petsc-3.8.4/src/snes/interface/snes.c >>>>> [0]PETSC ERROR: #10 TimeStep() line 1751 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>> [0]PETSC ERROR: #11 SolveTimeDependent() line 1315 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>> [0]PETSC ERROR: #12 JoabCtxSolve() line 263 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>> [0]PETSC ERROR: #13 main() line 158 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>> >>> > > From rongliang.chan at gmail.com Sat Apr 7 22:32:51 2018 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Sun, 8 Apr 2018 11:32:51 +0800 Subject: [petsc-users] Index set does not match blocks In-Reply-To: References: <09b11f09-9cf7-c081-cdaf-1ad2a49dc621@gmail.com> <86823E90-74A6-4CAF-AAF5-A474E6D88D38@anl.gov> <3c371769-c585-2218-e7d7-092768f1e87f@gmail.com> <3E6D2524-423A-4A00-9062-4AE2DD1EFDE1@mcs.anl.gov> <291727b9-5e48-f36c-9ea0-8eb9c5364acd@gmail.com> Message-ID: <959ed452-6db2-ed8f-3871-e5ad1a3a894d@gmail.com> My code is in C. I will try the 3.7 and let you know the result. Thanks. Best, Rongliang On 04/08/2018 11:22 AM, Smith, Barry F. wrote: > >> On Apr 7, 2018, at 10:11 PM, Rongliang Chen wrote: >> >> Thanks Barry! I created the ISLocalToGlobalMapping after I loaded in the mesh and before the matrix creation. The block size for the ISLocalToGlobalMappingCreate is 7. In my code, the mapping is already in the form that one number per block since the petsc-3.6 requests this. So when I updated from 3.6 to 3.8, I did not make any changes for the ISLocalToGlobalMapping. The petsc-3.8 updated this mapping? And what should I do for this? > > The block change was from 3.5 to 3.6 so I am not sure what the problem is. > > is your code Fortran or C? > > Barry > > It's slower but you could try to see if you can get your code working with 3.7 first to determine if the problem comes up between 3.6 and 3.7 > >> Best, >> Rongliang >> >> On 04/08/2018 05:22 AM, Smith, Barry F. wrote: >>>> On Apr 7, 2018, at 1:57 PM, Rongliang Chen wrote: >>>> >>>> Hi Barry, >>>> >>>> Thanks for your reply. I checked the isrow with ISView (see bellow for the output) and I found that the output of the 3.6 (output-3.6.txt) and 3.8 (output-3.8.txt) are totally different. I do not know why. By the way, the block size of my code should be 7 but the output of the isrow in 3.8 shows that it is 8. >>>> >>>> I did not set the ASM blocks by myself. I only created a local to global mapping and use it to create the matrix with MatCreateBAIJ. The ASM setup is done by petsc. >>> There are some changes with the local to global mapping that is perhaps causing your difficulties. There use to be an MatSetLocalToGlobalMapping() and MatSetLocalToGlobalMappingBlock() they are gone and there is now a block size associated with the mapping. There is now one number per block instead of 1 number per index. Where do you get the ISLocalToGlobalMapping from() and what is its block size set too? >>> >>> Barry >>> >>>> Best, >>>> Rongliang >>>> >>>> >>>> On 04/07/2018 11:20 PM, Smith, Barry F. wrote: >>>>> For BAIJ matrices with ASM preconditioner the index sets that define the ASM blocks must contain all the points in a block or none of them. The code is detecting this is not the case; I don't know why this would be different between 3.8 and 3.6 >>>>> >>>>> You can use the option -start_in_debugger and put a break point in MatCreateSubMatrix_SeqBAIJ() then do >>>>> >>>>> call ISView(isrow,0) >>>>> >>>>> and look at the indices printed. Do the sets of indices match the block size? >>>>> >>>>> How do you set the ASM blocks for your code? They must be not set correctly when bs > 1. >>>>> >>>>> >>>>> Barry >>>>> >>>>> Perhaps in 3.6 we did less error checking and so did not detect the error in that case. >>>>> >>>>> >>>>> >>>>>> On Apr 7, 2018, at 12:24 AM, Rongliang Chen wrote: >>>>>> >>>>>> Hello everyone, >>>>>> >>>>>> >>>>>> I updated my code from petsc-3.6 to petsc-3.8 and got the following error. But it works well if I set the block size to be 1. I am solving an unstructured mesh problem. Any suggestions for debugging this are welcome. Thanks. >>>>>> >>>>>> >>>>>> Best, >>>>>> >>>>>> Rongliang >>>>>> >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Nonconforming object sizes >>>>>> [0]PETSC ERROR: Index set does not match blocks >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 >>>>>> [0]PETSC ERROR: ./HiBflow on a 64bit-debug named rongliang-chen by rlchen Sat Apr 7 13:14:20 2018 >>>>>> [0]PETSC ERROR: Configure options --download-mpich --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-debugging=1 --download-fblaslapack --download-ctetgen --with-64-bit-indices >>>>>> [0]PETSC ERROR: #1 MatCreateSubMatrix_SeqBAIJ() line 184 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>>>> [0]PETSC ERROR: #2 MatCreateSubMatrices_SeqBAIJ() line 264 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>>>> [0]PETSC ERROR: #3 MatCreateSubMatrices() line 6764 in /home/rlchen/soft/petsc-3.8.4/src/mat/interface/matrix.c >>>>>> [0]PETSC ERROR: #4 PCSetUp_ASM() line 326 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/impls/asm/asm.c >>>>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #6 KSPSetUp() line 381 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #7 KSPSolve() line 612 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /home/rlchen/soft/petsc-3.8.4/src/snes/impls/ls/ls.c >>>>>> [0]PETSC ERROR: #9 SNESSolve() line 4131 in /home/rlchen/soft/petsc-3.8.4/src/snes/interface/snes.c >>>>>> [0]PETSC ERROR: #10 TimeStep() line 1751 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>> [0]PETSC ERROR: #11 SolveTimeDependent() line 1315 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>> [0]PETSC ERROR: #12 JoabCtxSolve() line 263 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>> [0]PETSC ERROR: #13 main() line 158 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>> >>>> >> From rongliang.chan at gmail.com Sat Apr 7 23:03:55 2018 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Sun, 8 Apr 2018 12:03:55 +0800 Subject: [petsc-users] Index set does not match blocks In-Reply-To: References: <09b11f09-9cf7-c081-cdaf-1ad2a49dc621@gmail.com> <86823E90-74A6-4CAF-AAF5-A474E6D88D38@anl.gov> <3c371769-c585-2218-e7d7-092768f1e87f@gmail.com> <3E6D2524-423A-4A00-9062-4AE2DD1EFDE1@mcs.anl.gov> <291727b9-5e48-f36c-9ea0-8eb9c5364acd@gmail.com> Message-ID: <9df6a184-6c01-ea24-eaad-cb39dbf832d2@gmail.com> Hi Barry, I tried the petsc-3.7.7 and my code works fine. Any suggestions for debug this? Thanks. Best, Rongliang On 04/08/2018 11:22 AM, Smith, Barry F. wrote: > >> On Apr 7, 2018, at 10:11 PM, Rongliang Chen wrote: >> >> Thanks Barry! I created the ISLocalToGlobalMapping after I loaded in the mesh and before the matrix creation. The block size for the ISLocalToGlobalMappingCreate is 7. In my code, the mapping is already in the form that one number per block since the petsc-3.6 requests this. So when I updated from 3.6 to 3.8, I did not make any changes for the ISLocalToGlobalMapping. The petsc-3.8 updated this mapping? And what should I do for this? > > The block change was from 3.5 to 3.6 so I am not sure what the problem is. > > is your code Fortran or C? > > Barry > > It's slower but you could try to see if you can get your code working with 3.7 first to determine if the problem comes up between 3.6 and 3.7 > >> Best, >> Rongliang >> >> On 04/08/2018 05:22 AM, Smith, Barry F. wrote: >>>> On Apr 7, 2018, at 1:57 PM, Rongliang Chen wrote: >>>> >>>> Hi Barry, >>>> >>>> Thanks for your reply. I checked the isrow with ISView (see bellow for the output) and I found that the output of the 3.6 (output-3.6.txt) and 3.8 (output-3.8.txt) are totally different. I do not know why. By the way, the block size of my code should be 7 but the output of the isrow in 3.8 shows that it is 8. >>>> >>>> I did not set the ASM blocks by myself. I only created a local to global mapping and use it to create the matrix with MatCreateBAIJ. The ASM setup is done by petsc. >>> There are some changes with the local to global mapping that is perhaps causing your difficulties. There use to be an MatSetLocalToGlobalMapping() and MatSetLocalToGlobalMappingBlock() they are gone and there is now a block size associated with the mapping. There is now one number per block instead of 1 number per index. Where do you get the ISLocalToGlobalMapping from() and what is its block size set too? >>> >>> Barry >>> >>>> Best, >>>> Rongliang >>>> >>>> >>>> On 04/07/2018 11:20 PM, Smith, Barry F. wrote: >>>>> For BAIJ matrices with ASM preconditioner the index sets that define the ASM blocks must contain all the points in a block or none of them. The code is detecting this is not the case; I don't know why this would be different between 3.8 and 3.6 >>>>> >>>>> You can use the option -start_in_debugger and put a break point in MatCreateSubMatrix_SeqBAIJ() then do >>>>> >>>>> call ISView(isrow,0) >>>>> >>>>> and look at the indices printed. Do the sets of indices match the block size? >>>>> >>>>> How do you set the ASM blocks for your code? They must be not set correctly when bs > 1. >>>>> >>>>> >>>>> Barry >>>>> >>>>> Perhaps in 3.6 we did less error checking and so did not detect the error in that case. >>>>> >>>>> >>>>> >>>>>> On Apr 7, 2018, at 12:24 AM, Rongliang Chen wrote: >>>>>> >>>>>> Hello everyone, >>>>>> >>>>>> >>>>>> I updated my code from petsc-3.6 to petsc-3.8 and got the following error. But it works well if I set the block size to be 1. I am solving an unstructured mesh problem. Any suggestions for debugging this are welcome. Thanks. >>>>>> >>>>>> >>>>>> Best, >>>>>> >>>>>> Rongliang >>>>>> >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Nonconforming object sizes >>>>>> [0]PETSC ERROR: Index set does not match blocks >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 >>>>>> [0]PETSC ERROR: ./HiBflow on a 64bit-debug named rongliang-chen by rlchen Sat Apr 7 13:14:20 2018 >>>>>> [0]PETSC ERROR: Configure options --download-mpich --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-debugging=1 --download-fblaslapack --download-ctetgen --with-64-bit-indices >>>>>> [0]PETSC ERROR: #1 MatCreateSubMatrix_SeqBAIJ() line 184 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>>>> [0]PETSC ERROR: #2 MatCreateSubMatrices_SeqBAIJ() line 264 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>>>> [0]PETSC ERROR: #3 MatCreateSubMatrices() line 6764 in /home/rlchen/soft/petsc-3.8.4/src/mat/interface/matrix.c >>>>>> [0]PETSC ERROR: #4 PCSetUp_ASM() line 326 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/impls/asm/asm.c >>>>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #6 KSPSetUp() line 381 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #7 KSPSolve() line 612 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /home/rlchen/soft/petsc-3.8.4/src/snes/impls/ls/ls.c >>>>>> [0]PETSC ERROR: #9 SNESSolve() line 4131 in /home/rlchen/soft/petsc-3.8.4/src/snes/interface/snes.c >>>>>> [0]PETSC ERROR: #10 TimeStep() line 1751 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>> [0]PETSC ERROR: #11 SolveTimeDependent() line 1315 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>> [0]PETSC ERROR: #12 JoabCtxSolve() line 263 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>> [0]PETSC ERROR: #13 main() line 158 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>> >>>> >> From bsmith at mcs.anl.gov Sun Apr 8 10:18:30 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 8 Apr 2018 15:18:30 +0000 Subject: [petsc-users] Index set does not match blocks In-Reply-To: <9df6a184-6c01-ea24-eaad-cb39dbf832d2@gmail.com> References: <09b11f09-9cf7-c081-cdaf-1ad2a49dc621@gmail.com> <86823E90-74A6-4CAF-AAF5-A474E6D88D38@anl.gov> <3c371769-c585-2218-e7d7-092768f1e87f@gmail.com> <3E6D2524-423A-4A00-9062-4AE2DD1EFDE1@mcs.anl.gov> <291727b9-5e48-f36c-9ea0-8eb9c5364acd@gmail.com> <9df6a184-6c01-ea24-eaad-cb39dbf832d2@gmail.com> Message-ID: <66F72FF5-4993-4AB5-9D33-09EC662A0898@mcs.anl.gov> > On Apr 7, 2018, at 11:03 PM, Rongliang Chen wrote: > > Hi Barry, > > I tried the petsc-3.7.7 and my code works fine. Any suggestions for debug this? Thanks. Works for 3.7.7 but not 3.8? I'm afraid you'll need to use git bisect, this will tell you exactly what change in PETSc breaks your code. Barry > > Best, > Rongliang > > On 04/08/2018 11:22 AM, Smith, Barry F. wrote: >> >>> On Apr 7, 2018, at 10:11 PM, Rongliang Chen wrote: >>> >>> Thanks Barry! I created the ISLocalToGlobalMapping after I loaded in the mesh and before the matrix creation. The block size for the ISLocalToGlobalMappingCreate is 7. In my code, the mapping is already in the form that one number per block since the petsc-3.6 requests this. So when I updated from 3.6 to 3.8, I did not make any changes for the ISLocalToGlobalMapping. The petsc-3.8 updated this mapping? And what should I do for this? >> The block change was from 3.5 to 3.6 so I am not sure what the problem is. >> >> is your code Fortran or C? >> >> Barry >> >> It's slower but you could try to see if you can get your code working with 3.7 first to determine if the problem comes up between 3.6 and 3.7 >> >>> Best, >>> Rongliang >>> >>> On 04/08/2018 05:22 AM, Smith, Barry F. wrote: >>>>> On Apr 7, 2018, at 1:57 PM, Rongliang Chen wrote: >>>>> >>>>> Hi Barry, >>>>> >>>>> Thanks for your reply. I checked the isrow with ISView (see bellow for the output) and I found that the output of the 3.6 (output-3.6.txt) and 3.8 (output-3.8.txt) are totally different. I do not know why. By the way, the block size of my code should be 7 but the output of the isrow in 3.8 shows that it is 8. >>>>> >>>>> I did not set the ASM blocks by myself. I only created a local to global mapping and use it to create the matrix with MatCreateBAIJ. The ASM setup is done by petsc. >>>> There are some changes with the local to global mapping that is perhaps causing your difficulties. There use to be an MatSetLocalToGlobalMapping() and MatSetLocalToGlobalMappingBlock() they are gone and there is now a block size associated with the mapping. There is now one number per block instead of 1 number per index. Where do you get the ISLocalToGlobalMapping from() and what is its block size set too? >>>> >>>> Barry >>>> >>>>> Best, >>>>> Rongliang >>>>> >>>>> >>>>> On 04/07/2018 11:20 PM, Smith, Barry F. wrote: >>>>>> For BAIJ matrices with ASM preconditioner the index sets that define the ASM blocks must contain all the points in a block or none of them. The code is detecting this is not the case; I don't know why this would be different between 3.8 and 3.6 >>>>>> >>>>>> You can use the option -start_in_debugger and put a break point in MatCreateSubMatrix_SeqBAIJ() then do >>>>>> >>>>>> call ISView(isrow,0) >>>>>> >>>>>> and look at the indices printed. Do the sets of indices match the block size? >>>>>> >>>>>> How do you set the ASM blocks for your code? They must be not set correctly when bs > 1. >>>>>> >>>>>> >>>>>> Barry >>>>>> >>>>>> Perhaps in 3.6 we did less error checking and so did not detect the error in that case. >>>>>> >>>>>> >>>>>> >>>>>>> On Apr 7, 2018, at 12:24 AM, Rongliang Chen wrote: >>>>>>> >>>>>>> Hello everyone, >>>>>>> >>>>>>> >>>>>>> I updated my code from petsc-3.6 to petsc-3.8 and got the following error. But it works well if I set the block size to be 1. I am solving an unstructured mesh problem. Any suggestions for debugging this are welcome. Thanks. >>>>>>> >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> Rongliang >>>>>>> >>>>>>> >>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Nonconforming object sizes >>>>>>> [0]PETSC ERROR: Index set does not match blocks >>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 >>>>>>> [0]PETSC ERROR: ./HiBflow on a 64bit-debug named rongliang-chen by rlchen Sat Apr 7 13:14:20 2018 >>>>>>> [0]PETSC ERROR: Configure options --download-mpich --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-debugging=1 --download-fblaslapack --download-ctetgen --with-64-bit-indices >>>>>>> [0]PETSC ERROR: #1 MatCreateSubMatrix_SeqBAIJ() line 184 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>>>>> [0]PETSC ERROR: #2 MatCreateSubMatrices_SeqBAIJ() line 264 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>>>>> [0]PETSC ERROR: #3 MatCreateSubMatrices() line 6764 in /home/rlchen/soft/petsc-3.8.4/src/mat/interface/matrix.c >>>>>>> [0]PETSC ERROR: #4 PCSetUp_ASM() line 326 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/impls/asm/asm.c >>>>>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/interface/precon.c >>>>>>> [0]PETSC ERROR: #6 KSPSetUp() line 381 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>>>>> [0]PETSC ERROR: #7 KSPSolve() line 612 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>>>>> [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /home/rlchen/soft/petsc-3.8.4/src/snes/impls/ls/ls.c >>>>>>> [0]PETSC ERROR: #9 SNESSolve() line 4131 in /home/rlchen/soft/petsc-3.8.4/src/snes/interface/snes.c >>>>>>> [0]PETSC ERROR: #10 TimeStep() line 1751 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>>> [0]PETSC ERROR: #11 SolveTimeDependent() line 1315 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>>> [0]PETSC ERROR: #12 JoabCtxSolve() line 263 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>>> [0]PETSC ERROR: #13 main() line 158 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>>> >>>>> >>> > From srossi at email.unc.edu Sun Apr 8 11:12:00 2018 From: srossi at email.unc.edu (Rossi, Simone) Date: Sun, 8 Apr 2018 16:12:00 +0000 Subject: [petsc-users] PCFIELDSPLIT prefix Message-ID: Hello, I?m trying to set a prefix for a ksp context (from the libMesh PetscLinearSolver) that solves a ?bidomain? problem. Currently I?m simply calling KSPSetOptionsPrefix(ksp,"bidomain_"); PCFieldSplitSetIS(ksp->pc,"v",is_v_local); PCFieldSplitSetIS(ksp->pc,"ve",is_ve_local); where is_v_local and is_ve_local are the local index sets of the 2 fields, v and ve. In this way, I can use the -bidomain_pc_type option to set the preconditioner. But using the FIELDSPLIT preconditioner with: -bidomain_pc_type fieldsplit I actually get the error: "Unhandled case, must have at least two fields, not 1? If I specify the block size -bidomain_pc_type fieldsplit -bidomain_pc_fieldsplit_block_size 2 The fields are not split using the provided index sets and the splitnames correspond to the number of the split, i.e. -bidomain_fieldsplit_0 -bidomain_fieldsplit_1 On the contrary using: -pc_type fieldsplit the preconditioner uses the index set provided in the code, and the splitnames are correct -bidomain_fieldsplit_v -bidomain_fieldsplit_ve How can I call "-bidomain_pc_type fieldsplit" and get the splitting that uses the index sets provided in the code? Thanks for the help, Simone From bsmith at mcs.anl.gov Sun Apr 8 11:24:07 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 8 Apr 2018 16:24:07 +0000 Subject: [petsc-users] PCFIELDSPLIT prefix In-Reply-To: References: Message-ID: <0374C775-06FC-4593-88C7-0BA16AB0AEFB@anl.gov> I don't understand your question but suspect that the issue is related to the order you are calling functions. You may need a KSPSetFromOptions() after the KSPSetOptionsPrefix() but before the PCFieldSplitSetIS() Barry Note that it is ok to have multiple calls to KSPSetFromOptions() for the same KSP but depending on where you set the options prefix they may have different results. > On Apr 8, 2018, at 11:12 AM, Rossi, Simone wrote: > > Hello, > I?m trying to set a prefix for a ksp context (from the libMesh PetscLinearSolver) that solves a ?bidomain? problem. > > Currently I?m simply calling > > KSPSetOptionsPrefix(ksp,"bidomain_"); > PCFieldSplitSetIS(ksp->pc,"v",is_v_local); > PCFieldSplitSetIS(ksp->pc,"ve",is_ve_local); > > where is_v_local and is_ve_local are the local index sets of the 2 fields, v and ve. > > In this way, I can use the -bidomain_pc_type option to set the preconditioner. But using the FIELDSPLIT preconditioner with: > -bidomain_pc_type fieldsplit > I actually get the error: > "Unhandled case, must have at least two fields, not 1? > If I specify the block size > -bidomain_pc_type fieldsplit -bidomain_pc_fieldsplit_block_size 2 > The fields are not split using the provided index sets and the splitnames correspond to the number of the split, i.e. > -bidomain_fieldsplit_0 > -bidomain_fieldsplit_1 > > > On the contrary using: > -pc_type fieldsplit > the preconditioner uses the index set provided in the code, and the splitnames are correct > -bidomain_fieldsplit_v > -bidomain_fieldsplit_ve > > How can I call "-bidomain_pc_type fieldsplit" and get the splitting that uses the index sets provided in the code? > > Thanks for the help, > Simone > From srossi at email.unc.edu Sun Apr 8 11:39:33 2018 From: srossi at email.unc.edu (Rossi, Simone) Date: Sun, 8 Apr 2018 16:39:33 +0000 Subject: [petsc-users] PCFIELDSPLIT prefix In-Reply-To: <0374C775-06FC-4593-88C7-0BA16AB0AEFB@anl.gov> References: <0374C775-06FC-4593-88C7-0BA16AB0AEFB@anl.gov> Message-ID: <5FF2E11A-E93C-422A-862A-0FA32388F6BB@ad.unc.edu> Thanks Barry, That did it: I only had to specify KSPSetFromOptions() between KSPSetOptionsPrefix() and PCFieldSplitSetIS(). May I ask you how does that work? What do you mean by ?depending on where you set the options prefix they may have different results?? I could not find a clear explanation in the documentation. Thanks again for the help, Simone On Apr 8, 2018, at 12:24 PM, Smith, Barry F. > wrote: KSPSetFromOptions() -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Apr 8 11:42:50 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 8 Apr 2018 16:42:50 +0000 Subject: [petsc-users] PCFIELDSPLIT prefix In-Reply-To: <5FF2E11A-E93C-422A-862A-0FA32388F6BB@ad.unc.edu> References: <0374C775-06FC-4593-88C7-0BA16AB0AEFB@anl.gov> <5FF2E11A-E93C-422A-862A-0FA32388F6BB@ad.unc.edu> Message-ID: <506121FE-F018-4726-8B7E-D000FDCAC4CB@mcs.anl.gov> > On Apr 8, 2018, at 11:39 AM, Rossi, Simone wrote: > > Thanks Barry, > That did it: I only had to specify KSPSetFromOptions() between KSPSetOptionsPrefix() and PCFieldSplitSetIS(). > > May I ask you how does that work? What do you mean by ?depending on where you set the options prefix they may have different results?? I could not find a clear explanation in the documentation. The KSPSetFromOptions() uses the -current- list of options and -current- values of prefixes (that is what the values are when the KSPSetFromOptions is called). If you change and option or options prefix that change won't effect anything unless you call the KSPSetFromOptions() after the change. Barry > > Thanks again for the help, > Simone > >> On Apr 8, 2018, at 12:24 PM, Smith, Barry F. wrote: >> >> KSPSetFromOptions() > From srossi at email.unc.edu Sun Apr 8 15:08:44 2018 From: srossi at email.unc.edu (Rossi, Simone) Date: Sun, 8 Apr 2018 20:08:44 +0000 Subject: [petsc-users] PCFIELDSPLIT prefix In-Reply-To: <506121FE-F018-4726-8B7E-D000FDCAC4CB@mcs.anl.gov> References: <0374C775-06FC-4593-88C7-0BA16AB0AEFB@anl.gov> <5FF2E11A-E93C-422A-862A-0FA32388F6BB@ad.unc.edu> <506121FE-F018-4726-8B7E-D000FDCAC4CB@mcs.anl.gov> Message-ID: <9F45406D-31AA-4A82-8295-D242DE83819B@ad.unc.edu> That?s good to know. Thanks. Simone > On Apr 8, 2018, at 12:42 PM, Smith, Barry F. wrote: > > > >> On Apr 8, 2018, at 11:39 AM, Rossi, Simone wrote: >> >> Thanks Barry, >> That did it: I only had to specify KSPSetFromOptions() between KSPSetOptionsPrefix() and PCFieldSplitSetIS(). >> >> May I ask you how does that work? What do you mean by ?depending on where you set the options prefix they may have different results?? I could not find a clear explanation in the documentation. > > The KSPSetFromOptions() uses the -current- list of options and -current- values of prefixes (that is what the values are when the KSPSetFromOptions is called). If you change and option or options prefix that change won't effect anything unless you call the KSPSetFromOptions() after the change. > > > Barry > >> >> Thanks again for the help, >> Simone >> >>> On Apr 8, 2018, at 12:24 PM, Smith, Barry F. wrote: >>> >>> KSPSetFromOptions() >> > From rongliang.chan at gmail.com Sun Apr 8 19:17:05 2018 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Mon, 9 Apr 2018 08:17:05 +0800 Subject: [petsc-users] Index set does not match blocks In-Reply-To: <66F72FF5-4993-4AB5-9D33-09EC662A0898@mcs.anl.gov> References: <09b11f09-9cf7-c081-cdaf-1ad2a49dc621@gmail.com> <86823E90-74A6-4CAF-AAF5-A474E6D88D38@anl.gov> <3c371769-c585-2218-e7d7-092768f1e87f@gmail.com> <3E6D2524-423A-4A00-9062-4AE2DD1EFDE1@mcs.anl.gov> <291727b9-5e48-f36c-9ea0-8eb9c5364acd@gmail.com> <9df6a184-6c01-ea24-eaad-cb39dbf832d2@gmail.com> <66F72FF5-4993-4AB5-9D33-09EC662A0898@mcs.anl.gov> Message-ID: Thanks Barry. I will try it but it seems highly nontrivial. Best, Rongliang On 04/08/2018 11:18 PM, Smith, Barry F. wrote: > >> On Apr 7, 2018, at 11:03 PM, Rongliang Chen wrote: >> >> Hi Barry, >> >> I tried the petsc-3.7.7 and my code works fine. Any suggestions for debug this? Thanks. > Works for 3.7.7 but not 3.8? I'm afraid you'll need to use git bisect, this will tell you exactly what change in PETSc breaks your code. > > Barry > >> Best, >> Rongliang >> >> On 04/08/2018 11:22 AM, Smith, Barry F. wrote: >>>> On Apr 7, 2018, at 10:11 PM, Rongliang Chen wrote: >>>> >>>> Thanks Barry! I created the ISLocalToGlobalMapping after I loaded in the mesh and before the matrix creation. The block size for the ISLocalToGlobalMappingCreate is 7. In my code, the mapping is already in the form that one number per block since the petsc-3.6 requests this. So when I updated from 3.6 to 3.8, I did not make any changes for the ISLocalToGlobalMapping. The petsc-3.8 updated this mapping? And what should I do for this? >>> The block change was from 3.5 to 3.6 so I am not sure what the problem is. >>> >>> is your code Fortran or C? >>> >>> Barry >>> >>> It's slower but you could try to see if you can get your code working with 3.7 first to determine if the problem comes up between 3.6 and 3.7 >>> >>>> Best, >>>> Rongliang >>>> >>>> On 04/08/2018 05:22 AM, Smith, Barry F. wrote: >>>>>> On Apr 7, 2018, at 1:57 PM, Rongliang Chen wrote: >>>>>> >>>>>> Hi Barry, >>>>>> >>>>>> Thanks for your reply. I checked the isrow with ISView (see bellow for the output) and I found that the output of the 3.6 (output-3.6.txt) and 3.8 (output-3.8.txt) are totally different. I do not know why. By the way, the block size of my code should be 7 but the output of the isrow in 3.8 shows that it is 8. >>>>>> >>>>>> I did not set the ASM blocks by myself. I only created a local to global mapping and use it to create the matrix with MatCreateBAIJ. The ASM setup is done by petsc. >>>>> There are some changes with the local to global mapping that is perhaps causing your difficulties. There use to be an MatSetLocalToGlobalMapping() and MatSetLocalToGlobalMappingBlock() they are gone and there is now a block size associated with the mapping. There is now one number per block instead of 1 number per index. Where do you get the ISLocalToGlobalMapping from() and what is its block size set too? >>>>> >>>>> Barry >>>>> >>>>>> Best, >>>>>> Rongliang >>>>>> >>>>>> >>>>>> On 04/07/2018 11:20 PM, Smith, Barry F. wrote: >>>>>>> For BAIJ matrices with ASM preconditioner the index sets that define the ASM blocks must contain all the points in a block or none of them. The code is detecting this is not the case; I don't know why this would be different between 3.8 and 3.6 >>>>>>> >>>>>>> You can use the option -start_in_debugger and put a break point in MatCreateSubMatrix_SeqBAIJ() then do >>>>>>> >>>>>>> call ISView(isrow,0) >>>>>>> >>>>>>> and look at the indices printed. Do the sets of indices match the block size? >>>>>>> >>>>>>> How do you set the ASM blocks for your code? They must be not set correctly when bs > 1. >>>>>>> >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> Perhaps in 3.6 we did less error checking and so did not detect the error in that case. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Apr 7, 2018, at 12:24 AM, Rongliang Chen wrote: >>>>>>>> >>>>>>>> Hello everyone, >>>>>>>> >>>>>>>> >>>>>>>> I updated my code from petsc-3.6 to petsc-3.8 and got the following error. But it works well if I set the block size to be 1. I am solving an unstructured mesh problem. Any suggestions for debugging this are welcome. Thanks. >>>>>>>> >>>>>>>> >>>>>>>> Best, >>>>>>>> >>>>>>>> Rongliang >>>>>>>> >>>>>>>> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: Nonconforming object sizes >>>>>>>> [0]PETSC ERROR: Index set does not match blocks >>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 >>>>>>>> [0]PETSC ERROR: ./HiBflow on a 64bit-debug named rongliang-chen by rlchen Sat Apr 7 13:14:20 2018 >>>>>>>> [0]PETSC ERROR: Configure options --download-mpich --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-debugging=1 --download-fblaslapack --download-ctetgen --with-64-bit-indices >>>>>>>> [0]PETSC ERROR: #1 MatCreateSubMatrix_SeqBAIJ() line 184 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>>>>>> [0]PETSC ERROR: #2 MatCreateSubMatrices_SeqBAIJ() line 264 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>>>>>> [0]PETSC ERROR: #3 MatCreateSubMatrices() line 6764 in /home/rlchen/soft/petsc-3.8.4/src/mat/interface/matrix.c >>>>>>>> [0]PETSC ERROR: #4 PCSetUp_ASM() line 326 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/impls/asm/asm.c >>>>>>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/interface/precon.c >>>>>>>> [0]PETSC ERROR: #6 KSPSetUp() line 381 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>>>>>> [0]PETSC ERROR: #7 KSPSolve() line 612 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>>>>>> [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /home/rlchen/soft/petsc-3.8.4/src/snes/impls/ls/ls.c >>>>>>>> [0]PETSC ERROR: #9 SNESSolve() line 4131 in /home/rlchen/soft/petsc-3.8.4/src/snes/interface/snes.c >>>>>>>> [0]PETSC ERROR: #10 TimeStep() line 1751 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>>>> [0]PETSC ERROR: #11 SolveTimeDependent() line 1315 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>>>> [0]PETSC ERROR: #12 JoabCtxSolve() line 263 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>>>> [0]PETSC ERROR: #13 main() line 158 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>>>> >>>>>> From bsmith at mcs.anl.gov Sun Apr 8 19:46:13 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 9 Apr 2018 00:46:13 +0000 Subject: [petsc-users] Index set does not match blocks In-Reply-To: References: <09b11f09-9cf7-c081-cdaf-1ad2a49dc621@gmail.com> <86823E90-74A6-4CAF-AAF5-A474E6D88D38@anl.gov> <3c371769-c585-2218-e7d7-092768f1e87f@gmail.com> <3E6D2524-423A-4A00-9062-4AE2DD1EFDE1@mcs.anl.gov> <291727b9-5e48-f36c-9ea0-8eb9c5364acd@gmail.com> <9df6a184-6c01-ea24-eaad-cb39dbf832d2@gmail.com> <66F72FF5-4993-4AB5-9D33-09EC662A0898@mcs.anl.gov> Message-ID: <42DCF9D6-C20A-4A5C-8261-BC5A95D123E3@mcs.anl.gov> > On Apr 8, 2018, at 7:17 PM, Rongliang Chen wrote: > > Thanks Barry. I will try it but it seems highly nontrivial. The first time you do it it seems overwhelming but once you get the hang of it it is actually pretty simple and very powerful. Much better than random guesses at what could be causing the problem. Barry > > Best, > Rongliang > > On 04/08/2018 11:18 PM, Smith, Barry F. wrote: >> >>> On Apr 7, 2018, at 11:03 PM, Rongliang Chen wrote: >>> >>> Hi Barry, >>> >>> I tried the petsc-3.7.7 and my code works fine. Any suggestions for debug this? Thanks. >> Works for 3.7.7 but not 3.8? I'm afraid you'll need to use git bisect, this will tell you exactly what change in PETSc breaks your code. >> >> Barry >> >>> Best, >>> Rongliang >>> >>> On 04/08/2018 11:22 AM, Smith, Barry F. wrote: >>>>> On Apr 7, 2018, at 10:11 PM, Rongliang Chen wrote: >>>>> >>>>> Thanks Barry! I created the ISLocalToGlobalMapping after I loaded in the mesh and before the matrix creation. The block size for the ISLocalToGlobalMappingCreate is 7. In my code, the mapping is already in the form that one number per block since the petsc-3.6 requests this. So when I updated from 3.6 to 3.8, I did not make any changes for the ISLocalToGlobalMapping. The petsc-3.8 updated this mapping? And what should I do for this? >>>> The block change was from 3.5 to 3.6 so I am not sure what the problem is. >>>> >>>> is your code Fortran or C? >>>> >>>> Barry >>>> >>>> It's slower but you could try to see if you can get your code working with 3.7 first to determine if the problem comes up between 3.6 and 3.7 >>>> >>>>> Best, >>>>> Rongliang >>>>> >>>>> On 04/08/2018 05:22 AM, Smith, Barry F. wrote: >>>>>>> On Apr 7, 2018, at 1:57 PM, Rongliang Chen wrote: >>>>>>> >>>>>>> Hi Barry, >>>>>>> >>>>>>> Thanks for your reply. I checked the isrow with ISView (see bellow for the output) and I found that the output of the 3.6 (output-3.6.txt) and 3.8 (output-3.8.txt) are totally different. I do not know why. By the way, the block size of my code should be 7 but the output of the isrow in 3.8 shows that it is 8. >>>>>>> >>>>>>> I did not set the ASM blocks by myself. I only created a local to global mapping and use it to create the matrix with MatCreateBAIJ. The ASM setup is done by petsc. >>>>>> There are some changes with the local to global mapping that is perhaps causing your difficulties. There use to be an MatSetLocalToGlobalMapping() and MatSetLocalToGlobalMappingBlock() they are gone and there is now a block size associated with the mapping. There is now one number per block instead of 1 number per index. Where do you get the ISLocalToGlobalMapping from() and what is its block size set too? >>>>>> >>>>>> Barry >>>>>> >>>>>>> Best, >>>>>>> Rongliang >>>>>>> >>>>>>> >>>>>>> On 04/07/2018 11:20 PM, Smith, Barry F. wrote: >>>>>>>> For BAIJ matrices with ASM preconditioner the index sets that define the ASM blocks must contain all the points in a block or none of them. The code is detecting this is not the case; I don't know why this would be different between 3.8 and 3.6 >>>>>>>> >>>>>>>> You can use the option -start_in_debugger and put a break point in MatCreateSubMatrix_SeqBAIJ() then do >>>>>>>> >>>>>>>> call ISView(isrow,0) >>>>>>>> >>>>>>>> and look at the indices printed. Do the sets of indices match the block size? >>>>>>>> >>>>>>>> How do you set the ASM blocks for your code? They must be not set correctly when bs > 1. >>>>>>>> >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> Perhaps in 3.6 we did less error checking and so did not detect the error in that case. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Apr 7, 2018, at 12:24 AM, Rongliang Chen wrote: >>>>>>>>> >>>>>>>>> Hello everyone, >>>>>>>>> >>>>>>>>> >>>>>>>>> I updated my code from petsc-3.6 to petsc-3.8 and got the following error. But it works well if I set the block size to be 1. I am solving an unstructured mesh problem. Any suggestions for debugging this are welcome. Thanks. >>>>>>>>> >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> >>>>>>>>> Rongliang >>>>>>>>> >>>>>>>>> >>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>> [0]PETSC ERROR: Nonconforming object sizes >>>>>>>>> [0]PETSC ERROR: Index set does not match blocks >>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 >>>>>>>>> [0]PETSC ERROR: ./HiBflow on a 64bit-debug named rongliang-chen by rlchen Sat Apr 7 13:14:20 2018 >>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-debugging=1 --download-fblaslapack --download-ctetgen --with-64-bit-indices >>>>>>>>> [0]PETSC ERROR: #1 MatCreateSubMatrix_SeqBAIJ() line 184 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>>>>>>> [0]PETSC ERROR: #2 MatCreateSubMatrices_SeqBAIJ() line 264 in /home/rlchen/soft/petsc-3.8.4/src/mat/impls/baij/seq/baij2.c >>>>>>>>> [0]PETSC ERROR: #3 MatCreateSubMatrices() line 6764 in /home/rlchen/soft/petsc-3.8.4/src/mat/interface/matrix.c >>>>>>>>> [0]PETSC ERROR: #4 PCSetUp_ASM() line 326 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/impls/asm/asm.c >>>>>>>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/rlchen/soft/petsc-3.8.4/src/ksp/pc/interface/precon.c >>>>>>>>> [0]PETSC ERROR: #6 KSPSetUp() line 381 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() line 612 in /home/rlchen/soft/petsc-3.8.4/src/ksp/ksp/interface/itfunc.c >>>>>>>>> [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /home/rlchen/soft/petsc-3.8.4/src/snes/impls/ls/ls.c >>>>>>>>> [0]PETSC ERROR: #9 SNESSolve() line 4131 in /home/rlchen/soft/petsc-3.8.4/src/snes/interface/snes.c >>>>>>>>> [0]PETSC ERROR: #10 TimeStep() line 1751 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>>>>> [0]PETSC ERROR: #11 SolveTimeDependent() line 1315 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>>>>> [0]PETSC ERROR: #12 JoabCtxSolve() line 263 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>>>>> [0]PETSC ERROR: #13 main() line 158 in /home/rlchen/soft/3D_fluid/3DBloodFlow/Cerebral/Twolevel-3.8/HiBflow.c >>>>>>>>> >>>>>>> > > From st.chu at outlook.com Mon Apr 9 07:55:16 2018 From: st.chu at outlook.com (Chu Songtao) Date: Mon, 9 Apr 2018 12:55:16 +0000 Subject: [petsc-users] How to access Vector data properly Message-ID: Hello, I am just a beginner. Now I get confused on how to correctly use Vec and DMDA. I set 1D DMDA global vector with natural values for a test. Then use VecView and DMDAVecGetArray to view the data, but the results are different. I don't know why. this is the code: PetscMPIInt rank; PetscErrorCode ierr; Vec global,local,natural; DM da; PetscReal *xnatural,*val; PetscInt i,start,end,x,xm; ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); ierr = DMDACreateNaturalVector(da,&natural);CHKERRQ(ierr); ierr = VecGetOwnershipRange(natural,&start,&end);CHKERRQ(ierr); ierr = VecGetArray(natural,&xnatural);CHKERRQ(ierr); for (i=start; i From hzhang at mcs.anl.gov Mon Apr 9 09:04:31 2018 From: hzhang at mcs.anl.gov (Hong) Date: Mon, 9 Apr 2018 09:04:31 -0500 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: Message-ID: Songtao : You may need adding PetscSynchronizedFlush() to the end. See petsc/src/dm/examples/tests/ex43.c Hong > Hello, > I am just a beginner. Now I get confused on how to correctly use Vec > and DMDA. > I set 1D DMDA global vector with natural values for a test. Then use > VecView and DMDAVecGetArray to view the data, but the results are > different. I don't know why. > > this is the code: > > PetscMPIInt rank; > PetscErrorCode ierr; > Vec global,local,natural; > DM da; > PetscReal *xnatural,*val; > PetscInt i,start,end,x,xm; > > > ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > ierr = DMSetUp(da);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > ierr = DMDACreateNaturalVector(da,&natural);CHKERRQ(ierr); > ierr = VecGetOwnershipRange(natural,&start,&end);CHKERRQ(ierr); > ierr = VecGetArray(natural,&xnatural);CHKERRQ(ierr); > for (i=start; i xnatural[i-start] = i; > } > ierr = VecRestoreArray(natural,&xnatural);CHKERRQ(ierr); > ierr = DMDANaturalToGlobalBegin(da,natural,INSERT_VALUES,global);CHKERRQ(ierr); > ierr = DMDANaturalToGlobalEnd(da,natural,INSERT_VALUES,global);CHKERRQ(ierr); > ierr = VecDestroy(&natural);CHKERRQ(ierr); > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > and the results are: > > $ ./test0 > Vec Object: 1 MPI processes > type: seq > Vec Object: Vec_0x7ffff3cbc500_0 1 MPI processes > type: mpi > Process [0] > 0. > 1. > 2. > 3. > 4. > Rank=0 > 0 1 2 3 4 > > $ mpiexec -n 2 ./test0 > Vec Object: 2 MPI processes > type: mpi > Vec Object: Vec_0x7fffcf948a90_0 2 MPI processes > type: mpi > Process [0] > 0. > 1. > 2. > Process [1] > 3. > 4. > Rank=0 > 0 0 0 Rank=1 > 0 0 > > > the seq version is correct, but the mpi version is wrong. Every element in > val is 0. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From st.chu at outlook.com Mon Apr 9 10:19:54 2018 From: st.chu at outlook.com (Chu Songtao) Date: Mon, 9 Apr 2018 15:19:54 +0000 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: , Message-ID: Hong, I added the function to the end, but the results didn't change. Then I tested another thing, assigning global by Get(Restore)Array instead of DMDANaturalToGlobal. And the results switched. VecView outputs 0 and PetscSynchronizedPrintf outputs correct values. Strange. ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { val[i] = i; } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); VecView(global,PETSC_VIEWER_STDOUT_WORLD); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); $ mpiexec -n 2 ./test0 Rank=0 Rank=1 Vec Object: 2 MPI processes type: mpi Vec Object: Vec_0x7fffd6629a90_0 2 MPI processes type: mpi Rank=1 3 4 Process [0] 0. 0. 0. Process [1] 0. 0. Rank=0 0 1 2 ________________________________ Songtao : You may need adding PetscSynchronizedFlush() to the end. See petsc/src/dm/examples/tests/ex43.c Hong Hello, I am just a beginner. Now I get confused on how to correctly use Vec and DMDA. I set 1D DMDA global vector with natural values for a test. Then use VecView and DMDAVecGetArray to view the data, but the results are different. I don't know why. this is the code: PetscMPIInt rank; PetscErrorCode ierr; Vec global,local,natural; DM da; PetscReal *xnatural,*val; PetscInt i,start,end,x,xm; ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); ierr = DMDACreateNaturalVector(da,&natural);CHKERRQ(ierr); ierr = VecGetOwnershipRange(natural,&start,&end);CHKERRQ(ierr); ierr = VecGetArray(natural,&xnatural);CHKERRQ(ierr); for (i=start; i From knepley at gmail.com Mon Apr 9 10:29:36 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Apr 2018 11:29:36 -0400 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: Message-ID: On Mon, Apr 9, 2018 at 11:19 AM, Chu Songtao wrote: > Hong, > I added the function to the end, but the results didn't change. > Then I tested another thing, assigning global by Get(Restore)Array instead > of DMDANaturalToGlobal. And the results switched. VecView outputs 0 > and PetscSynchronizedPrintf outputs correct values. Strange. > Send the whole code. You have a logic error somewhere. Matt > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > ierr = DMSetUp(da);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > val[i] = i; > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); > > $ mpiexec -n 2 ./test0 > Rank=0 > Rank=1 > Vec Object: 2 MPI processes > type: mpi > Vec Object: Vec_0x7fffd6629a90_0 2 MPI processes > type: mpi > Rank=1 > 3 4 Process [0] > 0. > 0. > 0. > Process [1] > 0. > 0. > Rank=0 > 0 1 2 > > > ------------------------------ > > Songtao : > You may need adding PetscSynchronizedFlush() to the end. > See petsc/src/dm/examples/tests/ex43.c > Hong > > Hello, > I am just a beginner. Now I get confused on how to correctly use Vec > and DMDA. > I set 1D DMDA global vector with natural values for a test. Then use > VecView and DMDAVecGetArray to view the data, but the results are > different. I don't know why. > > this is the code: > > PetscMPIInt rank; > PetscErrorCode ierr; > Vec global,local,natural; > DM da; > PetscReal *xnatural,*val; > PetscInt i,start,end,x,xm; > > > ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > ierr = DMSetUp(da);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > ierr = DMDACreateNaturalVector(da,&natural);CHKERRQ(ierr); > ierr = VecGetOwnershipRange(natural,&start,&end);CHKERRQ(ierr); > ierr = VecGetArray(natural,&xnatural);CHKERRQ(ierr); > for (i=start; i xnatural[i-start] = i; > } > ierr = VecRestoreArray(natural,&xnatural);CHKERRQ(ierr); > ierr = DMDANaturalToGlobalBegin(da,natural,INSERT_VALUES,global);CHKERRQ(ierr); > ierr = DMDANaturalToGlobalEnd(da,natural,INSERT_VALUES,global);CHKERRQ(ierr); > ierr = VecDestroy(&natural);CHKERRQ(ierr); > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > and the results are: > > $ ./test0 > Vec Object: 1 MPI processes > type: seq > Vec Object: Vec_0x7ffff3cbc500_0 1 MPI processes > type: mpi > Process [0] > 0. > 1. > 2. > 3. > 4. > Rank=0 > 0 1 2 3 4 > > $ mpiexec -n 2 ./test0 > Vec Object: 2 MPI processes > type: mpi > Vec Object: Vec_0x7fffcf948a90_0 2 MPI processes > type: mpi > Process [0] > 0. > 1. > 2. > Process [1] > 3. > 4. > Rank=0 > 0 0 0 Rank=1 > 0 0 > > > the seq version is correct, but the mpi version is wrong. Every element in > val is 0. > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From st.chu at outlook.com Mon Apr 9 10:51:30 2018 From: st.chu at outlook.com (Songtao Chu) Date: Mon, 9 Apr 2018 15:51:30 +0000 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: , Message-ID: static char help[] = "\n\n"; #include #include int main(int argc,char **argv) { PetscMPIInt rank; PetscErrorCode ierr; Vec global,local,natural; DM da; PetscReal *val; PetscInt i,x,xm; ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { val[i] = i; } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); VecView(global,PETSC_VIEWER_STDOUT_WORLD); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); ierr = PetscFinalize(); return ierr; } ________________________________ From: Matthew Knepley Sent: Monday, April 9, 2018 23:29 To: Chu Songtao Cc: Hong; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] How to access Vector data properly On Mon, Apr 9, 2018 at 11:19 AM, Chu Songtao > wrote: Hong, I added the function to the end, but the results didn't change. Then I tested another thing, assigning global by Get(Restore)Array instead of DMDANaturalToGlobal. And the results switched. VecView outputs 0 and PetscSynchronizedPrintf outputs correct values. Strange. Send the whole code. You have a logic error somewhere. Matt ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { val[i] = i; } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); VecView(global,PETSC_VIEWER_STDOUT_WORLD); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); $ mpiexec -n 2 ./test0 Rank=0 Rank=1 Vec Object: 2 MPI processes type: mpi Vec Object: Vec_0x7fffd6629a90_0 2 MPI processes type: mpi Rank=1 3 4 Process [0] 0. 0. 0. Process [1] 0. 0. Rank=0 0 1 2 ________________________________ Songtao : You may need adding PetscSynchronizedFlush() to the end. See petsc/src/dm/examples/tests/ex43.c Hong Hello, I am just a beginner. Now I get confused on how to correctly use Vec and DMDA. I set 1D DMDA global vector with natural values for a test. Then use VecView and DMDAVecGetArray to view the data, but the results are different. I don't know why. this is the code: PetscMPIInt rank; PetscErrorCode ierr; Vec global,local,natural; DM da; PetscReal *xnatural,*val; PetscInt i,start,end,x,xm; ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); ierr = DMDACreateNaturalVector(da,&natural);CHKERRQ(ierr); ierr = VecGetOwnershipRange(natural,&start,&end);CHKERRQ(ierr); ierr = VecGetArray(natural,&xnatural);CHKERRQ(ierr); for (i=start; i -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 9 11:01:10 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Apr 2018 12:01:10 -0400 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: Message-ID: This works fine for me: knepley/feature-pc-patch $:/PETSc3/petsc/petsc-pylith/tmp$ ../arch-master-debug/bin/mpiexec -n 2 ./tester Vec Object: 2 MPI processes type: mpi Process [0] 0. 1. 2. Process [1] 3. 4. Rank=0 0 1 2 Rank=1 3 4 What is the problem? Matt On Mon, Apr 9, 2018 at 11:51 AM, Songtao Chu wrote: > static char help[] = "\n\n"; > #include > #include > > > int main(int argc,char **argv) > { > PetscMPIInt rank; > PetscErrorCode ierr; > Vec global,local,natural; > DM da; > PetscReal *val; > PetscInt i,x,xm; > > > ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > ierr = DMSetUp(da);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > val[i] = i; > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); > > ierr = PetscFinalize(); > return ierr; > } > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 9, 2018 23:29 > *To:* Chu Songtao > *Cc:* Hong; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] How to access Vector data properly > > On Mon, Apr 9, 2018 at 11:19 AM, Chu Songtao wrote: > > Hong, > I added the function to the end, but the results didn't change. > Then I tested another thing, assigning global by Get(Restore)Array instead > of DMDANaturalToGlobal. And the results switched. VecView outputs 0 > and PetscSynchronizedPrintf outputs correct values. Strange. > > > Send the whole code. You have a logic error somewhere. > > Matt > > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > ierr = DMSetUp(da);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > val[i] = i; > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); > > $ mpiexec -n 2 ./test0 > Rank=0 > Rank=1 > Vec Object: 2 MPI processes > type: mpi > Vec Object: Vec_0x7fffd6629a90_0 2 MPI processes > type: mpi > Rank=1 > 3 4 Process [0] > 0. > 0. > 0. > Process [1] > 0. > 0. > Rank=0 > 0 1 2 > > > ------------------------------ > > Songtao : > You may need adding PetscSynchronizedFlush() to the end. > See petsc/src/dm/examples/tests/ex43.c > Hong > > Hello, > I am just a beginner. Now I get confused on how to correctly use Vec > and DMDA. > I set 1D DMDA global vector with natural values for a test. Then use > VecView and DMDAVecGetArray to view the data, but the results are > different. I don't know why. > > this is the code: > > PetscMPIInt rank; > PetscErrorCode ierr; > Vec global,local,natural; > DM da; > PetscReal *xnatural,*val; > PetscInt i,start,end,x,xm; > > > ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > ierr = DMSetUp(da);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > ierr = DMDACreateNaturalVector(da,&natural);CHKERRQ(ierr); > ierr = VecGetOwnershipRange(natural,&start,&end);CHKERRQ(ierr); > ierr = VecGetArray(natural,&xnatural);CHKERRQ(ierr); > for (i=start; i xnatural[i-start] = i; > } > ierr = VecRestoreArray(natural,&xnatural);CHKERRQ(ierr); > ierr = DMDANaturalToGlobalBegin(da,natural,INSERT_VALUES,global);CHKERRQ(ierr); > ierr = DMDANaturalToGlobalEnd(da,natural,INSERT_VALUES,global);CHKERRQ(ierr); > ierr = VecDestroy(&natural);CHKERRQ(ierr); > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > and the results are: > > $ ./test0 > Vec Object: 1 MPI processes > type: seq > Vec Object: Vec_0x7ffff3cbc500_0 1 MPI processes > type: mpi > Process [0] > 0. > 1. > 2. > 3. > 4. > Rank=0 > 0 1 2 3 4 > > $ mpiexec -n 2 ./test0 > Vec Object: 2 MPI processes > type: mpi > Vec Object: Vec_0x7fffcf948a90_0 2 MPI processes > type: mpi > Process [0] > 0. > 1. > 2. > Process [1] > 3. > 4. > Rank=0 > 0 0 0 Rank=1 > 0 0 > > > the seq version is correct, but the mpi version is wrong. Every element in > val is 0. > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tester.c Type: application/octet-stream Size: 1623 bytes Desc: not available URL: From st.chu at outlook.com Mon Apr 9 11:13:39 2018 From: st.chu at outlook.com (Songtao Chu) Date: Mon, 9 Apr 2018 16:13:39 +0000 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: , Message-ID: The two output results are different in my test. If I use the way of assignment variables in current code. I got error output of VecView as below: Vec Object: 2 MPI processes type: mpi Vec Object: Vec_0x7fffd6895a90_0 2 MPI processes type: mpi Process [0] 0. 0. 0. Process [1] 0. 0. Or if I use the previous way (DMDANaturalToGlobal), I got error output of PetscSynchronizedPrintf: Rank=0 0 0 0 Rank=1 0 0 Both are fine if I run without mpiexec. Besides, my development environment is CLion + WSL + GCC 7.3. ________________________________ From: Matthew Knepley Sent: Tuesday, April 10, 2018 0:01 To: Songtao Chu Cc: Hong; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] How to access Vector data properly This works fine for me: knepley/feature-pc-patch $:/PETSc3/petsc/petsc-pylith/tmp$ ../arch-master-debug/bin/mpiexec -n 2 ./tester Vec Object: 2 MPI processes type: mpi Process [0] 0. 1. 2. Process [1] 3. 4. Rank=0 0 1 2 Rank=1 3 4 What is the problem? Matt On Mon, Apr 9, 2018 at 11:51 AM, Songtao Chu > wrote: static char help[] = "\n\n"; #include #include int main(int argc,char **argv) { PetscMPIInt rank; PetscErrorCode ierr; Vec global,local,natural; DM da; PetscReal *val; PetscInt i,x,xm; ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { val[i] = i; } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); VecView(global,PETSC_VIEWER_STDOUT_WORLD); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); ierr = PetscFinalize(); return ierr; } ________________________________ From: Matthew Knepley > Sent: Monday, April 9, 2018 23:29 To: Chu Songtao Cc: Hong; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] How to access Vector data properly On Mon, Apr 9, 2018 at 11:19 AM, Chu Songtao > wrote: Hong, I added the function to the end, but the results didn't change. Then I tested another thing, assigning global by Get(Restore)Array instead of DMDANaturalToGlobal. And the results switched. VecView outputs 0 and PetscSynchronizedPrintf outputs correct values. Strange. Send the whole code. You have a logic error somewhere. Matt ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { val[i] = i; } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); VecView(global,PETSC_VIEWER_STDOUT_WORLD); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); $ mpiexec -n 2 ./test0 Rank=0 Rank=1 Vec Object: 2 MPI processes type: mpi Vec Object: Vec_0x7fffd6629a90_0 2 MPI processes type: mpi Rank=1 3 4 Process [0] 0. 0. 0. Process [1] 0. 0. Rank=0 0 1 2 ________________________________ Songtao : You may need adding PetscSynchronizedFlush() to the end. See petsc/src/dm/examples/tests/ex43.c Hong Hello, I am just a beginner. Now I get confused on how to correctly use Vec and DMDA. I set 1D DMDA global vector with natural values for a test. Then use VecView and DMDAVecGetArray to view the data, but the results are different. I don't know why. this is the code: PetscMPIInt rank; PetscErrorCode ierr; Vec global,local,natural; DM da; PetscReal *xnatural,*val; PetscInt i,start,end,x,xm; ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); ierr = DMDACreateNaturalVector(da,&natural);CHKERRQ(ierr); ierr = VecGetOwnershipRange(natural,&start,&end);CHKERRQ(ierr); ierr = VecGetArray(natural,&xnatural);CHKERRQ(ierr); for (i=start; i -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Apr 9 11:20:40 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 9 Apr 2018 11:20:40 -0500 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: , Message-ID: On Mon, 9 Apr 2018, Songtao Chu wrote: > static char help[] = "\n\n"; > #include > #include > > > int main(int argc,char **argv) > { > PetscMPIInt rank; > PetscErrorCode ierr; > Vec global,local,natural; > DM da; > PetscReal *val; > PetscInt i,x,xm; > > > ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > ierr = DMSetUp(da);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); Shouldn't this call be removed? Nothing of substance is printed here. > for (i = x; i < x + xm; ++i) { > val[i] = i; > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); Shouldn't this be on PETSC_COMM_WORLD? > for (i = x; i < x + xm; ++i) { > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); Again PETSC_COMM_WORLD? > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); > > ierr = PetscFinalize(); > return ierr; > } Attaching the fixed code. Satish ------- balay at asterix /home/balay/download-pine $ ./ex1 Vec Object: 1 MPI processes type: seq 0. 1. 2. 3. 4. Rank=0 0 1 2 3 4 balay at asterix /home/balay/download-pine $ mpiexec -n 2 ./ex1 Vec Object: 2 MPI processes type: mpi Process [0] 0. 1. 2. Process [1] 3. 4. Rank=0 0 1 2 Rank=1 3 4 balay at asterix /home/balay/download-pine $ mpiexec -n 3 ./ex1 Vec Object: 3 MPI processes type: mpi Process [0] 0. 1. Process [1] 2. 3. Process [2] 4. Rank=0 0 1 Rank=1 2 3 Rank=2 4 balay at asterix /home/balay/download-pine $ -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ex1.c URL: From knepley at gmail.com Mon Apr 9 11:42:11 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Apr 2018 12:42:11 -0400 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: Message-ID: Did you build EXACTLY the code I sent back? Thanks, Matt On Mon, Apr 9, 2018 at 12:13 PM, Songtao Chu wrote: > The two output results are different in my test. > If I use the way of assignment variables in current code. I got error > output of VecView as below: > Vec Object: 2 MPI processes > type: mpi > Vec Object: Vec_0x7fffd6895a90_0 2 MPI processes > type: mpi > Process [0] > 0. > 0. > 0. > Process [1] > 0. > 0. > > Or if I use the previous way (DMDANaturalToGlobal), I got error output > of PetscSynchronizedPrintf: > Rank=0 > 0 0 0 > Rank=1 > 0 0 > > Both are fine if I run without mpiexec. > > Besides, my development environment is CLion + WSL + GCC 7.3. > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, April 10, 2018 0:01 > *To:* Songtao Chu > *Cc:* Hong; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] How to access Vector data properly > > This works fine for me: > > knepley/feature-pc-patch $:/PETSc3/petsc/petsc-pylith/tmp$ > ../arch-master-debug/bin/mpiexec -n 2 ./tester > > Vec Object: 2 MPI processes > > type: mpi > > Process [0] > > 0. > > 1. > > 2. > > Process [1] > > 3. > > 4. > > Rank=0 > > 0 1 2 > > Rank=1 > > 3 4 > > What is the problem? > > Matt > > > On Mon, Apr 9, 2018 at 11:51 AM, Songtao Chu wrote: > > static char help[] = "\n\n"; > #include > #include > > > int main(int argc,char **argv) > { > PetscMPIInt rank; > PetscErrorCode ierr; > Vec global,local,natural; > DM da; > PetscReal *val; > PetscInt i,x,xm; > > > ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > ierr = DMSetUp(da);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > val[i] = i; > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); > > ierr = PetscFinalize(); > return ierr; > } > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 9, 2018 23:29 > *To:* Chu Songtao > *Cc:* Hong; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] How to access Vector data properly > > On Mon, Apr 9, 2018 at 11:19 AM, Chu Songtao wrote: > > Hong, > I added the function to the end, but the results didn't change. > Then I tested another thing, assigning global by Get(Restore)Array instead > of DMDANaturalToGlobal. And the results switched. VecView outputs 0 > and PetscSynchronizedPrintf outputs correct values. Strange. > > > Send the whole code. You have a logic error somewhere. > > Matt > > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > ierr = DMSetUp(da);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > val[i] = i; > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); > > $ mpiexec -n 2 ./test0 > Rank=0 > Rank=1 > Vec Object: 2 MPI processes > type: mpi > Vec Object: Vec_0x7fffd6629a90_0 2 MPI processes > type: mpi > Rank=1 > 3 4 Process [0] > 0. > 0. > 0. > Process [1] > 0. > 0. > Rank=0 > 0 1 2 > > > ------------------------------ > > Songtao : > You may need adding PetscSynchronizedFlush() to the end. > See petsc/src/dm/examples/tests/ex43.c > Hong > > Hello, > I am just a beginner. Now I get confused on how to correctly use Vec > and DMDA. > I set 1D DMDA global vector with natural values for a test. Then use > VecView and DMDAVecGetArray to view the data, but the results are > different. I don't know why. > > this is the code: > > PetscMPIInt rank; > PetscErrorCode ierr; > Vec global,local,natural; > DM da; > PetscReal *xnatural,*val; > PetscInt i,start,end,x,xm; > > > ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > ierr = DMSetUp(da);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > ierr = DMDACreateNaturalVector(da,&natural);CHKERRQ(ierr); > ierr = VecGetOwnershipRange(natural,&start,&end);CHKERRQ(ierr); > ierr = VecGetArray(natural,&xnatural);CHKERRQ(ierr); > for (i=start; i xnatural[i-start] = i; > } > ierr = VecRestoreArray(natural,&xnatural);CHKERRQ(ierr); > ierr = DMDANaturalToGlobalBegin(da,natural,INSERT_VALUES,global);CHKERRQ(ierr); > ierr = DMDANaturalToGlobalEnd(da,natural,INSERT_VALUES,global);CHKERRQ(ierr); > ierr = VecDestroy(&natural);CHKERRQ(ierr); > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > for (i = x; i < x + xm; ++i) { > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > and the results are: > > $ ./test0 > Vec Object: 1 MPI processes > type: seq > Vec Object: Vec_0x7ffff3cbc500_0 1 MPI processes > type: mpi > Process [0] > 0. > 1. > 2. > 3. > 4. > Rank=0 > 0 1 2 3 4 > > $ mpiexec -n 2 ./test0 > Vec Object: 2 MPI processes > type: mpi > Vec Object: Vec_0x7fffcf948a90_0 2 MPI processes > type: mpi > Process [0] > 0. > 1. > 2. > Process [1] > 3. > 4. > Rank=0 > 0 0 0 Rank=1 > 0 0 > > > the seq version is correct, but the mpi version is wrong. Every element in > val is 0. > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From st.chu at outlook.com Mon Apr 9 12:13:30 2018 From: st.chu at outlook.com (Songtao Chu) Date: Mon, 9 Apr 2018 17:13:30 +0000 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: , Message-ID: Yes, but the bug remains the same. Then I switched my development environment to Linux just now, this code ran without any bug. So it seems this is a bug of WSL? ________________________________ From: Matthew Knepley Sent: Tuesday, April 10, 2018 0:42 To: Songtao Chu Cc: Hong; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] How to access Vector data properly Did you build EXACTLY the code I sent back? Thanks, Matt On Mon, Apr 9, 2018 at 12:13 PM, Songtao Chu > wrote: The two output results are different in my test. If I use the way of assignment variables in current code. I got error output of VecView as below: Vec Object: 2 MPI processes type: mpi Vec Object: Vec_0x7fffd6895a90_0 2 MPI processes type: mpi Process [0] 0. 0. 0. Process [1] 0. 0. Or if I use the previous way (DMDANaturalToGlobal), I got error output of PetscSynchronizedPrintf: Rank=0 0 0 0 Rank=1 0 0 Both are fine if I run without mpiexec. Besides, my development environment is CLion + WSL + GCC 7.3. ________________________________ From: Matthew Knepley > Sent: Tuesday, April 10, 2018 0:01 To: Songtao Chu Cc: Hong; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] How to access Vector data properly This works fine for me: knepley/feature-pc-patch $:/PETSc3/petsc/petsc-pylith/tmp$ ../arch-master-debug/bin/mpiexec -n 2 ./tester Vec Object: 2 MPI processes type: mpi Process [0] 0. 1. 2. Process [1] 3. 4. Rank=0 0 1 2 Rank=1 3 4 What is the problem? Matt On Mon, Apr 9, 2018 at 11:51 AM, Songtao Chu > wrote: static char help[] = "\n\n"; #include #include int main(int argc,char **argv) { PetscMPIInt rank; PetscErrorCode ierr; Vec global,local,natural; DM da; PetscReal *val; PetscInt i,x,xm; ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { val[i] = i; } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); VecView(global,PETSC_VIEWER_STDOUT_WORLD); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); ierr = PetscFinalize(); return ierr; } ________________________________ From: Matthew Knepley > Sent: Monday, April 9, 2018 23:29 To: Chu Songtao Cc: Hong; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] How to access Vector data properly On Mon, Apr 9, 2018 at 11:19 AM, Chu Songtao > wrote: Hong, I added the function to the end, but the results didn't change. Then I tested another thing, assigning global by Get(Restore)Array instead of DMDANaturalToGlobal. And the results switched. VecView outputs 0 and PetscSynchronizedPrintf outputs correct values. Strange. Send the whole code. You have a logic error somewhere. Matt ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { val[i] = i; } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); VecView(global,PETSC_VIEWER_STDOUT_WORLD); ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); for (i = x; i < x + xm; ++i) { ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); } ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); $ mpiexec -n 2 ./test0 Rank=0 Rank=1 Vec Object: 2 MPI processes type: mpi Vec Object: Vec_0x7fffd6629a90_0 2 MPI processes type: mpi Rank=1 3 4 Process [0] 0. 0. 0. Process [1] 0. 0. Rank=0 0 1 2 ________________________________ Songtao : You may need adding PetscSynchronizedFlush() to the end. See petsc/src/dm/examples/tests/ex43.c Hong Hello, I am just a beginner. Now I get confused on how to correctly use Vec and DMDA. I set 1D DMDA global vector with natural values for a test. Then use VecView and DMDAVecGetArray to view the data, but the results are different. I don't know why. this is the code: PetscMPIInt rank; PetscErrorCode ierr; Vec global,local,natural; DM da; PetscReal *xnatural,*val; PetscInt i,start,end,x,xm; ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); ierr = DMDACreateNaturalVector(da,&natural);CHKERRQ(ierr); ierr = VecGetOwnershipRange(natural,&start,&end);CHKERRQ(ierr); ierr = VecGetArray(natural,&xnatural);CHKERRQ(ierr); for (i=start; i -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From st.chu at outlook.com Mon Apr 9 12:15:41 2018 From: st.chu at outlook.com (Songtao Chu) Date: Mon, 9 Apr 2018 17:15:41 +0000 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: , , Message-ID: Thank you for your correction. I rebuilt the code under Linux, the bug disappeared. It seems this is a bug of WSL. ________________________________ From: Satish Balay Sent: Tuesday, April 10, 2018 0:20 To: Songtao Chu Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] How to access Vector data properly On Mon, 9 Apr 2018, Songtao Chu wrote: > static char help[] = "\n\n"; > #include > #include > > > int main(int argc,char **argv) > { > PetscMPIInt rank; > PetscErrorCode ierr; > Vec global,local,natural; > DM da; > PetscReal *val; > PetscInt i,x,xm; > > > ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > ierr = DMSetUp(da);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); Shouldn't this call be removed? Nothing of substance is printed here. > for (i = x; i < x + xm; ++i) { > val[i] = i; > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); Shouldn't this be on PETSC_COMM_WORLD? > for (i = x; i < x + xm; ++i) { > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); Again PETSC_COMM_WORLD? > } > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); > > ierr = PetscFinalize(); > return ierr; > } Attaching the fixed code. Satish ------- balay at asterix /home/balay/download-pine $ ./ex1 Vec Object: 1 MPI processes type: seq 0. 1. 2. 3. 4. Rank=0 0 1 2 3 4 balay at asterix /home/balay/download-pine $ mpiexec -n 2 ./ex1 Vec Object: 2 MPI processes type: mpi Process [0] 0. 1. 2. Process [1] 3. 4. Rank=0 0 1 2 Rank=1 3 4 balay at asterix /home/balay/download-pine $ mpiexec -n 3 ./ex1 Vec Object: 3 MPI processes type: mpi Process [0] 0. 1. Process [1] 2. 3. Process [2] 4. Rank=0 0 1 Rank=1 2 3 Rank=2 4 balay at asterix /home/balay/download-pine $ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 9 12:19:17 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Apr 2018 13:19:17 -0400 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: Message-ID: On Mon, Apr 9, 2018 at 1:15 PM, Songtao Chu wrote: > Thank you for your correction. > I rebuilt the code under Linux, the bug disappeared. It seems this is a > bug of WSL. > Thanks, Matt > ------------------------------ > *From:* Satish Balay > *Sent:* Tuesday, April 10, 2018 0:20 > *To:* Songtao Chu > *Cc:* Matthew Knepley; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] How to access Vector data properly > > On Mon, 9 Apr 2018, Songtao Chu wrote: > > > static char help[] = "\n\n"; > > #include > > #include > > > > > > int main(int argc,char **argv) > > { > > PetscMPIInt rank; > > PetscErrorCode ierr; > > Vec global,local,natural; > > DM da; > > PetscReal *val; > > PetscInt i,x,xm; > > > > > > ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return > ierr; > > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > > > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,& > da);CHKERRQ(ierr); > > ierr = DMSetUp(da);CHKERRQ(ierr); > > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", > rank);CHKERRQ(ierr); > > Shouldn't this call be removed? Nothing of substance is printed here. > > > for (i = x; i < x + xm; ++i) { > > val[i] = i; > > } > > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", > rank);CHKERRQ(ierr); > > Shouldn't this be on PETSC_COMM_WORLD? > > > for (i = x; i < x + xm; ++i) { > > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", > val[i]);CHKERRQ(ierr); > > Again PETSC_COMM_WORLD? > > > > } > > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); > > > > ierr = PetscFinalize(); > > return ierr; > > } > > Attaching the fixed code. > > Satish > ------- > balay at asterix /home/balay/download-pine > $ ./ex1 > Vec Object: 1 MPI processes > type: seq > 0. > 1. > 2. > 3. > 4. > Rank=0 > 0 1 2 3 4 > balay at asterix /home/balay/download-pine > $ mpiexec -n 2 ./ex1 > Vec Object: 2 MPI processes > type: mpi > Process [0] > 0. > 1. > 2. > Process [1] > 3. > 4. > Rank=0 > 0 1 2 > Rank=1 > 3 4 > balay at asterix /home/balay/download-pine > $ mpiexec -n 3 ./ex1 > Vec Object: 3 MPI processes > type: mpi > Process [0] > 0. > 1. > Process [1] > 2. > 3. > Process [2] > 4. > Rank=0 > 0 1 > Rank=1 > 2 3 > Rank=2 > 4 > balay at asterix /home/balay/download-pine > $ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Apr 9 13:25:30 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 9 Apr 2018 13:25:30 -0500 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: , , Message-ID: Does my corrected code also print wrong info on WSL? Satish On Mon, 9 Apr 2018, Songtao Chu wrote: > Thank you for your correction. > I rebuilt the code under Linux, the bug disappeared. It seems this is a bug of WSL. > > ________________________________ > From: Satish Balay > Sent: Tuesday, April 10, 2018 0:20 > To: Songtao Chu > Cc: Matthew Knepley; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] How to access Vector data properly > > On Mon, 9 Apr 2018, Songtao Chu wrote: > > > static char help[] = "\n\n"; > > #include > > #include > > > > > > int main(int argc,char **argv) > > { > > PetscMPIInt rank; > > PetscErrorCode ierr; > > Vec global,local,natural; > > DM da; > > PetscReal *val; > > PetscInt i,x,xm; > > > > > > ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; > > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > > > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > > ierr = DMSetUp(da);CHKERRQ(ierr); > > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > > Shouldn't this call be removed? Nothing of substance is printed here. > > > for (i = x; i < x + xm; ++i) { > > val[i] = i; > > } > > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > > Shouldn't this be on PETSC_COMM_WORLD? > > > for (i = x; i < x + xm; ++i) { > > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); > > Again PETSC_COMM_WORLD? > > > > } > > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); > > > > ierr = PetscFinalize(); > > return ierr; > > } > > Attaching the fixed code. > > Satish > ------- > balay at asterix /home/balay/download-pine > $ ./ex1 > Vec Object: 1 MPI processes > type: seq > 0. > 1. > 2. > 3. > 4. > Rank=0 > 0 1 2 3 4 > balay at asterix /home/balay/download-pine > $ mpiexec -n 2 ./ex1 > Vec Object: 2 MPI processes > type: mpi > Process [0] > 0. > 1. > 2. > Process [1] > 3. > 4. > Rank=0 > 0 1 2 > Rank=1 > 3 4 > balay at asterix /home/balay/download-pine > $ mpiexec -n 3 ./ex1 > Vec Object: 3 MPI processes > type: mpi > Process [0] > 0. > 1. > Process [1] > 2. > 3. > Process [2] > 4. > Rank=0 > 0 1 > Rank=1 > 2 3 > Rank=2 > 4 > balay at asterix /home/balay/download-pine > $ > From st.chu at outlook.com Mon Apr 9 13:32:30 2018 From: st.chu at outlook.com (Songtao Chu) Date: Mon, 9 Apr 2018 18:32:30 +0000 Subject: [petsc-users] How to access Vector data properly In-Reply-To: References: , , , Message-ID: Yes, the output of VecView is still 0 on WSL. ________________________________ From: Satish Balay Sent: Tuesday, April 10, 2018 2:25 To: Songtao Chu Cc: petsc-users Subject: Re: [petsc-users] How to access Vector data properly Does my corrected code also print wrong info on WSL? Satish On Mon, 9 Apr 2018, Songtao Chu wrote: > Thank you for your correction. > I rebuilt the code under Linux, the bug disappeared. It seems this is a bug of WSL. > > ________________________________ > From: Satish Balay > Sent: Tuesday, April 10, 2018 0:20 > To: Songtao Chu > Cc: Matthew Knepley; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] How to access Vector data properly > > On Mon, 9 Apr 2018, Songtao Chu wrote: > > > static char help[] = "\n\n"; > > #include > > #include > > > > > > int main(int argc,char **argv) > > { > > PetscMPIInt rank; > > PetscErrorCode ierr; > > Vec global,local,natural; > > DM da; > > PetscReal *val; > > PetscInt i,x,xm; > > > > > > ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; > > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > > > > ierr = DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,5,1,1,NULL,&da);CHKERRQ(ierr); > > ierr = DMSetUp(da);CHKERRQ(ierr); > > ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr); > > > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > > Shouldn't this call be removed? Nothing of substance is printed here. > > > for (i = x; i < x + xm; ++i) { > > val[i] = i; > > } > > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > > > VecView(global,PETSC_VIEWER_STDOUT_WORLD); > > > > ierr = DMDAGetCorners(da,&x,NULL,NULL,&xm,NULL,NULL);CHKERRQ(ierr); > > ierr = DMDAVecGetArray(da, global, &val);CHKERRQ(ierr); > > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "Rank=%d\n", rank);CHKERRQ(ierr); > > Shouldn't this be on PETSC_COMM_WORLD? > > > for (i = x; i < x + xm; ++i) { > > ierr = PetscSynchronizedPrintf(PETSC_COMM_SELF, "%4.f ", val[i]);CHKERRQ(ierr); > > Again PETSC_COMM_WORLD? > > > > } > > ierr = DMDAVecRestoreArray(da, global, &val);CHKERRQ(ierr); > > PetscSynchronizedFlush(PETSC_COMM_SELF, PETSC_STDOUT); > > > > ierr = PetscFinalize(); > > return ierr; > > } > > Attaching the fixed code. > > Satish > ------- > balay at asterix /home/balay/download-pine > $ ./ex1 > Vec Object: 1 MPI processes > type: seq > 0. > 1. > 2. > 3. > 4. > Rank=0 > 0 1 2 3 4 > balay at asterix /home/balay/download-pine > $ mpiexec -n 2 ./ex1 > Vec Object: 2 MPI processes > type: mpi > Process [0] > 0. > 1. > 2. > Process [1] > 3. > 4. > Rank=0 > 0 1 2 > Rank=1 > 3 4 > balay at asterix /home/balay/download-pine > $ mpiexec -n 3 ./ex1 > Vec Object: 3 MPI processes > type: mpi > Process [0] > 0. > 1. > Process [1] > 2. > 3. > Process [2] > 4. > Rank=0 > 0 1 > Rank=1 > 2 3 > Rank=2 > 4 > balay at asterix /home/balay/download-pine > $ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernardomartinsrocha at gmail.com Mon Apr 9 13:51:08 2018 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Mon, 9 Apr 2018 15:51:08 -0300 Subject: [petsc-users] DMPlexGetCone in Fortran In-Reply-To: References: Message-ID: OK, just found my mistake. I was using other include files such as "petscksp.h" and etc that I needed from previous codes. I had to include ONLY "#include " to fix the problem. After that the GetCone, GetTransitiveClosure and other routines returning arrays stop to result in Segmentation Fault. It would be nice to give some warning when compiling to Fortran 90 and including the wrong header files. Anyway, thanks. Best regards, Bernardo On Sat, Apr 7, 2018 at 4:13 PM, Bernardo Rocha < bernardomartinsrocha at gmail.com> wrote: > OK, here is the entire code. > I created a simple mesh to test DMPlex. > As you can see, the GetCone is not working. > As I said, I followed one of the examples for DMPlex using Fortran, I'm > not an exper in F90. > > Best regards, > Bernardo M. Rocha > > > On Fri, Apr 6, 2018 at 11:48 PM, Matthew Knepley > wrote: > >> If you send the whole code, we can try it here. I am assuming that the >> examples run for you. >> >> Matt >> >> On Fri, Apr 6, 2018 at 5:47 PM, Bernardo Rocha < >> bernardomartinsrocha at gmail.com> wrote: >> >>> Hello, >>> >>> I can't get a a simple code to work for DMPlexGetCone. >>> I have tried different ways of handling the array/pointer which is the >>> third argument of DMPlexGetCone, >>> but neither of them worked for me. >>> >>> The last version that I got compiling is this one, which is very much >>> based on the >>> following example: .../petsc-3.7.6/src/dm/impls/p >>> lex/examples/tutorials/ex1f90.F >>> >>> PetscInt, target, dimension(6) :: icone >>> PetscInt, pointer :: pIcone(:) >>> >>> call DMPlexCreateFromCellList(PETSC_COMM_WORLD, >>> & ndim,numel,numnp,nen,PETSC_TRUE, >>> & ielement,ndim,xnodes,dm,ierr) >>> ... >>> call DMPlexGetCone(dm,0,pIcone,ierr) >>> write(*,*) icone(1),icone(2),icone(3),icone(4),icone(5),icone(6) >>> call DMPlexRestoreCone(dm,0,pIcone,ierr) >>> >>> I always get a segmentation fault. >>> >>> I've created a simple cuboid mesh with 2x2x2 hexaedral elements. >>> Many other routines are working and give reasonable results. >>> >>> Output of DMView >>> >>> DM Object: 1 MPI processes >>> type: plex >>> DM_0x84000000_0 in 3 dimensions: >>> 0-cells: 27 >>> 1-cells: 66 >>> 2-cells: 44 >>> 3-cells: 8 >>> Labels: >>> depth: 4 strata of sizes (27, 66, 44, 8) >>> >>> >>> >>> >>> On Thu, Apr 5, 2018 at 10:57 PM, Bernardo Rocha < >>> bernardomartinsrocha at gmail.com> wrote: >>> >>>> OK, thanks a lot. >>>> I had to rename the files and add -ffixed-form as argument to the >>>> compiler. >>>> Best regards, >>>> Bernardo >>>> >>>> On Thu, Apr 5, 2018 at 8:21 PM, Matthew Knepley >>>> wrote: >>>> >>>>> On Thu, Apr 5, 2018 at 7:16 PM, Bernardo Rocha < >>>>> bernardomartinsrocha at gmail.com> wrote: >>>>> >>>>>> Hello everyone, >>>>>> >>>>>> I've been trying to use DMPlex in a Fortran 77 application >>>>>> (unfortunately this is legacy code and I can't move to F90). >>>>>> >>>>>> Is there a way to use DMPlexGetCone on it? >>>>>> Although the documentation online says it is only available for F90, >>>>>> I'm wondering if there is a trick like the ones in the users manual >>>>>> for routines >>>>>> that return an array. I tried, but I'm not sure it it is right. >>>>>> >>>>> >>>>> We have standardized our array handling with F90, and I don't think we >>>>> will go back >>>>> to supporting new things for F77-style arrays. >>>>> >>>>> That being said, I think all you have to do is compile that source >>>>> with an F90 compiler. You >>>>> should not have to change anything. If you want the F90 array to look >>>>> like an F77 array, >>>>> just pass it as a function argument (I think this works). >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Best regards, >>>>>> Bernardo M. Rocha >>>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Apr 9 14:00:38 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 9 Apr 2018 14:00:38 -0500 Subject: [petsc-users] DMPlexGetCone in Fortran In-Reply-To: References: Message-ID: I see you are using petsc-3.7. PETSc 3.8 has replaced [most of] the include file usage with modules - so there is better typechecking. Check 'Fortran' section of http://www.mcs.anl.gov/petsc/documentation/changes/38.html So suggest upgrading to using latest petsc release [currently 3.9] Unfortunately fortran doesn't enforce prototype requirements - so some checks [that C/C++ does wrt includes] is not possible with fortran. Satish On Mon, 9 Apr 2018, Bernardo Rocha wrote: > OK, just found my mistake. > > I was using other include files such as "petscksp.h" and etc that I needed > from previous codes. > > I had to include ONLY > "#include " > to fix the problem. > > After that the GetCone, GetTransitiveClosure and other routines returning > arrays stop > to result in Segmentation Fault. > > It would be nice to give some warning when compiling to Fortran 90 and > including > the wrong header files. > > Anyway, thanks. > Best regards, > Bernardo > > > > On Sat, Apr 7, 2018 at 4:13 PM, Bernardo Rocha < > bernardomartinsrocha at gmail.com> wrote: > > > OK, here is the entire code. > > I created a simple mesh to test DMPlex. > > As you can see, the GetCone is not working. > > As I said, I followed one of the examples for DMPlex using Fortran, I'm > > not an exper in F90. > > > > Best regards, > > Bernardo M. Rocha > > > > > > On Fri, Apr 6, 2018 at 11:48 PM, Matthew Knepley > > wrote: > > > >> If you send the whole code, we can try it here. I am assuming that the > >> examples run for you. > >> > >> Matt > >> > >> On Fri, Apr 6, 2018 at 5:47 PM, Bernardo Rocha < > >> bernardomartinsrocha at gmail.com> wrote: > >> > >>> Hello, > >>> > >>> I can't get a a simple code to work for DMPlexGetCone. > >>> I have tried different ways of handling the array/pointer which is the > >>> third argument of DMPlexGetCone, > >>> but neither of them worked for me. > >>> > >>> The last version that I got compiling is this one, which is very much > >>> based on the > >>> following example: .../petsc-3.7.6/src/dm/impls/p > >>> lex/examples/tutorials/ex1f90.F > >>> > >>> PetscInt, target, dimension(6) :: icone > >>> PetscInt, pointer :: pIcone(:) > >>> > >>> call DMPlexCreateFromCellList(PETSC_COMM_WORLD, > >>> & ndim,numel,numnp,nen,PETSC_TRUE, > >>> & ielement,ndim,xnodes,dm,ierr) > >>> ... > >>> call DMPlexGetCone(dm,0,pIcone,ierr) > >>> write(*,*) icone(1),icone(2),icone(3),icone(4),icone(5),icone(6) > >>> call DMPlexRestoreCone(dm,0,pIcone,ierr) > >>> > >>> I always get a segmentation fault. > >>> > >>> I've created a simple cuboid mesh with 2x2x2 hexaedral elements. > >>> Many other routines are working and give reasonable results. > >>> > >>> Output of DMView > >>> > >>> DM Object: 1 MPI processes > >>> type: plex > >>> DM_0x84000000_0 in 3 dimensions: > >>> 0-cells: 27 > >>> 1-cells: 66 > >>> 2-cells: 44 > >>> 3-cells: 8 > >>> Labels: > >>> depth: 4 strata of sizes (27, 66, 44, 8) > >>> > >>> > >>> > >>> > >>> On Thu, Apr 5, 2018 at 10:57 PM, Bernardo Rocha < > >>> bernardomartinsrocha at gmail.com> wrote: > >>> > >>>> OK, thanks a lot. > >>>> I had to rename the files and add -ffixed-form as argument to the > >>>> compiler. > >>>> Best regards, > >>>> Bernardo > >>>> > >>>> On Thu, Apr 5, 2018 at 8:21 PM, Matthew Knepley > >>>> wrote: > >>>> > >>>>> On Thu, Apr 5, 2018 at 7:16 PM, Bernardo Rocha < > >>>>> bernardomartinsrocha at gmail.com> wrote: > >>>>> > >>>>>> Hello everyone, > >>>>>> > >>>>>> I've been trying to use DMPlex in a Fortran 77 application > >>>>>> (unfortunately this is legacy code and I can't move to F90). > >>>>>> > >>>>>> Is there a way to use DMPlexGetCone on it? > >>>>>> Although the documentation online says it is only available for F90, > >>>>>> I'm wondering if there is a trick like the ones in the users manual > >>>>>> for routines > >>>>>> that return an array. I tried, but I'm not sure it it is right. > >>>>>> > >>>>> > >>>>> We have standardized our array handling with F90, and I don't think we > >>>>> will go back > >>>>> to supporting new things for F77-style arrays. > >>>>> > >>>>> That being said, I think all you have to do is compile that source > >>>>> with an F90 compiler. You > >>>>> should not have to change anything. If you want the F90 array to look > >>>>> like an F77 array, > >>>>> just pass it as a function argument (I think this works). > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Matt > >>>>> > >>>>> > >>>>>> Best regards, > >>>>>> Bernardo M. Rocha > >>>>>> > >>>>> -- > >>>>> What most experimenters take for granted before they begin their > >>>>> experiments is infinitely more interesting than any results to which their > >>>>> experiments lead. > >>>>> -- Norbert Wiener > >>>>> > >>>>> https://www.cse.buffalo.edu/~knepley/ > >>>>> > >>>>> > >>>> > >>>> > >>> > >> > >> > >> -- > >> What most experimenters take for granted before they begin their > >> experiments is infinitely more interesting than any results to which their > >> experiments lead. > >> -- Norbert Wiener > >> > >> https://www.cse.buffalo.edu/~knepley/ > >> > > > > > From bsmith at mcs.anl.gov Mon Apr 9 16:22:23 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 9 Apr 2018 21:22:23 +0000 Subject: [petsc-users] PETSc 3.9 release Message-ID: <45E3843D-DFCE-4F3F-97B5-BE4BABACADB1@mcs.anl.gov> PETSc users We are pleased to announce the release of PETSc version 3.9 at http://www.mcs.anl.gov/petsc The major changes and updates can be found at http://www.mcs.anl.gov/petsc/documentation/changes/39.html We recommend upgrading to PETSc 3.9 soon (the sooner you switch the more we can help you with the switch). As always, please report problems to petsc-maint at mcs.anl.gov and ask questions at petsc-users at mcs.anl.gov This release includes contributions from Adrian Croucher Alejandro Lamas Davi?a Alp Dener Barry Smith Bernhard M. Wiedemann Blaise Bourdin Daniel Kokron Dave May David Wobrock eaulisa Fande Kong Hong Zhang Hong Zhang Jakub Kruzik Jed Brown Jose E. Roman Junchao Zhang Karl Rupp Lawrence Mitchell Lisandro Dalcin Mari Pecha Mark Adams Mark Lohry Matthew G. Knepley Nicolas Barral Oana Marin Patrick Sanan Pieter Ghysels Richard Tran Mills Satish Balay Scott Kruger Shri Abhyankar Siegfried Cools Stefano Zampini Thomas Mijieux Toby Isaac Todd Munson Tristan Konolige V?clav Hapla and bug reports/patches/proposed improvements received from Adrian Croucher Albert Cowie "Alberto F. Mart?n" Alp Dener Antonio Trande apostol Barry Smith "Charles A.(Sandy) Mader" Daniele Tartarini Danyang Su Drew Parsons Ed Bueler Eric Chamberland Eugenio Aulisa Evan Um Fabrice JURADO Fadi El Haddad Fande Kong Febrian Setianto Florian Lindner Franck Houssen Francois Pechereau Garth N. Wells GONG Ding Greg Meyer Henrik Buesing Hong Zhang Hong Zhang ilya Jaroslaw Piwonski Jean Philippe Fran?ois Jed Brown Jin Chen J?rgen Dokken Jose E. Roman Karl Rupp Keith Lindsay "Klaij, Christiaan" Lawrence Mitchell Lisandro Dalcin Manav Bhatia Marco Schauer Marius Buerkle Mark Adams Martin Diehl Matthew Knepley Matthieu Vitse Michael Legu?be "Miller, Laniece E." Oana Marin Pablo Brubeck Patrick Farrell Patrick Sanan Peter Meisrimel Philip Eisenlohr Pierre Jolivet Praveen C Rafael Monteiro da Silva Randall Mackie Randy Michael Churchill Richard Tran Mills Roland Richter Samuel Lanthaler Sanjay Govindjee Satish Balay Serge Van Criekingen Stefano Zampini Stephan Kramer Syam Vangara Tim Steinhoff Tobias Goerler Todd Munson Tzanio Kolev Victor Eijkhout Vinh Pham-Xuan Xiangdong Xiaoye S. Li Yiting Wang As always, thanks for your support, Barry From s_g at berkeley.edu Mon Apr 9 16:51:55 2018 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Mon, 9 Apr 2018 23:51:55 +0200 Subject: [petsc-users] VecGetArray vs. VecGetArrayReadF90 Message-ID: <0054474c-81b4-d7b3-c74a-d1ee60bf0ca6@berkeley.edu> I am wondering about a change in behavior that I noticed with v3.8.3 (versus 3.7.6). I have a Vec object which in 3.7.6 was stored in a common block. For 3.8.3 we have moved it to a module since gfortran was unhappy about having it in a common block. In our code, we extract values from the vector using VecGetArray( ).? With ifort the behavior is the same (and correct) for both versions of our code.? With gfortran, we are seeing incorrect values when using VecGetArray( ) when the Vec is contained in a module as needed for 3.8.3.? If we switch over to VecGetArrayReadF90( ) gfortran once again produces correct results. Is there a good reason to expect this behavior?? I'm am trying to understand the underlying concepts that are a play here. -sanjay From bsmith at mcs.anl.gov Mon Apr 9 16:57:02 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 9 Apr 2018 21:57:02 +0000 Subject: [petsc-users] VecGetArray vs. VecGetArrayReadF90 In-Reply-To: <0054474c-81b4-d7b3-c74a-d1ee60bf0ca6@berkeley.edu> References: <0054474c-81b4-d7b3-c74a-d1ee60bf0ca6@berkeley.edu> Message-ID: <7700CBBA-8B24-47C5-A176-DE763702A8E7@anl.gov> > On Apr 9, 2018, at 4:51 PM, Sanjay Govindjee wrote: > > I am wondering about a change in behavior that I noticed with v3.8.3 (versus 3.7.6). > > I have a Vec object which in 3.7.6 was stored in a common block. For 3.8.3 we have moved it to a module since gfortran was unhappy about having it in a common block. > > In our code, we extract values from the vector using VecGetArray( ). With ifort the behavior is the same (and correct) for both versions of our code. With gfortran, we are seeing incorrect values when using VecGetArray( ) when the Vec is contained in a module as needed for 3.8.3. If we switch over to VecGetArrayReadF90( ) gfortran once again produces correct results. > > Is there a good reason to expect this behavior? I have no explanation for why this should happen. > I'm am trying to understand the underlying concepts that are a play here. VecGetArray() "cheats" and simply returns the index that one must shift that array address to access the values in the array. The F90 version uses F90 features to access the values. Moving forward we always recommend using the F90 versions, they are simpler to use, the non-F90 versions are hacks that one must use if one does not have F90 (but everyone has F90 now). Barry The only reason we don't remove the non-F90 is for legacy codes where the users haven't updated yet. > > -sanjay > > From mvalera-w at sdsu.edu Mon Apr 9 17:12:48 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Mon, 9 Apr 2018 15:12:48 -0700 Subject: [petsc-users] Cuda libraries and DMDA Message-ID: Hello guys, I've made advances in my CUDA acceleration project, as you remember i have a CFD model in need of better execution times. So far i have been able to solve the pressure system in the GPU and the rest in serial, using PETSc only for this pressure solve, the library i got to work was ViennaCL. First question, do i still have to switch installations to use either CUDA library? this was a suggestion before, so in order to use CUSP instead of ViennaCL, for example, i currently have to change installations, is this still the case? Now, i started working in a fully parallelized version of the model, which uses the DMs and DMDAs to distribute the arrays, if i try the same flags as before i get an error saying "Currently only handles ViennaCL matrices" when trying to solve for pressure, i get this is a feature still not implemented? what options do i have to solve pressure, or assign a DMDA array update to be done specifically in a GPU device? I was thinking of using the VecScatterCreateToZero for a regular vector, but then i would have to create a vector and copy the DMDAVec into it, is this accomplished with DMDAVecGetArrayReadF90 and then just copy? do you think this will generate too much overhead? Thanks so much for your input, Manuel -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 9 18:09:00 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Apr 2018 19:09:00 -0400 Subject: [petsc-users] Cuda libraries and DMDA In-Reply-To: References: Message-ID: On Mon, Apr 9, 2018 at 6:12 PM, Manuel Valera wrote: > Hello guys, > > I've made advances in my CUDA acceleration project, as you remember i have > a CFD model in need of better execution times. > > So far i have been able to solve the pressure system in the GPU and the > rest in serial, using PETSc only for this pressure solve, the library i got > to work was ViennaCL. First question, do i still have to switch > installations to use either CUDA library? this was a suggestion before, so > in order to use CUSP instead of ViennaCL, for example, i currently have to > change installations, is this still the case? > I am not sure what that means exactly. However, you can build a PETSc with CUDA and ViennaCL support. The type of Vec/Mat is selected at runtime. > Now, i started working in a fully parallelized version of the model, which > uses the DMs and DMDAs to distribute the arrays, if i try the same flags as > before i get an error saying "Currently only handles ViennaCL matrices" > when trying to solve for pressure, i get this is a feature still not > implemented? what options do i have to solve pressure, or assign a DMDA > array update to be done specifically in a GPU device? > If we can't see the error, we are just guessing. Please send the entire error message. Note, we only do linear algebra on the GPU, so none of the FormFunction/FormJacobian stuff for DMDA would be on the GPU. > I was thinking of using the VecScatterCreateToZero for a regular vector, > Why do you want a serial vector? > but then i would have to create a vector and copy the DMDAVec into it, > I do not understand what it means to copy the DM into the Vec. Thanks, Matt > is this accomplished with DMDAVecGetArrayReadF90 and then just copy? do > you think this will generate too much overhead? > > Thanks so much for your input, > > Manuel > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera-w at sdsu.edu Mon Apr 9 18:27:53 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Mon, 9 Apr 2018 16:27:53 -0700 Subject: [petsc-users] Cuda libraries and DMDA In-Reply-To: References: Message-ID: On Mon, Apr 9, 2018 at 4:09 PM, Matthew Knepley wrote: > On Mon, Apr 9, 2018 at 6:12 PM, Manuel Valera wrote: > >> Hello guys, >> >> I've made advances in my CUDA acceleration project, as you remember i >> have a CFD model in need of better execution times. >> >> So far i have been able to solve the pressure system in the GPU and the >> rest in serial, using PETSc only for this pressure solve, the library i got >> to work was ViennaCL. First question, do i still have to switch >> installations to use either CUDA library? this was a suggestion before, so >> in order to use CUSP instead of ViennaCL, for example, i currently have to >> change installations, is this still the case? >> > > I am not sure what that means exactly. However, you can build a PETSc with > CUDA and ViennaCL support. The type of Vec/Mat is selected at runtime. > Karl Rupp wrote in a previous email: * * Right now only one of {native CUDA, CUSP, ViennaCL} can be activated at configure time. This will be fixed later this month.* I was asking if this was already solved in 3.9, > > >> Now, i started working in a fully parallelized version of the model, >> which uses the DMs and DMDAs to distribute the arrays, if i try the same >> flags as before i get an error saying "Currently only handles ViennaCL >> matrices" when trying to solve for pressure, i get this is a feature still >> not implemented? what options do i have to solve pressure, or assign a DMDA >> array update to be done specifically in a GPU device? >> > > If we can't see the error, we are just guessing. Please send the entire > error message. > Got it, I will paste the error at the end of this email > > Note, we only do linear algebra on the GPU, so none of the > FormFunction/FormJacobian stuff for DMDA would be on the GPU. > Yes, we only use it for linear algebra, e.g. solving a linear system and updating an array with a problematic algorithm. > > >> I was thinking of using the VecScatterCreateToZero for a regular vector, >> > > Why do you want a serial vector? > Because it looks live ViennaCL doesn't handle arrays created with DMDAVec, it was just an idea > but then i would have to create a vector and copy the DMDAVec into it, >> > > I do not understand what it means to copy the DM into the Vec. > I meant copying a DMDAVec into a Vec object, the first is created with a DMDA object for it's mapping across processors, > > Thanks, > > Matt > > >> is this accomplished with DMDAVecGetArrayReadF90 and then just copy? do >> you think this will generate too much overhead? >> >> Thanks so much for your input, >> >> Manuel >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > The error happens when trying to use KSPSolve() for a vector made with DMDAVec routines, the matrix is created without any DMDA routines Error: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b GIT Date: 2018-04-02 11:59:41 +0200 [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 16:24:26 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=no --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #2 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b GIT Date: 2018-04-02 11:59:41 +0200 [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 16:24:26 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=no --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/interface/precon.c Finished setting up matrix objects Exiting PrepareNetCDF [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b GIT Date: 2018-04-02 11:59:41 +0200 [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 16:24:26 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=no --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp [0]PETSC ERROR: #6 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 9 18:45:09 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Apr 2018 19:45:09 -0400 Subject: [petsc-users] Cuda libraries and DMDA In-Reply-To: References: Message-ID: On Mon, Apr 9, 2018 at 7:27 PM, Manuel Valera wrote: > On Mon, Apr 9, 2018 at 4:09 PM, Matthew Knepley wrote: > >> On Mon, Apr 9, 2018 at 6:12 PM, Manuel Valera wrote: >> >>> Hello guys, >>> >>> I've made advances in my CUDA acceleration project, as you remember i >>> have a CFD model in need of better execution times. >>> >>> So far i have been able to solve the pressure system in the GPU and the >>> rest in serial, using PETSc only for this pressure solve, the library i got >>> to work was ViennaCL. First question, do i still have to switch >>> installations to use either CUDA library? this was a suggestion before, so >>> in order to use CUSP instead of ViennaCL, for example, i currently have to >>> change installations, is this still the case? >>> >> >> I am not sure what that means exactly. However, you can build a PETSc >> with CUDA and ViennaCL support. The type of Vec/Mat is selected at runtime. >> > > Karl Rupp wrote in a previous email: > > * * Right now only one of {native CUDA, CUSP, ViennaCL} can be activated > at configure time. This will be fixed later this month.* > > I was asking if this was already solved in 3.9, > Karl knows better than I do. I thought that was fixed, but maybe not in this release. > > >> >> >>> Now, i started working in a fully parallelized version of the model, >>> which uses the DMs and DMDAs to distribute the arrays, if i try the same >>> flags as before i get an error saying "Currently only handles ViennaCL >>> matrices" when trying to solve for pressure, i get this is a feature still >>> not implemented? what options do i have to solve pressure, or assign a DMDA >>> array update to be done specifically in a GPU device? >>> >> >> If we can't see the error, we are just guessing. Please send the entire >> error message. >> > > Got it, I will paste the error at the end of this email > It is asking for a ViennaCL matrix. You must tell the DM to create one: -dm_mat_type aijviennacl > > >> >> Note, we only do linear algebra on the GPU, so none of the >> FormFunction/FormJacobian stuff for DMDA would be on the GPU. >> > > Yes, we only use it for linear algebra, e.g. solving a linear system and > updating an array with a problematic algorithm. > > > >> >> >>> I was thinking of using the VecScatterCreateToZero for a regular vector, >>> >> >> Why do you want a serial vector? >> > > Because it looks live ViennaCL doesn't handle arrays created with DMDAVec, > it was just an idea > No, it just needs the right type. > > >> but then i would have to create a vector and copy the DMDAVec into it, >>> >> >> I do not understand what it means to copy the DM into the Vec. >> > > I meant copying a DMDAVec into a Vec object, the first is created with a > DMDA object for it's mapping across processors, > There is no such thing as a DMDAVec. Everything is just a Vec. Thanks, Matt > >> Thanks, >> >> Matt >> >> >>> is this accomplished with DMDAVecGetArrayReadF90 and then just copy? do >>> you think this will generate too much overhead? >>> >>> Thanks so much for your input, >>> >>> Manuel >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > The error happens when trying to use KSPSolve() for a vector made with > DMDAVec routines, the matrix is created without any DMDA routines > > Error: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b GIT > Date: 2018-04-02 11:59:41 +0200 > [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 > 16:24:26 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich > --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 > --with-shared-libraries=1 --download-hypre --with-debugging=no > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #2 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b GIT > Date: 2018-04-02 11:59:41 +0200 > [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 > 16:24:26 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich > --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 > --with-shared-libraries=1 --download-hypre --with-debugging=no > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp > [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > Finished setting up matrix objects > Exiting PrepareNetCDF > [0]PETSC ERROR: ------------------------------ > ------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b GIT > Date: 2018-04-02 11:59:41 +0200 > [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 > 16:24:26 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich > --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 > --with-shared-libraries=1 --download-hypre --with-debugging=no > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp > [0]PETSC ERROR: #6 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59 > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera-w at sdsu.edu Mon Apr 9 18:52:24 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Mon, 9 Apr 2018 16:52:24 -0700 Subject: [petsc-users] Cuda libraries and DMDA In-Reply-To: References: Message-ID: Ok thanks, i'm learning more every day, I still get the same error, i am running with -dm_vec_type viennacl -dm_mat_type aijviennacl -pc_type saviennacl The error: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500 [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090 [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #2 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500 [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090 [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #5 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/interface/precon.c Finished setting up matrix objects Exiting PrepareNetCDF [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500 [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090 [0]PETSC ERROR: #6 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #7 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #8 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c PETSC_Solve_Sol: 1.7096996307373047E-002 PETSC_Solve_SSG: 1.7321825027465820E-002 -.-.-.-.- [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500 [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090 [0]PETSC ERROR: #10 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #11 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #12 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #13 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c PETSC_Solve_Sol: 1.2702941894531250E-002 PETSC_Solve_SSG: 1.2929439544677734E-002 -.-.-.-.- [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500 [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090 [0]PETSC ERROR: #14 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #15 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #16 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #17 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c PETSC_Solve_Sol: 1.2219429016113281E-002 PETSC_Solve_SSG: 1.2480735778808594E-002 -.-.-.-.- [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500 [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090 [0]PETSC ERROR: #18 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #19 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #20 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #21 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c PETSC_Solve_Sol: 1.2894392013549805E-002 PETSC_Solve_SSG: 1.3124227523803711E-002 -.-.-.-.- [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500 [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090 [0]PETSC ERROR: #22 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #23 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #24 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #25 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c PETSC_Solve_Sol: 1.2183427810668945E-002 PETSC_Solve_SSG: 1.2410402297973633E-002 -.-.-.-.- [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500 [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090 [0]PETSC ERROR: #26 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #27 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #28 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #29 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c PETSC_Solve_Sol: 1.2546300888061523E-002 PETSC_Solve_SSG: 1.2778043746948242E-002 -.-.-.-.- [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500 [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090 [0]PETSC ERROR: #30 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #31 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #32 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #33 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c PETSC_Solve_Sol: 1.2255668640136719E-002 PETSC_Solve_SSG: 1.2504816055297852E-002 -.-.-.-.- [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500 [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090 [0]PETSC ERROR: #34 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #35 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #36 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #37 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c PETSC_Solve_Sol: 1.3138055801391602E-002 PETSC_Solve_SSG: 1.3362646102905273E-002 -.-.-.-.- [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500 [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090 [0]PETSC ERROR: #38 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #39 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #40 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #41 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c PETSC_Solve_Sol: 1.2072801589965820E-002 PETSC_Solve_SSG: 1.2319087982177734E-002 -.-.-.-.- [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Currently only handles ViennaCL matrices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500 [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018 [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090 [0]PETSC ERROR: #42 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu [0]PETSC ERROR: #43 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #44 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #45 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c PETSC_Solve_Sol: 1.2498617172241211E-002 PETSC_Solve_SSG: 1.2726306915283203E-002 -.-.-.-.- -.-.-.-.-.- TMain_Loop: 1.1020996570587158 -.-.-.-.-.- WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! On Mon, Apr 9, 2018 at 4:45 PM, Matthew Knepley wrote: > On Mon, Apr 9, 2018 at 7:27 PM, Manuel Valera wrote: > >> On Mon, Apr 9, 2018 at 4:09 PM, Matthew Knepley >> wrote: >> >>> On Mon, Apr 9, 2018 at 6:12 PM, Manuel Valera >>> wrote: >>> >>>> Hello guys, >>>> >>>> I've made advances in my CUDA acceleration project, as you remember i >>>> have a CFD model in need of better execution times. >>>> >>>> So far i have been able to solve the pressure system in the GPU and the >>>> rest in serial, using PETSc only for this pressure solve, the library i got >>>> to work was ViennaCL. First question, do i still have to switch >>>> installations to use either CUDA library? this was a suggestion before, so >>>> in order to use CUSP instead of ViennaCL, for example, i currently have to >>>> change installations, is this still the case? >>>> >>> >>> I am not sure what that means exactly. However, you can build a PETSc >>> with CUDA and ViennaCL support. The type of Vec/Mat is selected at runtime. >>> >> >> Karl Rupp wrote in a previous email: >> >> * * Right now only one of {native CUDA, CUSP, ViennaCL} can be activated >> at configure time. This will be fixed later this month.* >> >> I was asking if this was already solved in 3.9, >> > > Karl knows better than I do. I thought that was fixed, but maybe not in > this release. > > >> >> >>> >>> >>>> Now, i started working in a fully parallelized version of the model, >>>> which uses the DMs and DMDAs to distribute the arrays, if i try the same >>>> flags as before i get an error saying "Currently only handles ViennaCL >>>> matrices" when trying to solve for pressure, i get this is a feature still >>>> not implemented? what options do i have to solve pressure, or assign a DMDA >>>> array update to be done specifically in a GPU device? >>>> >>> >>> If we can't see the error, we are just guessing. Please send the entire >>> error message. >>> >> >> Got it, I will paste the error at the end of this email >> > > It is asking for a ViennaCL matrix. You must tell the DM to create one: > > -dm_mat_type aijviennacl > > >> >> >>> >>> Note, we only do linear algebra on the GPU, so none of the >>> FormFunction/FormJacobian stuff for DMDA would be on the GPU. >>> >> >> Yes, we only use it for linear algebra, e.g. solving a linear system and >> updating an array with a problematic algorithm. >> >> >> >>> >>> >>>> I was thinking of using the VecScatterCreateToZero for a regular vector, >>>> >>> >>> Why do you want a serial vector? >>> >> >> Because it looks live ViennaCL doesn't handle arrays created with >> DMDAVec, it was just an idea >> > > No, it just needs the right type. > > >> >> >>> but then i would have to create a vector and copy the DMDAVec into it, >>>> >>> >>> I do not understand what it means to copy the DM into the Vec. >>> >> >> I meant copying a DMDAVec into a Vec object, the first is created with a >> DMDA object for it's mapping across processors, >> > > There is no such thing as a DMDAVec. Everything is just a Vec. > > Thanks, > > Matt > > >> >>> Thanks, >>> >>> Matt >>> >>> >>>> is this accomplished with DMDAVecGetArrayReadF90 and then just copy? do >>>> you think this will generate too much overhead? >>>> >>>> Thanks so much for your input, >>>> >>>> Manuel >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> >> The error happens when trying to use KSPSolve() for a vector made with >> DMDAVec routines, the matrix is created without any DMDA routines >> >> Error: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b GIT >> Date: 2018-04-02 11:59:41 +0200 >> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >> 16:24:26 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >> --with-shared-libraries=1 --download-hypre --with-debugging=no >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp >> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #2 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b GIT >> Date: 2018-04-02 11:59:41 +0200 >> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >> 16:24:26 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >> --with-shared-libraries=1 --download-hypre --with-debugging=no >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp >> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> Finished setting up matrix objects >> Exiting PrepareNetCDF >> [0]PETSC ERROR: ------------------------------ >> ------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >> ocumentation/faq.html#valgrind >> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS >> X to find memory corruption errors >> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >> and run >> [0]PETSC ERROR: to get more information on the crash. >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Signal received >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b GIT >> Date: 2018-04-02 11:59:41 +0200 >> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >> 16:24:26 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >> --with-shared-libraries=1 --download-hypre --with-debugging=no >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp >> [0]PETSC ERROR: #6 User provided function() line 0 in unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59 >> >> >> >> >> >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 9 18:53:42 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Apr 2018 19:53:42 -0400 Subject: [petsc-users] Cuda libraries and DMDA In-Reply-To: References: Message-ID: On Mon, Apr 9, 2018 at 7:52 PM, Manuel Valera wrote: > Ok thanks, i'm learning more every day, > > I still get the same error, i am running with -dm_vec_type viennacl > -dm_mat_type aijviennacl -pc_type saviennacl > 1) Are you calling DMSetFromOptions()? 2) Does your DM have a prefix? Matt > The error: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT > Date: 2018-04-09 08:39:52 -0500 > [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr > 9 16:50:21 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich > --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 > FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp --download-cusp-commit=116b090 > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #2 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT > Date: 2018-04-09 08:39:52 -0500 > [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr > 9 16:50:21 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich > --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 > FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp --download-cusp-commit=116b090 > [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #5 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > Finished setting up matrix objects > Exiting PrepareNetCDF > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT > Date: 2018-04-09 08:39:52 -0500 > [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr > 9 16:50:21 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich > --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 > FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp --download-cusp-commit=116b090 > [0]PETSC ERROR: #6 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #7 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #8 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: #9 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > PETSC_Solve_Sol: 1.7096996307373047E-002 > PETSC_Solve_SSG: 1.7321825027465820E-002 > -.-.-.-.- > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT > Date: 2018-04-09 08:39:52 -0500 > [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr > 9 16:50:21 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich > --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 > FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp --download-cusp-commit=116b090 > [0]PETSC ERROR: #10 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #11 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #12 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: #13 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > PETSC_Solve_Sol: 1.2702941894531250E-002 > PETSC_Solve_SSG: 1.2929439544677734E-002 > -.-.-.-.- > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT > Date: 2018-04-09 08:39:52 -0500 > [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr > 9 16:50:21 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich > --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 > FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp --download-cusp-commit=116b090 > [0]PETSC ERROR: #14 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #15 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #16 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: #17 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > PETSC_Solve_Sol: 1.2219429016113281E-002 > PETSC_Solve_SSG: 1.2480735778808594E-002 > -.-.-.-.- > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT > Date: 2018-04-09 08:39:52 -0500 > [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr > 9 16:50:21 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich > --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 > FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp --download-cusp-commit=116b090 > [0]PETSC ERROR: #18 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #19 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #20 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: #21 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > PETSC_Solve_Sol: 1.2894392013549805E-002 > PETSC_Solve_SSG: 1.3124227523803711E-002 > -.-.-.-.- > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT > Date: 2018-04-09 08:39:52 -0500 > [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr > 9 16:50:21 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich > --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 > FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp --download-cusp-commit=116b090 > [0]PETSC ERROR: #22 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #23 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #24 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: #25 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > PETSC_Solve_Sol: 1.2183427810668945E-002 > PETSC_Solve_SSG: 1.2410402297973633E-002 > -.-.-.-.- > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT > Date: 2018-04-09 08:39:52 -0500 > [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr > 9 16:50:21 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich > --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 > FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp --download-cusp-commit=116b090 > [0]PETSC ERROR: #26 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #27 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #28 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: #29 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > PETSC_Solve_Sol: 1.2546300888061523E-002 > PETSC_Solve_SSG: 1.2778043746948242E-002 > -.-.-.-.- > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT > Date: 2018-04-09 08:39:52 -0500 > [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr > 9 16:50:21 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich > --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 > FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp --download-cusp-commit=116b090 > [0]PETSC ERROR: #30 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #31 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #32 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: #33 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > PETSC_Solve_Sol: 1.2255668640136719E-002 > PETSC_Solve_SSG: 1.2504816055297852E-002 > -.-.-.-.- > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT > Date: 2018-04-09 08:39:52 -0500 > [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr > 9 16:50:21 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich > --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 > FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp --download-cusp-commit=116b090 > [0]PETSC ERROR: #34 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #35 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #36 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: #37 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > PETSC_Solve_Sol: 1.3138055801391602E-002 > PETSC_Solve_SSG: 1.3362646102905273E-002 > -.-.-.-.- > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT > Date: 2018-04-09 08:39:52 -0500 > [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr > 9 16:50:21 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich > --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 > FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp --download-cusp-commit=116b090 > [0]PETSC ERROR: #38 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #39 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #40 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: #41 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > PETSC_Solve_Sol: 1.2072801589965820E-002 > PETSC_Solve_SSG: 1.2319087982177734E-002 > -.-.-.-.- > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT > Date: 2018-04-09 08:39:52 -0500 > [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr > 9 16:50:21 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich > --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 > FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 > --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl > --download-cusp --download-cusp-commit=116b090 > [0]PETSC ERROR: #42 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #43 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #44 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: #45 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > PETSC_Solve_Sol: 1.2498617172241211E-002 > PETSC_Solve_SSG: 1.2726306915283203E-002 > -.-.-.-.- > -.-.-.-.-.- > TMain_Loop: 1.1020996570587158 > -.-.-.-.-.- > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > > > On Mon, Apr 9, 2018 at 4:45 PM, Matthew Knepley wrote: > >> On Mon, Apr 9, 2018 at 7:27 PM, Manuel Valera wrote: >> >>> On Mon, Apr 9, 2018 at 4:09 PM, Matthew Knepley >>> wrote: >>> >>>> On Mon, Apr 9, 2018 at 6:12 PM, Manuel Valera >>>> wrote: >>>> >>>>> Hello guys, >>>>> >>>>> I've made advances in my CUDA acceleration project, as you remember i >>>>> have a CFD model in need of better execution times. >>>>> >>>>> So far i have been able to solve the pressure system in the GPU and >>>>> the rest in serial, using PETSc only for this pressure solve, the library i >>>>> got to work was ViennaCL. First question, do i still have to switch >>>>> installations to use either CUDA library? this was a suggestion before, so >>>>> in order to use CUSP instead of ViennaCL, for example, i currently have to >>>>> change installations, is this still the case? >>>>> >>>> >>>> I am not sure what that means exactly. However, you can build a PETSc >>>> with CUDA and ViennaCL support. The type of Vec/Mat is selected at runtime. >>>> >>> >>> Karl Rupp wrote in a previous email: >>> >>> * * Right now only one of {native CUDA, CUSP, ViennaCL} can be activated >>> at configure time. This will be fixed later this month.* >>> >>> I was asking if this was already solved in 3.9, >>> >> >> Karl knows better than I do. I thought that was fixed, but maybe not in >> this release. >> >> >>> >>> >>>> >>>> >>>>> Now, i started working in a fully parallelized version of the model, >>>>> which uses the DMs and DMDAs to distribute the arrays, if i try the same >>>>> flags as before i get an error saying "Currently only handles ViennaCL >>>>> matrices" when trying to solve for pressure, i get this is a feature still >>>>> not implemented? what options do i have to solve pressure, or assign a DMDA >>>>> array update to be done specifically in a GPU device? >>>>> >>>> >>>> If we can't see the error, we are just guessing. Please send the entire >>>> error message. >>>> >>> >>> Got it, I will paste the error at the end of this email >>> >> >> It is asking for a ViennaCL matrix. You must tell the DM to create one: >> >> -dm_mat_type aijviennacl >> >> >>> >>> >>>> >>>> Note, we only do linear algebra on the GPU, so none of the >>>> FormFunction/FormJacobian stuff for DMDA would be on the GPU. >>>> >>> >>> Yes, we only use it for linear algebra, e.g. solving a linear system and >>> updating an array with a problematic algorithm. >>> >>> >>> >>>> >>>> >>>>> I was thinking of using the VecScatterCreateToZero for a regular >>>>> vector, >>>>> >>>> >>>> Why do you want a serial vector? >>>> >>> >>> Because it looks live ViennaCL doesn't handle arrays created with >>> DMDAVec, it was just an idea >>> >> >> No, it just needs the right type. >> >> >>> >>> >>>> but then i would have to create a vector and copy the DMDAVec into it, >>>>> >>>> >>>> I do not understand what it means to copy the DM into the Vec. >>>> >>> >>> I meant copying a DMDAVec into a Vec object, the first is created with a >>> DMDA object for it's mapping across processors, >>> >> >> There is no such thing as a DMDAVec. Everything is just a Vec. >> >> Thanks, >> >> Matt >> >> >>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> is this accomplished with DMDAVecGetArrayReadF90 and then just copy? >>>>> do you think this will generate too much overhead? >>>>> >>>>> Thanks so much for your input, >>>>> >>>>> Manuel >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>> >>> The error happens when trying to use KSPSolve() for a vector made with >>> DMDAVec routines, the matrix is created without any DMDA routines >>> >>> Error: >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>> GIT Date: 2018-04-02 11:59:41 +0200 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>> 16:24:26 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp >>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #2 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>> GIT Date: 2018-04-02 11:59:41 +0200 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>> 16:24:26 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp >>> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> Finished setting up matrix objects >>> Exiting PrepareNetCDF >>> [0]PETSC ERROR: ------------------------------ >>> ------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>> probably memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or >>> -on_error_attach_debugger >>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >>> ocumentation/faq.html#valgrind >>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >>> OS X to find memory corruption errors >>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >>> and run >>> [0]PETSC ERROR: to get more information on the crash. >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Signal received >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>> GIT Date: 2018-04-02 11:59:41 +0200 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>> 16:24:26 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp >>> [0]PETSC ERROR: #6 User provided function() line 0 in unknown file >>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59 >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera-w at sdsu.edu Mon Apr 9 18:55:47 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Mon, 9 Apr 2018 16:55:47 -0700 Subject: [petsc-users] Cuda libraries and DMDA In-Reply-To: References: Message-ID: On Mon, Apr 9, 2018 at 4:53 PM, Matthew Knepley wrote: > On Mon, Apr 9, 2018 at 7:52 PM, Manuel Valera wrote: > >> Ok thanks, i'm learning more every day, >> >> I still get the same error, i am running with -dm_vec_type viennacl >> -dm_mat_type aijviennacl -pc_type saviennacl >> > > 1) Are you calling DMSetFromOptions()? > Yes, i get: [valera at node50 Src]$ grep DMSetFromOptions *.f90 DMDAmetrics.f90:call DMSetFromOptions(daDerivs,ierr) dmdaobjs.f90: call DMSetFromOptions(daGrid,ierr) dmdaobjs.f90: call DMSetFromOptions(daSingle,ierr) dmdaobjs.f90: call DMSetFromOptions(daConstants,ierr) dmdaobjs.f90: call DMSetFromOptions(daSgs,ierr) dmdaobjs.f90: call DMSetFromOptions(daLaplacian,ierr) dmdaobjs.f90: call DMSetFromOptions(daMetrics,ierr) dmdaobjs.f90: call DMSetFromOptions(daCenters,ierr) dmdaobjs.f90: call DMSetFromOptions(daPressureCoeffs,ierr) dmdaobjs.f90: call DMSetFromOptions(daDivSgs,ierr) dmdaobjs.f90: call DMSetFromOptions(daDummy,ierr) dmdaobjs.f90: call DMSetFromOptions(daVelocities,ierr) dmdaobjs.f90: call DMSetFromOptions(daScalars,ierr) dmdaobjs.f90: call DMSetFromOptions(daDensity,ierr) fileIO.f90: call DMSetFromOptions(daWriteCenters,ierr) fileIO.f90: call DMSetFromOptions(daWriteUgrid,ierr) fileIO.f90: call DMSetFromOptions(daWriteVgrid,ierr) fileIO.f90: call DMSetFromOptions(daWriteWgrid,ierr) romsmodule.f90: call DMSetFromOptions(daSeqEastWest_u,ierrp) romsmodule.f90: call DMSetFromOptions(daSeqEastWest_v,ierrp) romsmodule.f90: call DMSetFromOptions(daSeqEastWest_w,ierrp) seamountbeamroms.f90: call DMSetFromOptions(daMinPlaneZ,ierr) > > 2) Does your DM have a prefix? > What does this mean? the one used in the KSPSolve is daDummy Thanks, > > Matt > > >> The error: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >> Date: 2018-04-09 08:39:52 -0500 >> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr >> 9 16:50:21 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp --download-cusp-commit=116b090 >> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #2 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >> Date: 2018-04-09 08:39:52 -0500 >> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr >> 9 16:50:21 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp --download-cusp-commit=116b090 >> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #5 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> Finished setting up matrix objects >> Exiting PrepareNetCDF >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >> Date: 2018-04-09 08:39:52 -0500 >> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr >> 9 16:50:21 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp --download-cusp-commit=116b090 >> [0]PETSC ERROR: #6 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #7 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> [0]PETSC ERROR: #8 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> [0]PETSC ERROR: #9 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> PETSC_Solve_Sol: 1.7096996307373047E-002 >> PETSC_Solve_SSG: 1.7321825027465820E-002 >> -.-.-.-.- >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >> Date: 2018-04-09 08:39:52 -0500 >> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr >> 9 16:50:21 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp --download-cusp-commit=116b090 >> [0]PETSC ERROR: #10 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #11 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> [0]PETSC ERROR: #12 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> [0]PETSC ERROR: #13 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> PETSC_Solve_Sol: 1.2702941894531250E-002 >> PETSC_Solve_SSG: 1.2929439544677734E-002 >> -.-.-.-.- >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >> Date: 2018-04-09 08:39:52 -0500 >> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr >> 9 16:50:21 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp --download-cusp-commit=116b090 >> [0]PETSC ERROR: #14 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #15 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> [0]PETSC ERROR: #16 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> [0]PETSC ERROR: #17 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> PETSC_Solve_Sol: 1.2219429016113281E-002 >> PETSC_Solve_SSG: 1.2480735778808594E-002 >> -.-.-.-.- >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >> Date: 2018-04-09 08:39:52 -0500 >> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr >> 9 16:50:21 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp --download-cusp-commit=116b090 >> [0]PETSC ERROR: #18 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #19 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> [0]PETSC ERROR: #20 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> [0]PETSC ERROR: #21 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> PETSC_Solve_Sol: 1.2894392013549805E-002 >> PETSC_Solve_SSG: 1.3124227523803711E-002 >> -.-.-.-.- >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >> Date: 2018-04-09 08:39:52 -0500 >> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr >> 9 16:50:21 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp --download-cusp-commit=116b090 >> [0]PETSC ERROR: #22 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #23 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> [0]PETSC ERROR: #24 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> [0]PETSC ERROR: #25 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> PETSC_Solve_Sol: 1.2183427810668945E-002 >> PETSC_Solve_SSG: 1.2410402297973633E-002 >> -.-.-.-.- >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >> Date: 2018-04-09 08:39:52 -0500 >> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr >> 9 16:50:21 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp --download-cusp-commit=116b090 >> [0]PETSC ERROR: #26 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #27 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> [0]PETSC ERROR: #28 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> [0]PETSC ERROR: #29 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> PETSC_Solve_Sol: 1.2546300888061523E-002 >> PETSC_Solve_SSG: 1.2778043746948242E-002 >> -.-.-.-.- >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >> Date: 2018-04-09 08:39:52 -0500 >> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr >> 9 16:50:21 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp --download-cusp-commit=116b090 >> [0]PETSC ERROR: #30 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #31 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> [0]PETSC ERROR: #32 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> [0]PETSC ERROR: #33 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> PETSC_Solve_Sol: 1.2255668640136719E-002 >> PETSC_Solve_SSG: 1.2504816055297852E-002 >> -.-.-.-.- >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >> Date: 2018-04-09 08:39:52 -0500 >> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr >> 9 16:50:21 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp --download-cusp-commit=116b090 >> [0]PETSC ERROR: #34 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #35 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> [0]PETSC ERROR: #36 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> [0]PETSC ERROR: #37 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> PETSC_Solve_Sol: 1.3138055801391602E-002 >> PETSC_Solve_SSG: 1.3362646102905273E-002 >> -.-.-.-.- >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >> Date: 2018-04-09 08:39:52 -0500 >> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr >> 9 16:50:21 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp --download-cusp-commit=116b090 >> [0]PETSC ERROR: #38 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #39 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> [0]PETSC ERROR: #40 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> [0]PETSC ERROR: #41 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> PETSC_Solve_Sol: 1.2072801589965820E-002 >> PETSC_Solve_SSG: 1.2319087982177734E-002 >> -.-.-.-.- >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: Currently only handles ViennaCL matrices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >> Date: 2018-04-09 08:39:52 -0500 >> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr >> 9 16:50:21 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >> --download-cusp --download-cusp-commit=116b090 >> [0]PETSC ERROR: #42 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> [0]PETSC ERROR: #43 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >> interface/precon.c >> [0]PETSC ERROR: #44 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> [0]PETSC ERROR: #45 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> PETSC_Solve_Sol: 1.2498617172241211E-002 >> PETSC_Solve_SSG: 1.2726306915283203E-002 >> -.-.-.-.- >> -.-.-.-.-.- >> TMain_Loop: 1.1020996570587158 >> -.-.-.-.-.- >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> >> >> On Mon, Apr 9, 2018 at 4:45 PM, Matthew Knepley >> wrote: >> >>> On Mon, Apr 9, 2018 at 7:27 PM, Manuel Valera >>> wrote: >>> >>>> On Mon, Apr 9, 2018 at 4:09 PM, Matthew Knepley >>>> wrote: >>>> >>>>> On Mon, Apr 9, 2018 at 6:12 PM, Manuel Valera >>>>> wrote: >>>>> >>>>>> Hello guys, >>>>>> >>>>>> I've made advances in my CUDA acceleration project, as you remember i >>>>>> have a CFD model in need of better execution times. >>>>>> >>>>>> So far i have been able to solve the pressure system in the GPU and >>>>>> the rest in serial, using PETSc only for this pressure solve, the library i >>>>>> got to work was ViennaCL. First question, do i still have to switch >>>>>> installations to use either CUDA library? this was a suggestion before, so >>>>>> in order to use CUSP instead of ViennaCL, for example, i currently have to >>>>>> change installations, is this still the case? >>>>>> >>>>> >>>>> I am not sure what that means exactly. However, you can build a PETSc >>>>> with CUDA and ViennaCL support. The type of Vec/Mat is selected at runtime. >>>>> >>>> >>>> Karl Rupp wrote in a previous email: >>>> >>>> * * Right now only one of {native CUDA, CUSP, ViennaCL} can be >>>> activated at configure time. This will be fixed later this month.* >>>> >>>> I was asking if this was already solved in 3.9, >>>> >>> >>> Karl knows better than I do. I thought that was fixed, but maybe not in >>> this release. >>> >>> >>>> >>>> >>>>> >>>>> >>>>>> Now, i started working in a fully parallelized version of the model, >>>>>> which uses the DMs and DMDAs to distribute the arrays, if i try the same >>>>>> flags as before i get an error saying "Currently only handles ViennaCL >>>>>> matrices" when trying to solve for pressure, i get this is a feature still >>>>>> not implemented? what options do i have to solve pressure, or assign a DMDA >>>>>> array update to be done specifically in a GPU device? >>>>>> >>>>> >>>>> If we can't see the error, we are just guessing. Please send the >>>>> entire error message. >>>>> >>>> >>>> Got it, I will paste the error at the end of this email >>>> >>> >>> It is asking for a ViennaCL matrix. You must tell the DM to create one: >>> >>> -dm_mat_type aijviennacl >>> >>> >>>> >>>> >>>>> >>>>> Note, we only do linear algebra on the GPU, so none of the >>>>> FormFunction/FormJacobian stuff for DMDA would be on the GPU. >>>>> >>>> >>>> Yes, we only use it for linear algebra, e.g. solving a linear system >>>> and updating an array with a problematic algorithm. >>>> >>>> >>>> >>>>> >>>>> >>>>>> I was thinking of using the VecScatterCreateToZero for a regular >>>>>> vector, >>>>>> >>>>> >>>>> Why do you want a serial vector? >>>>> >>>> >>>> Because it looks live ViennaCL doesn't handle arrays created with >>>> DMDAVec, it was just an idea >>>> >>> >>> No, it just needs the right type. >>> >>> >>>> >>>> >>>>> but then i would have to create a vector and copy the DMDAVec into it, >>>>>> >>>>> >>>>> I do not understand what it means to copy the DM into the Vec. >>>>> >>>> >>>> I meant copying a DMDAVec into a Vec object, the first is created with >>>> a DMDA object for it's mapping across processors, >>>> >>> >>> There is no such thing as a DMDAVec. Everything is just a Vec. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> is this accomplished with DMDAVecGetArrayReadF90 and then just copy? >>>>>> do you think this will generate too much overhead? >>>>>> >>>>>> Thanks so much for your input, >>>>>> >>>>>> Manuel >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>>> The error happens when trying to use KSPSolve() for a vector made with >>>> DMDAVec routines, the matrix is created without any DMDA routines >>>> >>>> Error: >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>>> GIT Date: 2018-04-02 11:59:41 +0200 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>>> 16:24:26 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>> --download-cusp >>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #2 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>>> /interface/itfunc.c >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>>> GIT Date: 2018-04-02 11:59:41 +0200 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>>> 16:24:26 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>> --download-cusp >>>> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> Finished setting up matrix objects >>>> Exiting PrepareNetCDF >>>> [0]PETSC ERROR: ------------------------------ >>>> ------------------------------------------ >>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>>> probably memory access out of range >>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>> -on_error_attach_debugger >>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >>>> ocumentation/faq.html#valgrind >>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >>>> OS X to find memory corruption errors >>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >>>> and run >>>> [0]PETSC ERROR: to get more information on the crash. >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Signal received >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>>> GIT Date: 2018-04-02 11:59:41 +0200 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>>> 16:24:26 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>> --download-cusp >>>> [0]PETSC ERROR: #6 User provided function() line 0 in unknown file >>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >>>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Apr 9 18:59:21 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 9 Apr 2018 18:59:21 -0500 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> Message-ID: I'm able to reproduce this problem on knl box [with the attached test code]. But it goes away if I rebuild without the option --with-64-bit-indices. Will have to check further.. Satish On Thu, 5 Apr 2018, Randall Mackie wrote: > Dear PETSc users, > > I?m curious if anyone else experiences problems using DMDAVecGetArrayF90 in conjunction with Intel compilers? > We have had many problems (typically 11 SEGV segmentation violations) when PETSc is compiled in optimize mode (with various combinations of options). > These same codes run valgrind clean with gfortran, so I assume this is an Intel bug, but before we submit a bug report I wanted to see if anyone else had similar experiences? > We have basically gone back and replaced our calls to DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers into a ?local? subroutine that works fine. > > In case anyone is curious, the attached test code shows this behavior when PETSc is compiled with the following options: > > ./configure \ > --with-clean=1 \ > --with-debugging=0 \ > --with-fortran=1 \ > --with-64-bit-indices \ > --download-mpich=../mpich-3.3a2.tar.gz \ > --with-blas-lapack-dir=/opt/intel/mkl \ > --with-cc=icc \ > --with-fc=ifort \ > --with-cxx=icc \ > --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2? \ > > > > Thanks, Randy M. > > From mvalera-w at sdsu.edu Mon Apr 9 19:06:04 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Mon, 9 Apr 2018 17:06:04 -0700 Subject: [petsc-users] Cuda libraries and DMDA In-Reply-To: References: Message-ID: i dont have anything for DMSetOptionsPrefix On Mon, Apr 9, 2018 at 4:55 PM, Manuel Valera wrote: > > > On Mon, Apr 9, 2018 at 4:53 PM, Matthew Knepley wrote: > >> On Mon, Apr 9, 2018 at 7:52 PM, Manuel Valera wrote: >> >>> Ok thanks, i'm learning more every day, >>> >>> I still get the same error, i am running with -dm_vec_type viennacl >>> -dm_mat_type aijviennacl -pc_type saviennacl >>> >> >> 1) Are you calling DMSetFromOptions()? >> > > Yes, i get: > > [valera at node50 Src]$ grep DMSetFromOptions *.f90 > DMDAmetrics.f90:call DMSetFromOptions(daDerivs,ierr) > dmdaobjs.f90: call DMSetFromOptions(daGrid,ierr) > dmdaobjs.f90: call DMSetFromOptions(daSingle,ierr) > dmdaobjs.f90: call DMSetFromOptions(daConstants,ierr) > dmdaobjs.f90: call DMSetFromOptions(daSgs,ierr) > dmdaobjs.f90: call DMSetFromOptions(daLaplacian,ierr) > dmdaobjs.f90: call DMSetFromOptions(daMetrics,ierr) > dmdaobjs.f90: call DMSetFromOptions(daCenters,ierr) > dmdaobjs.f90: call DMSetFromOptions(daPressureCoeffs,ierr) > dmdaobjs.f90: call DMSetFromOptions(daDivSgs,ierr) > dmdaobjs.f90: call DMSetFromOptions(daDummy,ierr) > dmdaobjs.f90: call DMSetFromOptions(daVelocities,ierr) > dmdaobjs.f90: call DMSetFromOptions(daScalars,ierr) > dmdaobjs.f90: call DMSetFromOptions(daDensity,ierr) > fileIO.f90: call DMSetFromOptions(daWriteCenters,ierr) > fileIO.f90: call DMSetFromOptions(daWriteUgrid,ierr) > fileIO.f90: call DMSetFromOptions(daWriteVgrid,ierr) > fileIO.f90: call DMSetFromOptions(daWriteWgrid,ierr) > romsmodule.f90: call DMSetFromOptions(daSeqEastWest_u,ierrp) > romsmodule.f90: call DMSetFromOptions(daSeqEastWest_v,ierrp) > romsmodule.f90: call DMSetFromOptions(daSeqEastWest_w,ierrp) > seamountbeamroms.f90: call DMSetFromOptions(daMinPlaneZ, > ierr) > > >> >> 2) Does your DM have a prefix? >> > > What does this mean? the one used in the KSPSolve is daDummy > > Thanks, > > > > >> >> Matt >> >> >>> The error: >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #2 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #5 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> Finished setting up matrix objects >>> Exiting PrepareNetCDF >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #6 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #7 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #8 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #9 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.7096996307373047E-002 >>> PETSC_Solve_SSG: 1.7321825027465820E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #10 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #11 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #12 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #13 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2702941894531250E-002 >>> PETSC_Solve_SSG: 1.2929439544677734E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #14 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #15 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #16 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #17 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2219429016113281E-002 >>> PETSC_Solve_SSG: 1.2480735778808594E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #18 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #19 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #20 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #21 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2894392013549805E-002 >>> PETSC_Solve_SSG: 1.3124227523803711E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #22 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #23 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #24 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #25 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2183427810668945E-002 >>> PETSC_Solve_SSG: 1.2410402297973633E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #26 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #27 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #28 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #29 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2546300888061523E-002 >>> PETSC_Solve_SSG: 1.2778043746948242E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #30 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #31 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #32 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #33 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2255668640136719E-002 >>> PETSC_Solve_SSG: 1.2504816055297852E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #34 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #35 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #36 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #37 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.3138055801391602E-002 >>> PETSC_Solve_SSG: 1.3362646102905273E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #38 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #39 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #40 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #41 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2072801589965820E-002 >>> PETSC_Solve_SSG: 1.2319087982177734E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #42 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #43 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #44 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #45 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2498617172241211E-002 >>> PETSC_Solve_SSG: 1.2726306915283203E-002 >>> -.-.-.-.- >>> -.-.-.-.-.- >>> TMain_Loop: 1.1020996570587158 >>> -.-.-.-.-.- >>> WARNING! There are options you set that were not used! >>> WARNING! could be spelling mistake, etc! >>> >>> >>> On Mon, Apr 9, 2018 at 4:45 PM, Matthew Knepley >>> wrote: >>> >>>> On Mon, Apr 9, 2018 at 7:27 PM, Manuel Valera >>>> wrote: >>>> >>>>> On Mon, Apr 9, 2018 at 4:09 PM, Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Mon, Apr 9, 2018 at 6:12 PM, Manuel Valera >>>>>> wrote: >>>>>> >>>>>>> Hello guys, >>>>>>> >>>>>>> I've made advances in my CUDA acceleration project, as you remember >>>>>>> i have a CFD model in need of better execution times. >>>>>>> >>>>>>> So far i have been able to solve the pressure system in the GPU and >>>>>>> the rest in serial, using PETSc only for this pressure solve, the library i >>>>>>> got to work was ViennaCL. First question, do i still have to switch >>>>>>> installations to use either CUDA library? this was a suggestion before, so >>>>>>> in order to use CUSP instead of ViennaCL, for example, i currently have to >>>>>>> change installations, is this still the case? >>>>>>> >>>>>> >>>>>> I am not sure what that means exactly. However, you can build a PETSc >>>>>> with CUDA and ViennaCL support. The type of Vec/Mat is selected at runtime. >>>>>> >>>>> >>>>> Karl Rupp wrote in a previous email: >>>>> >>>>> * * Right now only one of {native CUDA, CUSP, ViennaCL} can be >>>>> activated at configure time. This will be fixed later this month.* >>>>> >>>>> I was asking if this was already solved in 3.9, >>>>> >>>> >>>> Karl knows better than I do. I thought that was fixed, but maybe not in >>>> this release. >>>> >>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>>> Now, i started working in a fully parallelized version of the model, >>>>>>> which uses the DMs and DMDAs to distribute the arrays, if i try the same >>>>>>> flags as before i get an error saying "Currently only handles ViennaCL >>>>>>> matrices" when trying to solve for pressure, i get this is a feature still >>>>>>> not implemented? what options do i have to solve pressure, or assign a DMDA >>>>>>> array update to be done specifically in a GPU device? >>>>>>> >>>>>> >>>>>> If we can't see the error, we are just guessing. Please send the >>>>>> entire error message. >>>>>> >>>>> >>>>> Got it, I will paste the error at the end of this email >>>>> >>>> >>>> It is asking for a ViennaCL matrix. You must tell the DM to create one: >>>> >>>> -dm_mat_type aijviennacl >>>> >>>> >>>>> >>>>> >>>>>> >>>>>> Note, we only do linear algebra on the GPU, so none of the >>>>>> FormFunction/FormJacobian stuff for DMDA would be on the GPU. >>>>>> >>>>> >>>>> Yes, we only use it for linear algebra, e.g. solving a linear system >>>>> and updating an array with a problematic algorithm. >>>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>>> I was thinking of using the VecScatterCreateToZero for a regular >>>>>>> vector, >>>>>>> >>>>>> >>>>>> Why do you want a serial vector? >>>>>> >>>>> >>>>> Because it looks live ViennaCL doesn't handle arrays created with >>>>> DMDAVec, it was just an idea >>>>> >>>> >>>> No, it just needs the right type. >>>> >>>> >>>>> >>>>> >>>>>> but then i would have to create a vector and copy the DMDAVec into it, >>>>>>> >>>>>> >>>>>> I do not understand what it means to copy the DM into the Vec. >>>>>> >>>>> >>>>> I meant copying a DMDAVec into a Vec object, the first is created with >>>>> a DMDA object for it's mapping across processors, >>>>> >>>> >>>> There is no such thing as a DMDAVec. Everything is just a Vec. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> is this accomplished with DMDAVecGetArrayReadF90 and then just copy? >>>>>>> do you think this will generate too much overhead? >>>>>>> >>>>>>> Thanks so much for your input, >>>>>>> >>>>>>> Manuel >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>> >>>>> The error happens when trying to use KSPSolve() for a vector made with >>>>> DMDAVec routines, the matrix is created without any DMDA routines >>>>> >>>>> Error: >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>>>> GIT Date: 2018-04-02 11:59:41 +0200 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>>>> 16:24:26 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>> --download-cusp >>>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #2 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/ >>>>> interface/precon.c >>>>> [0]PETSC ERROR: #3 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>>>> GIT Date: 2018-04-02 11:59:41 +0200 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>>>> 16:24:26 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>> --download-cusp >>>>> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/ >>>>> interface/precon.c >>>>> Finished setting up matrix objects >>>>> Exiting PrepareNetCDF >>>>> [0]PETSC ERROR: ------------------------------ >>>>> ------------------------------------------ >>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>>>> probably memory access out of range >>>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>>> -on_error_attach_debugger >>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html#valgrind >>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >>>>> OS X to find memory corruption errors >>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >>>>> and run >>>>> [0]PETSC ERROR: to get more information on the crash. >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Signal received >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>>>> GIT Date: 2018-04-02 11:59:41 +0200 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>>>> 16:24:26 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>> --download-cusp >>>>> [0]PETSC ERROR: #6 User provided function() line 0 in unknown file >>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >>>>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 9 19:57:52 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Apr 2018 20:57:52 -0400 Subject: [petsc-users] Cuda libraries and DMDA In-Reply-To: References: Message-ID: On Mon, Apr 9, 2018 at 7:55 PM, Manuel Valera wrote: > On Mon, Apr 9, 2018 at 4:53 PM, Matthew Knepley wrote: > >> On Mon, Apr 9, 2018 at 7:52 PM, Manuel Valera wrote: >> >>> Ok thanks, i'm learning more every day, >>> >>> I still get the same error, i am running with -dm_vec_type viennacl >>> -dm_mat_type aijviennacl -pc_type saviennacl >>> >> >> 1) Are you calling DMSetFromOptions()? >> > > Yes, i get: > Let me be more precise. For the DMDA you are using, you must call DMSetFromOptions() before you call DMCreateMatrix (or before its called automatically), in order to get the Mat type you want from the command line. If you do not get the right type, you know this has not happened. Matt > [valera at node50 Src]$ grep DMSetFromOptions *.f90 > DMDAmetrics.f90:call DMSetFromOptions(daDerivs,ierr) > dmdaobjs.f90: call DMSetFromOptions(daGrid,ierr) > dmdaobjs.f90: call DMSetFromOptions(daSingle,ierr) > dmdaobjs.f90: call DMSetFromOptions(daConstants,ierr) > dmdaobjs.f90: call DMSetFromOptions(daSgs,ierr) > dmdaobjs.f90: call DMSetFromOptions(daLaplacian,ierr) > dmdaobjs.f90: call DMSetFromOptions(daMetrics,ierr) > dmdaobjs.f90: call DMSetFromOptions(daCenters,ierr) > dmdaobjs.f90: call DMSetFromOptions(daPressureCoeffs,ierr) > dmdaobjs.f90: call DMSetFromOptions(daDivSgs,ierr) > dmdaobjs.f90: call DMSetFromOptions(daDummy,ierr) > dmdaobjs.f90: call DMSetFromOptions(daVelocities,ierr) > dmdaobjs.f90: call DMSetFromOptions(daScalars,ierr) > dmdaobjs.f90: call DMSetFromOptions(daDensity,ierr) > fileIO.f90: call DMSetFromOptions(daWriteCenters,ierr) > fileIO.f90: call DMSetFromOptions(daWriteUgrid,ierr) > fileIO.f90: call DMSetFromOptions(daWriteVgrid,ierr) > fileIO.f90: call DMSetFromOptions(daWriteWgrid,ierr) > romsmodule.f90: call DMSetFromOptions(daSeqEastWest_u,ierrp) > romsmodule.f90: call DMSetFromOptions(daSeqEastWest_v,ierrp) > romsmodule.f90: call DMSetFromOptions(daSeqEastWest_w,ierrp) > seamountbeamroms.f90: call DMSetFromOptions(daMinPlaneZ, > ierr) > > >> >> 2) Does your DM have a prefix? >> > > What does this mean? the one used in the KSPSolve is daDummy > > Thanks, > > > > >> >> Matt >> >> >>> The error: >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #2 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #5 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> Finished setting up matrix objects >>> Exiting PrepareNetCDF >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #6 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #7 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #8 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #9 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.7096996307373047E-002 >>> PETSC_Solve_SSG: 1.7321825027465820E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #10 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #11 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #12 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #13 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2702941894531250E-002 >>> PETSC_Solve_SSG: 1.2929439544677734E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #14 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #15 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #16 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #17 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2219429016113281E-002 >>> PETSC_Solve_SSG: 1.2480735778808594E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #18 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #19 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #20 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #21 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2894392013549805E-002 >>> PETSC_Solve_SSG: 1.3124227523803711E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #22 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #23 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #24 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #25 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2183427810668945E-002 >>> PETSC_Solve_SSG: 1.2410402297973633E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #26 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #27 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #28 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #29 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2546300888061523E-002 >>> PETSC_Solve_SSG: 1.2778043746948242E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #30 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #31 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #32 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #33 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2255668640136719E-002 >>> PETSC_Solve_SSG: 1.2504816055297852E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #34 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #35 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #36 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #37 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.3138055801391602E-002 >>> PETSC_Solve_SSG: 1.3362646102905273E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #38 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #39 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #40 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #41 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2072801589965820E-002 >>> PETSC_Solve_SSG: 1.2319087982177734E-002 >>> -.-.-.-.- >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>> Date: 2018-04-09 08:39:52 -0500 >>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>> Apr 9 16:50:21 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich >>> --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 >>> FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 >>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>> --download-cusp --download-cusp-commit=116b090 >>> [0]PETSC ERROR: #42 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #43 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: #44 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> [0]PETSC ERROR: #45 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>> /interface/itfunc.c >>> PETSC_Solve_Sol: 1.2498617172241211E-002 >>> PETSC_Solve_SSG: 1.2726306915283203E-002 >>> -.-.-.-.- >>> -.-.-.-.-.- >>> TMain_Loop: 1.1020996570587158 >>> -.-.-.-.-.- >>> WARNING! There are options you set that were not used! >>> WARNING! could be spelling mistake, etc! >>> >>> >>> On Mon, Apr 9, 2018 at 4:45 PM, Matthew Knepley >>> wrote: >>> >>>> On Mon, Apr 9, 2018 at 7:27 PM, Manuel Valera >>>> wrote: >>>> >>>>> On Mon, Apr 9, 2018 at 4:09 PM, Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Mon, Apr 9, 2018 at 6:12 PM, Manuel Valera >>>>>> wrote: >>>>>> >>>>>>> Hello guys, >>>>>>> >>>>>>> I've made advances in my CUDA acceleration project, as you remember >>>>>>> i have a CFD model in need of better execution times. >>>>>>> >>>>>>> So far i have been able to solve the pressure system in the GPU and >>>>>>> the rest in serial, using PETSc only for this pressure solve, the library i >>>>>>> got to work was ViennaCL. First question, do i still have to switch >>>>>>> installations to use either CUDA library? this was a suggestion before, so >>>>>>> in order to use CUSP instead of ViennaCL, for example, i currently have to >>>>>>> change installations, is this still the case? >>>>>>> >>>>>> >>>>>> I am not sure what that means exactly. However, you can build a PETSc >>>>>> with CUDA and ViennaCL support. The type of Vec/Mat is selected at runtime. >>>>>> >>>>> >>>>> Karl Rupp wrote in a previous email: >>>>> >>>>> * * Right now only one of {native CUDA, CUSP, ViennaCL} can be >>>>> activated at configure time. This will be fixed later this month.* >>>>> >>>>> I was asking if this was already solved in 3.9, >>>>> >>>> >>>> Karl knows better than I do. I thought that was fixed, but maybe not in >>>> this release. >>>> >>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>>> Now, i started working in a fully parallelized version of the model, >>>>>>> which uses the DMs and DMDAs to distribute the arrays, if i try the same >>>>>>> flags as before i get an error saying "Currently only handles ViennaCL >>>>>>> matrices" when trying to solve for pressure, i get this is a feature still >>>>>>> not implemented? what options do i have to solve pressure, or assign a DMDA >>>>>>> array update to be done specifically in a GPU device? >>>>>>> >>>>>> >>>>>> If we can't see the error, we are just guessing. Please send the >>>>>> entire error message. >>>>>> >>>>> >>>>> Got it, I will paste the error at the end of this email >>>>> >>>> >>>> It is asking for a ViennaCL matrix. You must tell the DM to create one: >>>> >>>> -dm_mat_type aijviennacl >>>> >>>> >>>>> >>>>> >>>>>> >>>>>> Note, we only do linear algebra on the GPU, so none of the >>>>>> FormFunction/FormJacobian stuff for DMDA would be on the GPU. >>>>>> >>>>> >>>>> Yes, we only use it for linear algebra, e.g. solving a linear system >>>>> and updating an array with a problematic algorithm. >>>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>>> I was thinking of using the VecScatterCreateToZero for a regular >>>>>>> vector, >>>>>>> >>>>>> >>>>>> Why do you want a serial vector? >>>>>> >>>>> >>>>> Because it looks live ViennaCL doesn't handle arrays created with >>>>> DMDAVec, it was just an idea >>>>> >>>> >>>> No, it just needs the right type. >>>> >>>> >>>>> >>>>> >>>>>> but then i would have to create a vector and copy the DMDAVec into it, >>>>>>> >>>>>> >>>>>> I do not understand what it means to copy the DM into the Vec. >>>>>> >>>>> >>>>> I meant copying a DMDAVec into a Vec object, the first is created with >>>>> a DMDA object for it's mapping across processors, >>>>> >>>> >>>> There is no such thing as a DMDAVec. Everything is just a Vec. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> is this accomplished with DMDAVecGetArrayReadF90 and then just copy? >>>>>>> do you think this will generate too much overhead? >>>>>>> >>>>>>> Thanks so much for your input, >>>>>>> >>>>>>> Manuel >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>> >>>>> The error happens when trying to use KSPSolve() for a vector made with >>>>> DMDAVec routines, the matrix is created without any DMDA routines >>>>> >>>>> Error: >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>>>> GIT Date: 2018-04-02 11:59:41 +0200 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>>>> 16:24:26 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>> --download-cusp >>>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #2 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/ >>>>> interface/precon.c >>>>> [0]PETSC ERROR: #3 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>>>> GIT Date: 2018-04-02 11:59:41 +0200 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>>>> 16:24:26 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>> --download-cusp >>>>> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/ >>>>> interface/precon.c >>>>> Finished setting up matrix objects >>>>> Exiting PrepareNetCDF >>>>> [0]PETSC ERROR: ------------------------------ >>>>> ------------------------------------------ >>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>>>> probably memory access out of range >>>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>>> -on_error_attach_debugger >>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html#valgrind >>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >>>>> OS X to find memory corruption errors >>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >>>>> and run >>>>> [0]PETSC ERROR: to get more information on the crash. >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Signal received >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>>>> GIT Date: 2018-04-02 11:59:41 +0200 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>>>> 16:24:26 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>> --download-cusp >>>>> [0]PETSC ERROR: #6 User provided function() line 0 in unknown file >>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >>>>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera-w at sdsu.edu Mon Apr 9 20:13:38 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Mon, 9 Apr 2018 18:13:38 -0700 Subject: [petsc-users] Cuda libraries and DMDA In-Reply-To: References: Message-ID: Oh ok, thanks Matt, I think the problem is that i am not using DMCreateMatrix at all but a regular MatCreate, i will follow ex44f and get back to you, I am calling DMSetFromOptions right after creating the DMDA, way before the matrix is created, On Mon, Apr 9, 2018 at 5:57 PM, Matthew Knepley wrote: > On Mon, Apr 9, 2018 at 7:55 PM, Manuel Valera wrote: > >> On Mon, Apr 9, 2018 at 4:53 PM, Matthew Knepley >> wrote: >> >>> On Mon, Apr 9, 2018 at 7:52 PM, Manuel Valera >>> wrote: >>> >>>> Ok thanks, i'm learning more every day, >>>> >>>> I still get the same error, i am running with -dm_vec_type viennacl >>>> -dm_mat_type aijviennacl -pc_type saviennacl >>>> >>> >>> 1) Are you calling DMSetFromOptions()? >>> >> >> Yes, i get: >> > > Let me be more precise. For the DMDA you are using, you must call > DMSetFromOptions() before you call > DMCreateMatrix (or before its called automatically), in order to get the > Mat type you want from the command line. > If you do not get the right type, you know this has not happened. > > Matt > > >> [valera at node50 Src]$ grep DMSetFromOptions *.f90 >> DMDAmetrics.f90:call DMSetFromOptions(daDerivs,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daGrid,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daSingle,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daConstants,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daSgs,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daLaplacian,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daMetrics,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daCenters,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daPressureCoeffs,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daDivSgs,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daDummy,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daVelocities,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daScalars,ierr) >> dmdaobjs.f90: call DMSetFromOptions(daDensity,ierr) >> fileIO.f90: call DMSetFromOptions(daWriteCenters,ierr) >> fileIO.f90: call DMSetFromOptions(daWriteUgrid,ierr) >> fileIO.f90: call DMSetFromOptions(daWriteVgrid,ierr) >> fileIO.f90: call DMSetFromOptions(daWriteWgrid,ierr) >> romsmodule.f90: call DMSetFromOptions(daSeqEastWest_u,ierrp) >> romsmodule.f90: call DMSetFromOptions(daSeqEastWest_v,ierrp) >> romsmodule.f90: call DMSetFromOptions(daSeqEastWest_w,ierrp) >> seamountbeamroms.f90: call DMSetFromOptions(daMinPlaneZ,i >> err) >> >> >>> >>> 2) Does your DM have a prefix? >>> >> >> What does this mean? the one used in the KSPSolve is daDummy >> >> Thanks, >> >> >> >> >>> >>> Matt >>> >>> >>>> The error: >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>> Date: 2018-04-09 08:39:52 -0500 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>> Apr 9 16:50:21 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #2 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>>> /interface/itfunc.c >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>> Date: 2018-04-09 08:39:52 -0500 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>> Apr 9 16:50:21 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #5 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> Finished setting up matrix objects >>>> Exiting PrepareNetCDF >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>> Date: 2018-04-09 08:39:52 -0500 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>> Apr 9 16:50:21 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>> [0]PETSC ERROR: #6 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #7 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> [0]PETSC ERROR: #8 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp >>>> /interface/itfunc.c >>>> [0]PETSC ERROR: #9 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp >>>> /interface/itfunc.c >>>> PETSC_Solve_Sol: 1.7096996307373047E-002 >>>> PETSC_Solve_SSG: 1.7321825027465820E-002 >>>> -.-.-.-.- >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>> Date: 2018-04-09 08:39:52 -0500 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>> Apr 9 16:50:21 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>> [0]PETSC ERROR: #10 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #11 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> [0]PETSC ERROR: #12 KSPSetUp() line 381 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #13 KSPSolve() line 612 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> PETSC_Solve_Sol: 1.2702941894531250E-002 >>>> PETSC_Solve_SSG: 1.2929439544677734E-002 >>>> -.-.-.-.- >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>> Date: 2018-04-09 08:39:52 -0500 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>> Apr 9 16:50:21 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>> [0]PETSC ERROR: #14 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #15 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> [0]PETSC ERROR: #16 KSPSetUp() line 381 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #17 KSPSolve() line 612 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> PETSC_Solve_Sol: 1.2219429016113281E-002 >>>> PETSC_Solve_SSG: 1.2480735778808594E-002 >>>> -.-.-.-.- >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>> Date: 2018-04-09 08:39:52 -0500 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>> Apr 9 16:50:21 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>> [0]PETSC ERROR: #18 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #19 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> [0]PETSC ERROR: #20 KSPSetUp() line 381 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #21 KSPSolve() line 612 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> PETSC_Solve_Sol: 1.2894392013549805E-002 >>>> PETSC_Solve_SSG: 1.3124227523803711E-002 >>>> -.-.-.-.- >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>> Date: 2018-04-09 08:39:52 -0500 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>> Apr 9 16:50:21 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>> [0]PETSC ERROR: #22 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #23 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> [0]PETSC ERROR: #24 KSPSetUp() line 381 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #25 KSPSolve() line 612 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> PETSC_Solve_Sol: 1.2183427810668945E-002 >>>> PETSC_Solve_SSG: 1.2410402297973633E-002 >>>> -.-.-.-.- >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>> Date: 2018-04-09 08:39:52 -0500 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>> Apr 9 16:50:21 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>> [0]PETSC ERROR: #26 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #27 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> [0]PETSC ERROR: #28 KSPSetUp() line 381 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #29 KSPSolve() line 612 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> PETSC_Solve_Sol: 1.2546300888061523E-002 >>>> PETSC_Solve_SSG: 1.2778043746948242E-002 >>>> -.-.-.-.- >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>> Date: 2018-04-09 08:39:52 -0500 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>> Apr 9 16:50:21 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>> [0]PETSC ERROR: #30 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #31 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> [0]PETSC ERROR: #32 KSPSetUp() line 381 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #33 KSPSolve() line 612 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> PETSC_Solve_Sol: 1.2255668640136719E-002 >>>> PETSC_Solve_SSG: 1.2504816055297852E-002 >>>> -.-.-.-.- >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>> Date: 2018-04-09 08:39:52 -0500 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>> Apr 9 16:50:21 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>> [0]PETSC ERROR: #34 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #35 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> [0]PETSC ERROR: #36 KSPSetUp() line 381 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #37 KSPSolve() line 612 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> PETSC_Solve_Sol: 1.3138055801391602E-002 >>>> PETSC_Solve_SSG: 1.3362646102905273E-002 >>>> -.-.-.-.- >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>> Date: 2018-04-09 08:39:52 -0500 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>> Apr 9 16:50:21 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>> [0]PETSC ERROR: #38 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #39 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> [0]PETSC ERROR: #40 KSPSetUp() line 381 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #41 KSPSolve() line 612 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> PETSC_Solve_Sol: 1.2072801589965820E-002 >>>> PETSC_Solve_SSG: 1.2319087982177734E-002 >>>> -.-.-.-.- >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>> Date: 2018-04-09 08:39:52 -0500 >>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>> Apr 9 16:50:21 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>> [0]PETSC ERROR: #42 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> [0]PETSC ERROR: #43 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>> interface/precon.c >>>> [0]PETSC ERROR: #44 KSPSetUp() line 381 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #45 KSPSolve() line 612 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> PETSC_Solve_Sol: 1.2498617172241211E-002 >>>> PETSC_Solve_SSG: 1.2726306915283203E-002 >>>> -.-.-.-.- >>>> -.-.-.-.-.- >>>> TMain_Loop: 1.1020996570587158 >>>> -.-.-.-.-.- >>>> WARNING! There are options you set that were not used! >>>> WARNING! could be spelling mistake, etc! >>>> >>>> >>>> On Mon, Apr 9, 2018 at 4:45 PM, Matthew Knepley >>>> wrote: >>>> >>>>> On Mon, Apr 9, 2018 at 7:27 PM, Manuel Valera >>>>> wrote: >>>>> >>>>>> On Mon, Apr 9, 2018 at 4:09 PM, Matthew Knepley >>>>>> wrote: >>>>>> >>>>>>> On Mon, Apr 9, 2018 at 6:12 PM, Manuel Valera >>>>>>> wrote: >>>>>>> >>>>>>>> Hello guys, >>>>>>>> >>>>>>>> I've made advances in my CUDA acceleration project, as you remember >>>>>>>> i have a CFD model in need of better execution times. >>>>>>>> >>>>>>>> So far i have been able to solve the pressure system in the GPU and >>>>>>>> the rest in serial, using PETSc only for this pressure solve, the library i >>>>>>>> got to work was ViennaCL. First question, do i still have to switch >>>>>>>> installations to use either CUDA library? this was a suggestion before, so >>>>>>>> in order to use CUSP instead of ViennaCL, for example, i currently have to >>>>>>>> change installations, is this still the case? >>>>>>>> >>>>>>> >>>>>>> I am not sure what that means exactly. However, you can build a >>>>>>> PETSc with CUDA and ViennaCL support. The type of Vec/Mat is selected at >>>>>>> runtime. >>>>>>> >>>>>> >>>>>> Karl Rupp wrote in a previous email: >>>>>> >>>>>> * * Right now only one of {native CUDA, CUSP, ViennaCL} can be >>>>>> activated at configure time. This will be fixed later this month.* >>>>>> >>>>>> I was asking if this was already solved in 3.9, >>>>>> >>>>> >>>>> Karl knows better than I do. I thought that was fixed, but maybe not >>>>> in this release. >>>>> >>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> Now, i started working in a fully parallelized version of the >>>>>>>> model, which uses the DMs and DMDAs to distribute the arrays, if i try the >>>>>>>> same flags as before i get an error saying "Currently only handles ViennaCL >>>>>>>> matrices" when trying to solve for pressure, i get this is a feature still >>>>>>>> not implemented? what options do i have to solve pressure, or assign a DMDA >>>>>>>> array update to be done specifically in a GPU device? >>>>>>>> >>>>>>> >>>>>>> If we can't see the error, we are just guessing. Please send the >>>>>>> entire error message. >>>>>>> >>>>>> >>>>>> Got it, I will paste the error at the end of this email >>>>>> >>>>> >>>>> It is asking for a ViennaCL matrix. You must tell the DM to create one: >>>>> >>>>> -dm_mat_type aijviennacl >>>>> >>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Note, we only do linear algebra on the GPU, so none of the >>>>>>> FormFunction/FormJacobian stuff for DMDA would be on the GPU. >>>>>>> >>>>>> >>>>>> Yes, we only use it for linear algebra, e.g. solving a linear system >>>>>> and updating an array with a problematic algorithm. >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> I was thinking of using the VecScatterCreateToZero for a regular >>>>>>>> vector, >>>>>>>> >>>>>>> >>>>>>> Why do you want a serial vector? >>>>>>> >>>>>> >>>>>> Because it looks live ViennaCL doesn't handle arrays created with >>>>>> DMDAVec, it was just an idea >>>>>> >>>>> >>>>> No, it just needs the right type. >>>>> >>>>> >>>>>> >>>>>> >>>>>>> but then i would have to create a vector and copy the DMDAVec into >>>>>>>> it, >>>>>>>> >>>>>>> >>>>>>> I do not understand what it means to copy the DM into the Vec. >>>>>>> >>>>>> >>>>>> I meant copying a DMDAVec into a Vec object, the first is created >>>>>> with a DMDA object for it's mapping across processors, >>>>>> >>>>> >>>>> There is no such thing as a DMDAVec. Everything is just a Vec. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> is this accomplished with DMDAVecGetArrayReadF90 and then just >>>>>>>> copy? do you think this will generate too much overhead? >>>>>>>> >>>>>>>> Thanks so much for your input, >>>>>>>> >>>>>>>> Manuel >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> >>>>>> >>>>>> The error happens when trying to use KSPSolve() for a vector made >>>>>> with DMDAVec routines, the matrix is created without any DMDA routines >>>>>> >>>>>> Error: >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>>>>> GIT Date: 2018-04-02 11:59:41 +0200 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>>>>> 16:24:26 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>>> --download-cusp >>>>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #2 PCSetUp() line 924 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #3 KSPSetUp() line 381 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>>>>> GIT Date: 2018-04-02 11:59:41 +0200 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>>>>> 16:24:26 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>>> --download-cusp >>>>>> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> Finished setting up matrix objects >>>>>> Exiting PrepareNetCDF >>>>>> [0]PETSC ERROR: ------------------------------ >>>>>> ------------------------------------------ >>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>>>>> probably memory access out of range >>>>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>>>> -on_error_attach_debugger >>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html#valgrind >>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple >>>>>> Mac OS X to find memory corruption errors >>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, >>>>>> link, and run >>>>>> [0]PETSC ERROR: to get more information on the crash. >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Signal received >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b >>>>>> GIT Date: 2018-04-02 11:59:41 +0200 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 >>>>>> 16:24:26 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>>> --download-cusp >>>>>> [0]PETSC ERROR: #6 User provided function() line 0 in unknown file >>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >>>>>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera-w at sdsu.edu Mon Apr 9 20:34:58 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Mon, 9 Apr 2018 18:34:58 -0700 Subject: [petsc-users] Cuda libraries and DMDA In-Reply-To: References: Message-ID: It looks like it's working now :) i needed to setup the DMMatrix and that made the trick, Thanks, On Mon, Apr 9, 2018 at 6:13 PM, Manuel Valera wrote: > Oh ok, thanks Matt, > > I think the problem is that i am not using DMCreateMatrix at all but a > regular MatCreate, i will follow ex44f and get back to you, > > I am calling DMSetFromOptions right after creating the DMDA, way before > the matrix is created, > > > > On Mon, Apr 9, 2018 at 5:57 PM, Matthew Knepley wrote: > >> On Mon, Apr 9, 2018 at 7:55 PM, Manuel Valera wrote: >> >>> On Mon, Apr 9, 2018 at 4:53 PM, Matthew Knepley >>> wrote: >>> >>>> On Mon, Apr 9, 2018 at 7:52 PM, Manuel Valera >>>> wrote: >>>> >>>>> Ok thanks, i'm learning more every day, >>>>> >>>>> I still get the same error, i am running with -dm_vec_type viennacl >>>>> -dm_mat_type aijviennacl -pc_type saviennacl >>>>> >>>> >>>> 1) Are you calling DMSetFromOptions()? >>>> >>> >>> Yes, i get: >>> >> >> Let me be more precise. For the DMDA you are using, you must call >> DMSetFromOptions() before you call >> DMCreateMatrix (or before its called automatically), in order to get the >> Mat type you want from the command line. >> If you do not get the right type, you know this has not happened. >> >> Matt >> >> >>> [valera at node50 Src]$ grep DMSetFromOptions *.f90 >>> DMDAmetrics.f90:call DMSetFromOptions(daDerivs,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daGrid,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daSingle,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daConstants,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daSgs,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daLaplacian,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daMetrics,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daCenters,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daPressureCoeffs,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daDivSgs,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daDummy,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daVelocities,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daScalars,ierr) >>> dmdaobjs.f90: call DMSetFromOptions(daDensity,ierr) >>> fileIO.f90: call DMSetFromOptions(daWriteCenters,ierr) >>> fileIO.f90: call DMSetFromOptions(daWriteUgrid,ierr) >>> fileIO.f90: call DMSetFromOptions(daWriteVgrid,ierr) >>> fileIO.f90: call DMSetFromOptions(daWriteWgrid,ierr) >>> romsmodule.f90: call DMSetFromOptions(daSeqEastWest_u,ierrp) >>> romsmodule.f90: call DMSetFromOptions(daSeqEastWest_v,ierrp) >>> romsmodule.f90: call DMSetFromOptions(daSeqEastWest_w,ierrp) >>> seamountbeamroms.f90: call DMSetFromOptions(daMinPlaneZ,i >>> err) >>> >>> >>>> >>>> 2) Does your DM have a prefix? >>>> >>> >>> What does this mean? the one used in the KSPSolve is daDummy >>> >>> Thanks, >>> >>> >>> >>> >>>> >>>> Matt >>>> >>>> >>>>> The error: >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>> Date: 2018-04-09 08:39:52 -0500 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>> Apr 9 16:50:21 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #2 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>>> interface/precon.c >>>>> [0]PETSC ERROR: #3 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>> Date: 2018-04-09 08:39:52 -0500 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>> Apr 9 16:50:21 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #5 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>>> interface/precon.c >>>>> Finished setting up matrix objects >>>>> Exiting PrepareNetCDF >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>> Date: 2018-04-09 08:39:52 -0500 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>> Apr 9 16:50:21 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>> [0]PETSC ERROR: #6 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #7 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/ >>>>> interface/precon.c >>>>> [0]PETSC ERROR: #8 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #9 KSPSolve() line 612 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> PETSC_Solve_Sol: 1.7096996307373047E-002 >>>>> PETSC_Solve_SSG: 1.7321825027465820E-002 >>>>> -.-.-.-.- >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>> Date: 2018-04-09 08:39:52 -0500 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>> Apr 9 16:50:21 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>> [0]PETSC ERROR: #10 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #11 PCSetUp() line 923 in >>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: #12 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #13 KSPSolve() line 612 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> PETSC_Solve_Sol: 1.2702941894531250E-002 >>>>> PETSC_Solve_SSG: 1.2929439544677734E-002 >>>>> -.-.-.-.- >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>> Date: 2018-04-09 08:39:52 -0500 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>> Apr 9 16:50:21 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>> [0]PETSC ERROR: #14 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #15 PCSetUp() line 923 in >>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: #16 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #17 KSPSolve() line 612 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> PETSC_Solve_Sol: 1.2219429016113281E-002 >>>>> PETSC_Solve_SSG: 1.2480735778808594E-002 >>>>> -.-.-.-.- >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>> Date: 2018-04-09 08:39:52 -0500 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>> Apr 9 16:50:21 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>> [0]PETSC ERROR: #18 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #19 PCSetUp() line 923 in >>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: #20 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #21 KSPSolve() line 612 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> PETSC_Solve_Sol: 1.2894392013549805E-002 >>>>> PETSC_Solve_SSG: 1.3124227523803711E-002 >>>>> -.-.-.-.- >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>> Date: 2018-04-09 08:39:52 -0500 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>> Apr 9 16:50:21 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>> [0]PETSC ERROR: #22 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #23 PCSetUp() line 923 in >>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: #24 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #25 KSPSolve() line 612 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> PETSC_Solve_Sol: 1.2183427810668945E-002 >>>>> PETSC_Solve_SSG: 1.2410402297973633E-002 >>>>> -.-.-.-.- >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>> Date: 2018-04-09 08:39:52 -0500 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>> Apr 9 16:50:21 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>> [0]PETSC ERROR: #26 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #27 PCSetUp() line 923 in >>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: #28 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #29 KSPSolve() line 612 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> PETSC_Solve_Sol: 1.2546300888061523E-002 >>>>> PETSC_Solve_SSG: 1.2778043746948242E-002 >>>>> -.-.-.-.- >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>> Date: 2018-04-09 08:39:52 -0500 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>> Apr 9 16:50:21 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>> [0]PETSC ERROR: #30 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #31 PCSetUp() line 923 in >>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: #32 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #33 KSPSolve() line 612 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> PETSC_Solve_Sol: 1.2255668640136719E-002 >>>>> PETSC_Solve_SSG: 1.2504816055297852E-002 >>>>> -.-.-.-.- >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>> Date: 2018-04-09 08:39:52 -0500 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>> Apr 9 16:50:21 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>> [0]PETSC ERROR: #34 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #35 PCSetUp() line 923 in >>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: #36 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #37 KSPSolve() line 612 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> PETSC_Solve_Sol: 1.3138055801391602E-002 >>>>> PETSC_Solve_SSG: 1.3362646102905273E-002 >>>>> -.-.-.-.- >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>> Date: 2018-04-09 08:39:52 -0500 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>> Apr 9 16:50:21 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>> [0]PETSC ERROR: #38 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #39 PCSetUp() line 923 in >>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: #40 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #41 KSPSolve() line 612 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> PETSC_Solve_Sol: 1.2072801589965820E-002 >>>>> PETSC_Solve_SSG: 1.2319087982177734E-002 >>>>> -.-.-.-.- >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>> Date: 2018-04-09 08:39:52 -0500 >>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>> Apr 9 16:50:21 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>> [0]PETSC ERROR: #42 PCSetUp_SAVIENNACL() line 47 in >>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>> [0]PETSC ERROR: #43 PCSetUp() line 923 in >>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: #44 KSPSetUp() line 381 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #45 KSPSolve() line 612 in >>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>> PETSC_Solve_Sol: 1.2498617172241211E-002 >>>>> PETSC_Solve_SSG: 1.2726306915283203E-002 >>>>> -.-.-.-.- >>>>> -.-.-.-.-.- >>>>> TMain_Loop: 1.1020996570587158 >>>>> -.-.-.-.-.- >>>>> WARNING! There are options you set that were not used! >>>>> WARNING! could be spelling mistake, etc! >>>>> >>>>> >>>>> On Mon, Apr 9, 2018 at 4:45 PM, Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Mon, Apr 9, 2018 at 7:27 PM, Manuel Valera >>>>>> wrote: >>>>>> >>>>>>> On Mon, Apr 9, 2018 at 4:09 PM, Matthew Knepley >>>>>>> wrote: >>>>>>> >>>>>>>> On Mon, Apr 9, 2018 at 6:12 PM, Manuel Valera >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hello guys, >>>>>>>>> >>>>>>>>> I've made advances in my CUDA acceleration project, as you >>>>>>>>> remember i have a CFD model in need of better execution times. >>>>>>>>> >>>>>>>>> So far i have been able to solve the pressure system in the GPU >>>>>>>>> and the rest in serial, using PETSc only for this pressure solve, the >>>>>>>>> library i got to work was ViennaCL. First question, do i still have to >>>>>>>>> switch installations to use either CUDA library? this was a suggestion >>>>>>>>> before, so in order to use CUSP instead of ViennaCL, for example, i >>>>>>>>> currently have to change installations, is this still the case? >>>>>>>>> >>>>>>>> >>>>>>>> I am not sure what that means exactly. However, you can build a >>>>>>>> PETSc with CUDA and ViennaCL support. The type of Vec/Mat is selected at >>>>>>>> runtime. >>>>>>>> >>>>>>> >>>>>>> Karl Rupp wrote in a previous email: >>>>>>> >>>>>>> * * Right now only one of {native CUDA, CUSP, ViennaCL} can be >>>>>>> activated at configure time. This will be fixed later this month.* >>>>>>> >>>>>>> I was asking if this was already solved in 3.9, >>>>>>> >>>>>> >>>>>> Karl knows better than I do. I thought that was fixed, but maybe not >>>>>> in this release. >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Now, i started working in a fully parallelized version of the >>>>>>>>> model, which uses the DMs and DMDAs to distribute the arrays, if i try the >>>>>>>>> same flags as before i get an error saying "Currently only handles ViennaCL >>>>>>>>> matrices" when trying to solve for pressure, i get this is a feature still >>>>>>>>> not implemented? what options do i have to solve pressure, or assign a DMDA >>>>>>>>> array update to be done specifically in a GPU device? >>>>>>>>> >>>>>>>> >>>>>>>> If we can't see the error, we are just guessing. Please send the >>>>>>>> entire error message. >>>>>>>> >>>>>>> >>>>>>> Got it, I will paste the error at the end of this email >>>>>>> >>>>>> >>>>>> It is asking for a ViennaCL matrix. You must tell the DM to create >>>>>> one: >>>>>> >>>>>> -dm_mat_type aijviennacl >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Note, we only do linear algebra on the GPU, so none of the >>>>>>>> FormFunction/FormJacobian stuff for DMDA would be on the GPU. >>>>>>>> >>>>>>> >>>>>>> Yes, we only use it for linear algebra, e.g. solving a linear system >>>>>>> and updating an array with a problematic algorithm. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> I was thinking of using the VecScatterCreateToZero for a regular >>>>>>>>> vector, >>>>>>>>> >>>>>>>> >>>>>>>> Why do you want a serial vector? >>>>>>>> >>>>>>> >>>>>>> Because it looks live ViennaCL doesn't handle arrays created with >>>>>>> DMDAVec, it was just an idea >>>>>>> >>>>>> >>>>>> No, it just needs the right type. >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> but then i would have to create a vector and copy the DMDAVec into >>>>>>>>> it, >>>>>>>>> >>>>>>>> >>>>>>>> I do not understand what it means to copy the DM into the Vec. >>>>>>>> >>>>>>> >>>>>>> I meant copying a DMDAVec into a Vec object, the first is created >>>>>>> with a DMDA object for it's mapping across processors, >>>>>>> >>>>>> >>>>>> There is no such thing as a DMDAVec. Everything is just a Vec. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> is this accomplished with DMDAVecGetArrayReadF90 and then just >>>>>>>>> copy? do you think this will generate too much overhead? >>>>>>>>> >>>>>>>>> Thanks so much for your input, >>>>>>>>> >>>>>>>>> Manuel >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> The error happens when trying to use KSPSolve() for a vector made >>>>>>> with DMDAVec routines, the matrix is created without any DMDA routines >>>>>>> >>>>>>> Error: >>>>>>> >>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>> v3.8.4-2418-gd9c423b GIT Date: 2018-04-02 11:59:41 +0200 >>>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr >>>>>>> 9 16:24:26 2018 >>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>>>> --download-cusp >>>>>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>>> [0]PETSC ERROR: #2 PCSetUp() line 924 in >>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>>> [0]PETSC ERROR: #3 KSPSetUp() line 381 in >>>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>> v3.8.4-2418-gd9c423b GIT Date: 2018-04-02 11:59:41 +0200 >>>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr >>>>>>> 9 16:24:26 2018 >>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>>>> --download-cusp >>>>>>> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in >>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>>> Finished setting up matrix objects >>>>>>> Exiting PrepareNetCDF >>>>>>> [0]PETSC ERROR: ------------------------------ >>>>>>> ------------------------------------------ >>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >>>>>>> Violation, probably memory access out of range >>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>>>>> -on_error_attach_debugger >>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >>>>>>> ocumentation/faq.html#valgrind >>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple >>>>>>> Mac OS X to find memory corruption errors >>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, >>>>>>> link, and run >>>>>>> [0]PETSC ERROR: to get more information on the crash. >>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Signal received >>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>> v3.8.4-2418-gd9c423b GIT Date: 2018-04-02 11:59:41 +0200 >>>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr >>>>>>> 9 16:24:26 2018 >>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>>>> --download-cusp >>>>>>> [0]PETSC ERROR: #6 User provided function() line 0 in unknown file >>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >>>>>>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 9 20:40:20 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Apr 2018 21:40:20 -0400 Subject: [petsc-users] Cuda libraries and DMDA In-Reply-To: References: Message-ID: On Mon, Apr 9, 2018 at 9:34 PM, Manuel Valera wrote: > It looks like it's working now :) > > i needed to setup the DMMatrix and that made the trick, > Glad its working. Thanks, Matt > Thanks, > > > > On Mon, Apr 9, 2018 at 6:13 PM, Manuel Valera wrote: > >> Oh ok, thanks Matt, >> >> I think the problem is that i am not using DMCreateMatrix at all but a >> regular MatCreate, i will follow ex44f and get back to you, >> >> I am calling DMSetFromOptions right after creating the DMDA, way before >> the matrix is created, >> >> >> >> On Mon, Apr 9, 2018 at 5:57 PM, Matthew Knepley >> wrote: >> >>> On Mon, Apr 9, 2018 at 7:55 PM, Manuel Valera >>> wrote: >>> >>>> On Mon, Apr 9, 2018 at 4:53 PM, Matthew Knepley >>>> wrote: >>>> >>>>> On Mon, Apr 9, 2018 at 7:52 PM, Manuel Valera >>>>> wrote: >>>>> >>>>>> Ok thanks, i'm learning more every day, >>>>>> >>>>>> I still get the same error, i am running with -dm_vec_type viennacl >>>>>> -dm_mat_type aijviennacl -pc_type saviennacl >>>>>> >>>>> >>>>> 1) Are you calling DMSetFromOptions()? >>>>> >>>> >>>> Yes, i get: >>>> >>> >>> Let me be more precise. For the DMDA you are using, you must call >>> DMSetFromOptions() before you call >>> DMCreateMatrix (or before its called automatically), in order to get the >>> Mat type you want from the command line. >>> If you do not get the right type, you know this has not happened. >>> >>> Matt >>> >>> >>>> [valera at node50 Src]$ grep DMSetFromOptions *.f90 >>>> DMDAmetrics.f90:call DMSetFromOptions(daDerivs,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daGrid,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daSingle,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daConstants,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daSgs,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daLaplacian,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daMetrics,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daCenters,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daPressureCoeffs,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daDivSgs,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daDummy,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daVelocities,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daScalars,ierr) >>>> dmdaobjs.f90: call DMSetFromOptions(daDensity,ierr) >>>> fileIO.f90: call DMSetFromOptions(daWriteCenters,ierr) >>>> fileIO.f90: call DMSetFromOptions(daWriteUgrid,ierr) >>>> fileIO.f90: call DMSetFromOptions(daWriteVgrid,ierr) >>>> fileIO.f90: call DMSetFromOptions(daWriteWgrid,ierr) >>>> romsmodule.f90: call DMSetFromOptions(daSeqEastWest_u,ierrp) >>>> romsmodule.f90: call DMSetFromOptions(daSeqEastWest_v,ierrp) >>>> romsmodule.f90: call DMSetFromOptions(daSeqEastWest_w,ierrp) >>>> seamountbeamroms.f90: call DMSetFromOptions(daMinPlaneZ,i >>>> err) >>>> >>>> >>>>> >>>>> 2) Does your DM have a prefix? >>>>> >>>> >>>> What does this mean? the one used in the KSPSolve is daDummy >>>> >>>> Thanks, >>>> >>>> >>>> >>>> >>>>> >>>>> Matt >>>>> >>>>> >>>>>> The error: >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>>> Date: 2018-04-09 08:39:52 -0500 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>>> Apr 9 16:50:21 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #2 PCSetUp() line 923 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #3 KSPSetUp() line 381 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>>> Date: 2018-04-09 08:39:52 -0500 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>>> Apr 9 16:50:21 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>>> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #5 PCSetUp() line 923 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> Finished setting up matrix objects >>>>>> Exiting PrepareNetCDF >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>>> Date: 2018-04-09 08:39:52 -0500 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>>> Apr 9 16:50:21 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>>> [0]PETSC ERROR: #6 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #7 PCSetUp() line 923 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #8 KSPSetUp() line 381 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #9 KSPSolve() line 612 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> PETSC_Solve_Sol: 1.7096996307373047E-002 >>>>>> PETSC_Solve_SSG: 1.7321825027465820E-002 >>>>>> -.-.-.-.- >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>>> Date: 2018-04-09 08:39:52 -0500 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>>> Apr 9 16:50:21 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>>> [0]PETSC ERROR: #10 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #11 PCSetUp() line 923 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #12 KSPSetUp() line 381 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #13 KSPSolve() line 612 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> PETSC_Solve_Sol: 1.2702941894531250E-002 >>>>>> PETSC_Solve_SSG: 1.2929439544677734E-002 >>>>>> -.-.-.-.- >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>>> Date: 2018-04-09 08:39:52 -0500 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>>> Apr 9 16:50:21 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>>> [0]PETSC ERROR: #14 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #15 PCSetUp() line 923 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #16 KSPSetUp() line 381 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #17 KSPSolve() line 612 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> PETSC_Solve_Sol: 1.2219429016113281E-002 >>>>>> PETSC_Solve_SSG: 1.2480735778808594E-002 >>>>>> -.-.-.-.- >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>>> Date: 2018-04-09 08:39:52 -0500 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>>> Apr 9 16:50:21 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>>> [0]PETSC ERROR: #18 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #19 PCSetUp() line 923 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #20 KSPSetUp() line 381 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #21 KSPSolve() line 612 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> PETSC_Solve_Sol: 1.2894392013549805E-002 >>>>>> PETSC_Solve_SSG: 1.3124227523803711E-002 >>>>>> -.-.-.-.- >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>>> Date: 2018-04-09 08:39:52 -0500 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>>> Apr 9 16:50:21 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>>> [0]PETSC ERROR: #22 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #23 PCSetUp() line 923 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #24 KSPSetUp() line 381 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #25 KSPSolve() line 612 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> PETSC_Solve_Sol: 1.2183427810668945E-002 >>>>>> PETSC_Solve_SSG: 1.2410402297973633E-002 >>>>>> -.-.-.-.- >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>>> Date: 2018-04-09 08:39:52 -0500 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>>> Apr 9 16:50:21 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>>> [0]PETSC ERROR: #26 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #27 PCSetUp() line 923 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #28 KSPSetUp() line 381 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #29 KSPSolve() line 612 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> PETSC_Solve_Sol: 1.2546300888061523E-002 >>>>>> PETSC_Solve_SSG: 1.2778043746948242E-002 >>>>>> -.-.-.-.- >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>>> Date: 2018-04-09 08:39:52 -0500 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>>> Apr 9 16:50:21 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>>> [0]PETSC ERROR: #30 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #31 PCSetUp() line 923 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #32 KSPSetUp() line 381 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #33 KSPSolve() line 612 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> PETSC_Solve_Sol: 1.2255668640136719E-002 >>>>>> PETSC_Solve_SSG: 1.2504816055297852E-002 >>>>>> -.-.-.-.- >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>>> Date: 2018-04-09 08:39:52 -0500 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>>> Apr 9 16:50:21 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>>> [0]PETSC ERROR: #34 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #35 PCSetUp() line 923 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #36 KSPSetUp() line 381 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #37 KSPSolve() line 612 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> PETSC_Solve_Sol: 1.3138055801391602E-002 >>>>>> PETSC_Solve_SSG: 1.3362646102905273E-002 >>>>>> -.-.-.-.- >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>>> Date: 2018-04-09 08:39:52 -0500 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>>> Apr 9 16:50:21 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>>> [0]PETSC ERROR: #38 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #39 PCSetUp() line 923 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #40 KSPSetUp() line 381 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #41 KSPSolve() line 612 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> PETSC_Solve_Sol: 1.2072801589965820E-002 >>>>>> PETSC_Solve_SSG: 1.2319087982177734E-002 >>>>>> -.-.-.-.- >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT >>>>>> Date: 2018-04-09 08:39:52 -0500 >>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon >>>>>> Apr 9 16:50:21 2018 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>> --download-mpich --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 >>>>>> CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre >>>>>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre >>>>>> --download-viennacl --download-cusp --download-cusp-commit=116b090 >>>>>> [0]PETSC ERROR: #42 PCSetUp_SAVIENNACL() line 47 in >>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>> [0]PETSC ERROR: #43 PCSetUp() line 923 in >>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: #44 KSPSetUp() line 381 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: #45 KSPSolve() line 612 in >>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>> PETSC_Solve_Sol: 1.2498617172241211E-002 >>>>>> PETSC_Solve_SSG: 1.2726306915283203E-002 >>>>>> -.-.-.-.- >>>>>> -.-.-.-.-.- >>>>>> TMain_Loop: 1.1020996570587158 >>>>>> -.-.-.-.-.- >>>>>> WARNING! There are options you set that were not used! >>>>>> WARNING! could be spelling mistake, etc! >>>>>> >>>>>> >>>>>> On Mon, Apr 9, 2018 at 4:45 PM, Matthew Knepley >>>>>> wrote: >>>>>> >>>>>>> On Mon, Apr 9, 2018 at 7:27 PM, Manuel Valera >>>>>>> wrote: >>>>>>> >>>>>>>> On Mon, Apr 9, 2018 at 4:09 PM, Matthew Knepley >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Mon, Apr 9, 2018 at 6:12 PM, Manuel Valera >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hello guys, >>>>>>>>>> >>>>>>>>>> I've made advances in my CUDA acceleration project, as you >>>>>>>>>> remember i have a CFD model in need of better execution times. >>>>>>>>>> >>>>>>>>>> So far i have been able to solve the pressure system in the GPU >>>>>>>>>> and the rest in serial, using PETSc only for this pressure solve, the >>>>>>>>>> library i got to work was ViennaCL. First question, do i still have to >>>>>>>>>> switch installations to use either CUDA library? this was a suggestion >>>>>>>>>> before, so in order to use CUSP instead of ViennaCL, for example, i >>>>>>>>>> currently have to change installations, is this still the case? >>>>>>>>>> >>>>>>>>> >>>>>>>>> I am not sure what that means exactly. However, you can build a >>>>>>>>> PETSc with CUDA and ViennaCL support. The type of Vec/Mat is selected at >>>>>>>>> runtime. >>>>>>>>> >>>>>>>> >>>>>>>> Karl Rupp wrote in a previous email: >>>>>>>> >>>>>>>> * * Right now only one of {native CUDA, CUSP, ViennaCL} can be >>>>>>>> activated at configure time. This will be fixed later this month.* >>>>>>>> >>>>>>>> I was asking if this was already solved in 3.9, >>>>>>>> >>>>>>> >>>>>>> Karl knows better than I do. I thought that was fixed, but maybe not >>>>>>> in this release. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Now, i started working in a fully parallelized version of the >>>>>>>>>> model, which uses the DMs and DMDAs to distribute the arrays, if i try the >>>>>>>>>> same flags as before i get an error saying "Currently only handles ViennaCL >>>>>>>>>> matrices" when trying to solve for pressure, i get this is a feature still >>>>>>>>>> not implemented? what options do i have to solve pressure, or assign a DMDA >>>>>>>>>> array update to be done specifically in a GPU device? >>>>>>>>>> >>>>>>>>> >>>>>>>>> If we can't see the error, we are just guessing. Please send the >>>>>>>>> entire error message. >>>>>>>>> >>>>>>>> >>>>>>>> Got it, I will paste the error at the end of this email >>>>>>>> >>>>>>> >>>>>>> It is asking for a ViennaCL matrix. You must tell the DM to create >>>>>>> one: >>>>>>> >>>>>>> -dm_mat_type aijviennacl >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Note, we only do linear algebra on the GPU, so none of the >>>>>>>>> FormFunction/FormJacobian stuff for DMDA would be on the GPU. >>>>>>>>> >>>>>>>> >>>>>>>> Yes, we only use it for linear algebra, e.g. solving a linear >>>>>>>> system and updating an array with a problematic algorithm. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> I was thinking of using the VecScatterCreateToZero for a regular >>>>>>>>>> vector, >>>>>>>>>> >>>>>>>>> >>>>>>>>> Why do you want a serial vector? >>>>>>>>> >>>>>>>> >>>>>>>> Because it looks live ViennaCL doesn't handle arrays created with >>>>>>>> DMDAVec, it was just an idea >>>>>>>> >>>>>>> >>>>>>> No, it just needs the right type. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> but then i would have to create a vector and copy the DMDAVec into >>>>>>>>>> it, >>>>>>>>>> >>>>>>>>> >>>>>>>>> I do not understand what it means to copy the DM into the Vec. >>>>>>>>> >>>>>>>> >>>>>>>> I meant copying a DMDAVec into a Vec object, the first is created >>>>>>>> with a DMDA object for it's mapping across processors, >>>>>>>> >>>>>>> >>>>>>> There is no such thing as a DMDAVec. Everything is just a Vec. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> is this accomplished with DMDAVecGetArrayReadF90 and then just >>>>>>>>>> copy? do you think this will generate too much overhead? >>>>>>>>>> >>>>>>>>>> Thanks so much for your input, >>>>>>>>>> >>>>>>>>>> Manuel >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> The error happens when trying to use KSPSolve() for a vector made >>>>>>>> with DMDAVec routines, the matrix is created without any DMDA routines >>>>>>>> >>>>>>>> Error: >>>>>>>> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>> v3.8.4-2418-gd9c423b GIT Date: 2018-04-02 11:59:41 +0200 >>>>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr >>>>>>>> 9 16:24:26 2018 >>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>>>>> --download-cusp >>>>>>>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>>>> [0]PETSC ERROR: #2 PCSetUp() line 924 in >>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>>>> [0]PETSC ERROR: #3 KSPSetUp() line 381 in >>>>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>>>>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>> v3.8.4-2418-gd9c423b GIT Date: 2018-04-02 11:59:41 +0200 >>>>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr >>>>>>>> 9 16:24:26 2018 >>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>>>>> --download-cusp >>>>>>>> [0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in >>>>>>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>>>>>> [0]PETSC ERROR: #5 PCSetUp() line 924 in >>>>>>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>>>>>> Finished setting up matrix objects >>>>>>>> Exiting PrepareNetCDF >>>>>>>> [0]PETSC ERROR: ------------------------------ >>>>>>>> ------------------------------------------ >>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >>>>>>>> Violation, probably memory access out of range >>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>>>>>> -on_error_attach_debugger >>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >>>>>>>> ocumentation/faq.html#valgrind >>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple >>>>>>>> Mac OS X to find memory corruption errors >>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, >>>>>>>> link, and run >>>>>>>> [0]PETSC ERROR: to get more information on the crash. >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: Signal received >>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>> v3.8.4-2418-gd9c423b GIT Date: 2018-04-02 11:59:41 +0200 >>>>>>>> [0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr >>>>>>>> 9 16:24:26 2018 >>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich >>>>>>>> --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 >>>>>>>> --with-shared-libraries=1 --download-hypre --with-debugging=no >>>>>>>> --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl >>>>>>>> --download-cusp >>>>>>>> [0]PETSC ERROR: #6 User provided function() line 0 in unknown file >>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >>>>>>>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rupp at iue.tuwien.ac.at Mon Apr 9 23:51:45 2018 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Tue, 10 Apr 2018 06:51:45 +0200 Subject: [petsc-users] [SPAM?] Re: Cuda libraries and DMDA In-Reply-To: References: Message-ID: <779bd6ed-4301-cce4-a518-4916e3581044@iue.tuwien.ac.at> Hi, > Karl Rupp wrote in a previous email: > > /?* Right now only one of {native CUDA, CUSP, ViennaCL} can be activated > at configure time. This will be fixed later this month./ > / > / > I was asking if this was already solved in 3.9, Yes, this was resolved by allowing native CUDA and ViennaCL to be used concurrently. CUSP has been removed for the release and is not available at the moment. Best regards, Karli From rlmackie862 at gmail.com Tue Apr 10 08:05:07 2018 From: rlmackie862 at gmail.com (Randall Mackie) Date: Tue, 10 Apr 2018 15:05:07 +0200 Subject: [petsc-users] PETSc 3.9 release In-Reply-To: <45E3843D-DFCE-4F3F-97B5-BE4BABACADB1@mcs.anl.gov> References: <45E3843D-DFCE-4F3F-97B5-BE4BABACADB1@mcs.anl.gov> Message-ID: <3D4D1F07-2009-45B7-9BBC-C1393A207485@gmail.com> Can you explain how VECNODE is different from normal vectors and where it might be useful? Thanks, Randy M. > On Apr 9, 2018, at 11:22 PM, Smith, Barry F. wrote: > > > PETSc users > > We are pleased to announce the release of PETSc version 3.9 at http://www.mcs.anl.gov/petsc > > The major changes and updates can be found at http://www.mcs.anl.gov/petsc/documentation/changes/39.html > > We recommend upgrading to PETSc 3.9 soon (the sooner you switch the more we can help you with the switch). > > As always, please report problems to petsc-maint at mcs.anl.gov and ask questions at petsc-users at mcs.anl.gov > > This release includes contributions from > > Adrian Croucher > Alejandro Lamas Davi?a > Alp Dener > Barry Smith > Bernhard M. Wiedemann > Blaise Bourdin > Daniel Kokron > Dave May > David Wobrock > eaulisa > Fande Kong > Hong Zhang > Hong Zhang > Jakub Kruzik > Jed Brown > Jose E. Roman > Junchao Zhang > Karl Rupp > Lawrence Mitchell > Lisandro Dalcin > Mari Pecha > Mark Adams > Mark Lohry > Matthew G. Knepley > Nicolas Barral > Oana Marin > Patrick Sanan > Pieter Ghysels > Richard Tran Mills > Satish Balay > Scott Kruger > Shri Abhyankar > Siegfried Cools > Stefano Zampini > Thomas Mijieux > Toby Isaac > Todd Munson > Tristan Konolige > V?clav Hapla > > and bug reports/patches/proposed improvements received from > > Adrian Croucher > Albert Cowie > "Alberto F. Mart?n" > Alp Dener > Antonio Trande > apostol > Barry Smith > "Charles A.(Sandy) Mader" > Daniele Tartarini > Danyang Su > Drew Parsons > Ed Bueler > Eric Chamberland > Eugenio Aulisa > Evan Um > Fabrice JURADO > Fadi El Haddad > Fande Kong > Febrian Setianto > Florian Lindner > Franck Houssen > Francois Pechereau > Garth N. Wells > GONG Ding > Greg Meyer > Henrik Buesing > Hong Zhang > Hong Zhang > ilya > Jaroslaw Piwonski > Jean Philippe Fran?ois > Jed Brown > Jin Chen > J?rgen Dokken > Jose E. Roman > Karl Rupp > Keith Lindsay > "Klaij, Christiaan" > Lawrence Mitchell > Lisandro Dalcin > Manav Bhatia > Marco Schauer > Marius Buerkle > Mark Adams > Martin Diehl > Matthew Knepley > Matthieu Vitse > Michael Legu?be > "Miller, Laniece E." > Oana Marin > Pablo Brubeck > Patrick Farrell > Patrick Sanan > Peter Meisrimel > Philip Eisenlohr > Pierre Jolivet > Praveen C > Rafael Monteiro da Silva > Randall Mackie > Randy Michael Churchill > Richard Tran Mills > Roland Richter > Samuel Lanthaler > Sanjay Govindjee > Satish Balay > Serge Van Criekingen > Stefano Zampini > Stephan Kramer > Syam Vangara > Tim Steinhoff > Tobias Goerler > Todd Munson > Tzanio Kolev > Victor Eijkhout > Vinh Pham-Xuan > Xiangdong > Xiaoye S. Li > Yiting Wang > > As always, thanks for your support, > > Barry > From dayedut123 at 163.com Tue Apr 10 08:42:19 2018 From: dayedut123 at 163.com (=?GBK?B?ztI=?=) Date: Tue, 10 Apr 2018 21:42:19 +0800 (CST) Subject: [petsc-users] Change the matrix row order Message-ID: <8c41787.11671b0.162afcb0256.Coremail.dayedut123@163.com> Hello, I want to change the row order of a sparse matrix. Are there any build-in functions and suggestions? Thank you very much! Daye -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 10 10:07:04 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 10 Apr 2018 15:07:04 +0000 Subject: [petsc-users] Change the matrix row order In-Reply-To: <8c41787.11671b0.162afcb0256.Coremail.dayedut123@163.com> References: <8c41787.11671b0.162afcb0256.Coremail.dayedut123@163.com> Message-ID: <88747128-D9EE-48A2-9E48-36399373CA3E@anl.gov> When and where do you want to change the row order? For PETSc and external factorization based solvers this is handled automatically, no need to do anything. The ordering of matrix rows corresponds to the ordering of the vectors; almost always one partitions and orders the vectors based on the mesh that underlies the problem and there is no reason to reorder once the layout is initially determined. Barry > On Apr 10, 2018, at 8:42 AM, ? wrote: > > Hello, > I want to change the row order of a sparse matrix. Are there any build-in functions and suggestions? > Thank you very much! > Daye > > > From hzhang at mcs.anl.gov Tue Apr 10 11:32:30 2018 From: hzhang at mcs.anl.gov (Hong) Date: Tue, 10 Apr 2018 11:32:30 -0500 Subject: [petsc-users] PETSc 3.9 release In-Reply-To: <3D4D1F07-2009-45B7-9BBC-C1393A207485@gmail.com> References: <45E3843D-DFCE-4F3F-97B5-BE4BABACADB1@mcs.anl.gov> <3D4D1F07-2009-45B7-9BBC-C1393A207485@gmail.com> Message-ID: Randall : > Can you explain how VECNODE is different from normal vectors and where it > might be useful? > Vector uses on-node shared memory to store its entries. http://www.mcs.anl.gov/petsc/documentation/changes/39.html This is an experimental work for using mpi3 support of shared memory. The vector entries are stored in shared memory, allowing direct access by on-node cores. You must use it with VECSCATTERMPI3NODE. where is might be useful? It was developed as one algorithmic approach of mpi3 vecscatter. For some tests, it accelerates vecscatter, but we have not seen practical impact to Mat-vec operations. Let us know if anyone is interested in using our new mpi3 vector scatters. Hong > > > On Apr 9, 2018, at 11:22 PM, Smith, Barry F. wrote: > > > > > > PETSc users > > > > We are pleased to announce the release of PETSc version 3.9 at > http://www.mcs.anl.gov/petsc > > > > The major changes and updates can be found at > http://www.mcs.anl.gov/petsc/documentation/changes/39.html > > > > We recommend upgrading to PETSc 3.9 soon (the sooner you switch the more > we can help you with the switch). > > > > As always, please report problems to petsc-maint at mcs.anl.gov and ask > questions at petsc-users at mcs.anl.gov > > > > This release includes contributions from > > > > Adrian Croucher > > Alejandro Lamas Davi?a > > Alp Dener > > Barry Smith > > Bernhard M. Wiedemann > > Blaise Bourdin > > Daniel Kokron > > Dave May > > David Wobrock > > eaulisa > > Fande Kong > > Hong Zhang > > Hong Zhang > > Jakub Kruzik > > Jed Brown > > Jose E. Roman > > Junchao Zhang > > Karl Rupp > > Lawrence Mitchell > > Lisandro Dalcin > > Mari Pecha > > Mark Adams > > Mark Lohry > > Matthew G. Knepley > > Nicolas Barral > > Oana Marin > > Patrick Sanan > > Pieter Ghysels > > Richard Tran Mills > > Satish Balay > > Scott Kruger > > Shri Abhyankar > > Siegfried Cools > > Stefano Zampini > > Thomas Mijieux > > Toby Isaac > > Todd Munson > > Tristan Konolige > > V?clav Hapla > > > > and bug reports/patches/proposed improvements received from > > > > Adrian Croucher > > Albert Cowie > > "Alberto F. Mart?n" > > Alp Dener > > Antonio Trande > > apostol > > Barry Smith > > "Charles A.(Sandy) Mader" > > Daniele Tartarini > > Danyang Su > > Drew Parsons > > Ed Bueler > > Eric Chamberland > > Eugenio Aulisa > > Evan Um > > Fabrice JURADO > > Fadi El Haddad > > Fande Kong > > Febrian Setianto > > Florian Lindner > > Franck Houssen > > Francois Pechereau > > Garth N. Wells > > GONG Ding > > Greg Meyer > > Henrik Buesing > > Hong Zhang > > Hong Zhang > > ilya > > Jaroslaw Piwonski > > Jean Philippe Fran?ois > > Jed Brown > > Jin Chen > > J?rgen Dokken > > Jose E. Roman > > Karl Rupp > > Keith Lindsay > > "Klaij, Christiaan" > > Lawrence Mitchell > > Lisandro Dalcin > > Manav Bhatia > > Marco Schauer > > Marius Buerkle > > Mark Adams > > Martin Diehl > > Matthew Knepley > > Matthieu Vitse > > Michael Legu?be > > "Miller, Laniece E." > > Oana Marin > > Pablo Brubeck > > Patrick Farrell > > Patrick Sanan > > Peter Meisrimel > > Philip Eisenlohr > > Pierre Jolivet > > Praveen C > > Rafael Monteiro da Silva > > Randall Mackie > > Randy Michael Churchill > > Richard Tran Mills > > Roland Richter > > Samuel Lanthaler > > Sanjay Govindjee > > Satish Balay > > Serge Van Criekingen > > Stefano Zampini > > Stephan Kramer > > Syam Vangara > > Tim Steinhoff > > Tobias Goerler > > Todd Munson > > Tzanio Kolev > > Victor Eijkhout > > Vinh Pham-Xuan > > Xiangdong > > Xiaoye S. Li > > Yiting Wang > > > > As always, thanks for your support, > > > > Barry > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernardomartinsrocha at gmail.com Tue Apr 10 13:19:29 2018 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Tue, 10 Apr 2018 15:19:29 -0300 Subject: [petsc-users] PETSc version 3.9 with Fortran and DMPlex questions Message-ID: Hi everyone, I have two questions. (Q1) The first one is related to the new PETSc 3.9. I've been using Fortran and using the "userctx" to wrap some data for the function calls within my program. Just like : call func(...,userctx) and retrieving data as: PetscFortranAddr userctx(*) b = userctx(2) With the new version I got the following error: b = userctx(2) Error: Can't convert INTEGER(8) to TYPE(tvec) at (1) Is there a way to keep using the trick with "userctx", instead of passing all the arguments to the functions separately? (Q2) The second is related to DMPlex. When I use the CreateBoxMesh with 2x1x1 cells, I get the expected output: DM Object: 1 MPI processes type: plex DM_0x84000000_0 in 3 dimensions: 0-cells: 12 * 1-cells: 20* *2-cells: 11* 3-cells: 2 Labels: depth: 4 strata with value/size (0 (12), 1 (20), 2 (11), 3 (2)) However, If I try to create that manually to reproduce the same case 2x1x1 cells (creating this simple element connectivity by hand) using DMPlexCreateFromCellList I get the following: DM Object: DM_0x84000000_0 1 MPI processes type: plex DM_0x84000000_0 in 3 dimensions: 0-cells: 12 *1-cells: 22* * 2-cells: 12* 3-cells: 2 Labels: depth: 4 strata with value/size (0 (12), 1 (22), 2 (12), 3 (2)) That is, it seems the shared face between the two cells is counted twice. Is that right? What am I missing here? Best regards, Bernardo -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernardomartinsrocha at gmail.com Tue Apr 10 13:31:19 2018 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Tue, 10 Apr 2018 15:31:19 -0300 Subject: [petsc-users] PETSc version 3.9 with Fortran and DMPlex questions In-Reply-To: References: Message-ID: Found my way to the answer of the first question: ksp/ksp/examples/tutorials/ex13f90.F90 With respect to question 2... I'm still confused about this behaviour of the DMPlex; or I am missing something. Best regards. Bernardo On Tue, Apr 10, 2018 at 3:19 PM, Bernardo Rocha < bernardomartinsrocha at gmail.com> wrote: > Hi everyone, > > I have two questions. > > (Q1) > The first one is related to the new PETSc 3.9. > I've been using Fortran and using the "userctx" to wrap some data > for the function calls within my program. > > Just like : call func(...,userctx) > and retrieving data as: > PetscFortranAddr userctx(*) > b = userctx(2) > > With the new version I got the following error: > b = userctx(2) > Error: Can't convert INTEGER(8) to TYPE(tvec) at (1) > > Is there a way to keep using the trick with "userctx", instead of passing > all the arguments to the functions separately? > > (Q2) > The second is related to DMPlex. > When I use the CreateBoxMesh with 2x1x1 cells, I get the expected output: > > DM Object: 1 MPI processes > type: plex > DM_0x84000000_0 in 3 dimensions: > 0-cells: 12 > * 1-cells: 20* > *2-cells: 11* > 3-cells: 2 > Labels: > depth: 4 strata with value/size (0 (12), 1 (20), 2 (11), 3 (2)) > > However, If I try to create that manually to reproduce the same case 2x1x1 > cells (creating this simple element connectivity by hand) using > DMPlexCreateFromCellList I get the following: > > DM Object: DM_0x84000000_0 1 MPI processes > type: plex > DM_0x84000000_0 in 3 dimensions: > 0-cells: 12 > *1-cells: 22* > * 2-cells: 12* > 3-cells: 2 > Labels: > depth: 4 strata with value/size (0 (12), 1 (22), 2 (12), 3 (2)) > > That is, it seems the shared face between the two cells is counted twice. > Is that right? What am I missing here? > > Best regards, > Bernardo > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 10 15:14:52 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 10 Apr 2018 16:14:52 -0400 Subject: [petsc-users] PETSc version 3.9 with Fortran and DMPlex questions In-Reply-To: References: Message-ID: On Tue, Apr 10, 2018 at 2:19 PM, Bernardo Rocha < bernardomartinsrocha at gmail.com> wrote: > Hi everyone, > > I have two questions. > > (Q1) > The first one is related to the new PETSc 3.9. > I've been using Fortran and using the "userctx" to wrap some data > for the function calls within my program. > > Just like : call func(...,userctx) > and retrieving data as: > PetscFortranAddr userctx(*) > b = userctx(2) > > With the new version I got the following error: > b = userctx(2) > Error: Can't convert INTEGER(8) to TYPE(tvec) at (1) > > Is there a way to keep using the trick with "userctx", instead of passing > all the arguments to the functions separately? > > (Q2) > The second is related to DMPlex. > When I use the CreateBoxMesh with 2x1x1 cells, I get the expected output: > > DM Object: 1 MPI processes > type: plex > DM_0x84000000_0 in 3 dimensions: > 0-cells: 12 > * 1-cells: 20* > *2-cells: 11* > 3-cells: 2 > Labels: > depth: 4 strata with value/size (0 (12), 1 (20), 2 (11), 3 (2)) > > However, If I try to create that manually to reproduce the same case 2x1x1 > cells (creating this simple element connectivity by hand) using > DMPlexCreateFromCellList I get the following: > > DM Object: DM_0x84000000_0 1 MPI processes > type: plex > DM_0x84000000_0 in 3 dimensions: > 0-cells: 12 > *1-cells: 22* > * 2-cells: 12* > 3-cells: 2 > Labels: > depth: 4 strata with value/size (0 (12), 1 (22), 2 (12), 3 (2)) > > That is, it seems the shared face between the two cells is counted twice. > Is that right? What am I missing here? > Since the input is cell-vertex, my guess is that the order of the vertices on a cell is not what I expect. I have readers for common formats, but FromCellList() is my own. You can see what vertices I think constitute the faces: https://bitbucket.org/petsc/petsc/src/05d412bc580c652749a091cc13ab756bb8816ec1/src/dm/impls/plex/plexinterpolate.c?at=master&fileviewer=file-view-default#plexinterpolate.c-113 Sorry, this was confusing. Most people are using the input formats, mesh generators, or predefined shapes, so I do not get a lot of feedback on this interface. Thanks, Matt > Best regards, > Bernardo > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Apr 10 16:23:00 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 10 Apr 2018 16:23:00 -0500 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> Message-ID: I tried a few builds with: '--with-64-bit-indices=1', '--with-memalign=64', '--with-blaslapack-dir=/home/intel/18/compilers_and_libraries_2018.0.128/linux/mkl', '--with-cc=icc', '--with-fc=ifort', '--with-cxx=0', '--with-debugging=0', '--with-mpi=0', And then changed the OPTFLAGS: 1. 'basic -g' - works fine 'COPTFLAGS=-g', 'FOPTFLAGS=-g', 2. 'avx512' - works fine 'COPTFLAGS=-g -xMIC-AVX512 -O3', 'FOPTFLAGS=-g -xMIC-AVX512 -O3', 3. 'avx2' - breaks. '--COPTFLAGS=-g -axcore-avx2', '--FOPTFLAGS=-g -axcore-avx2', with a breakpoint at dmdavecrestorearrayf903_() in gdb - I see - the stack is fine during the first call to dmdavecrestorearrayf903_() - but is corrupted when it goes to the second call to dmdavecrestorearrayf903_() i.e ierr=0x7fffffffb4a0 changes to ierr=0x0] >>>>>>>>>> Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030c0 , a=0x401abd , ierr=0x7fffffffb4a0) at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/zda1f90.c:153 153 { (gdb) where #0 dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030c0 , a=0x401abd , ierr=0x7fffffffb4a0) at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/zda1f90.c:153 #1 0x0000000000401abd in test () at ex1f.F90:80 #2 0x00000000004011ae in main () #3 0x00007fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6 #4 0x00000000004010b9 in _start () (gdb) c Continuing. Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030b8 , a=0x401ada , ierr=0x0) at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/zda1f90.c:153 153 { (gdb) where #0 dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030b8 , a=0x401ada , ierr=0x0) at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/zda1f90.c:153 #1 0x0000000000401ada in test () at ex1f.F90:81 #2 0x00000000004011ae in main () #3 0x00007fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6 #4 0x00000000004010b9 in _start () (gdb) >>>>>>>>> Its not clear to me why this happens. [and why it would work with -xMIC-AVX512 but breaks with -axcore-avx2]. Perhaps Richard, Jeff have better insight on this. BTW: The above run is with: bash-4.2$ icc --version icc (ICC) 18.0.0 20170811 Satish On Mon, 9 Apr 2018, Satish Balay wrote: > I'm able to reproduce this problem on knl box [with the attached test code]. But it goes away if I rebuild without the option --with-64-bit-indices. > > Will have to check further.. > > Satish > > > On Thu, 5 Apr 2018, Randall Mackie wrote: > > > Dear PETSc users, > > > > I?m curious if anyone else experiences problems using DMDAVecGetArrayF90 in conjunction with Intel compilers? > > We have had many problems (typically 11 SEGV segmentation violations) when PETSc is compiled in optimize mode (with various combinations of options). > > These same codes run valgrind clean with gfortran, so I assume this is an Intel bug, but before we submit a bug report I wanted to see if anyone else had similar experiences? > > We have basically gone back and replaced our calls to DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers into a ?local? subroutine that works fine. > > > > In case anyone is curious, the attached test code shows this behavior when PETSc is compiled with the following options: > > > > ./configure \ > > --with-clean=1 \ > > --with-debugging=0 \ > > --with-fortran=1 \ > > --with-64-bit-indices \ > > --download-mpich=../mpich-3.3a2.tar.gz \ > > --with-blas-lapack-dir=/opt/intel/mkl \ > > --with-cc=icc \ > > --with-fc=ifort \ > > --with-cxx=icc \ > > --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > > --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > > --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2? \ > > > > > > > > Thanks, Randy M. > > > > > From jeff.science at gmail.com Tue Apr 10 18:12:22 2018 From: jeff.science at gmail.com (Jeff Hammond) Date: Tue, 10 Apr 2018 16:12:22 -0700 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> Message-ID: This should generate an SSE2 binary: 'COPTFLAGS=-g', 'FOPTFLAGS=-g', This should generate a KNL binary: 'COPTFLAGS=-g -xMIC-AVX512 -O3', 'FOPTFLAGS=-g -xMIC-AVX512 -O3', This should generate a SSE2 binary that also supports CORE-AVX2 dispatch. '--COPTFLAGS=-g -axcore-avx2', '--FOPTFLAGS=-g -axcore-avx2', I don't see a good reason for the third option to fail. Please report this bug to Intel. You might also verify that this works: '--COPTFLAGS=-g -xCORE-AVX2', '--FOPTFLAGS=-g -xCORE-AVX2', In general, one should avoid compiling for SSE on KNL, because SSE-AVX transition penalties need to be avoided (google should find the details). Are you trying to generate a single binary that is portable to ancient Core/Xeon and KNL? I recommend that you use AVX (Sandy Bridge) - preferably AVX2 (Haswell) - as your oldest ISA target when generating a portable binary that includes KNL support. Jeff On Tue, Apr 10, 2018 at 2:23 PM, Satish Balay wrote: > I tried a few builds with: > > '--with-64-bit-indices=1', > '--with-memalign=64', > '--with-blaslapack-dir=/home/intel/18/compilers_and_ > libraries_2018.0.128/linux/mkl', > '--with-cc=icc', > '--with-fc=ifort', > '--with-cxx=0', > '--with-debugging=0', > '--with-mpi=0', > > And then changed the OPTFLAGS: > > 1. 'basic -g' - works fine > > 'COPTFLAGS=-g', > 'FOPTFLAGS=-g', > > 2. 'avx512' - works fine > > 'COPTFLAGS=-g -xMIC-AVX512 -O3', > 'FOPTFLAGS=-g -xMIC-AVX512 -O3', > > 3. 'avx2' - breaks. > > '--COPTFLAGS=-g -axcore-avx2', > '--FOPTFLAGS=-g -axcore-avx2', > > with a breakpoint at dmdavecrestorearrayf903_() in gdb - I see - the > stack is fine during the first call to dmdavecrestorearrayf903_() - > but is corrupted when it goes to the second call to > dmdavecrestorearrayf903_() i.e ierr=0x7fffffffb4a0 changes to > ierr=0x0] > > >>>>>>>>>> > > Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 , > v=0x6030c0 , a=0x401abd , > ierr=0x7fffffffb4a0) at /home/petsc/petsc.barry-test/ > src/dm/impls/da/f90-custom/zda1f90.c:153 > 153 { > (gdb) where > #0 dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030c0 > , a=0x401abd , ierr=0x7fffffffb4a0) > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/ > zda1f90.c:153 > #1 0x0000000000401abd in test () at ex1f.F90:80 > #2 0x00000000004011ae in main () > #3 0x00007fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6 > #4 0x00000000004010b9 in _start () > (gdb) c > Continuing. > > Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 , > v=0x6030b8 , a=0x401ada , ierr=0x0) > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/ > zda1f90.c:153 > 153 { > (gdb) where > #0 dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030b8 > , a=0x401ada , ierr=0x0) > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/ > zda1f90.c:153 > #1 0x0000000000401ada in test () at ex1f.F90:81 > #2 0x00000000004011ae in main () > #3 0x00007fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6 > #4 0x00000000004010b9 in _start () > (gdb) > > >>>>>>>>> > > Its not clear to me why this happens. [and why it would work with > -xMIC-AVX512 but breaks with -axcore-avx2]. > > Perhaps Richard, Jeff have better insight on this. > > BTW: The above run is with: > > bash-4.2$ icc --version > icc (ICC) 18.0.0 20170811 > > Satish > > On Mon, 9 Apr 2018, Satish Balay wrote: > > > I'm able to reproduce this problem on knl box [with the attached test > code]. But it goes away if I rebuild without the option > --with-64-bit-indices. > > > > Will have to check further.. > > > > Satish > > > > > > On Thu, 5 Apr 2018, Randall Mackie wrote: > > > > > Dear PETSc users, > > > > > > I?m curious if anyone else experiences problems using > DMDAVecGetArrayF90 in conjunction with Intel compilers? > > > We have had many problems (typically 11 SEGV segmentation violations) > when PETSc is compiled in optimize mode (with various combinations of > options). > > > These same codes run valgrind clean with gfortran, so I assume this is > an Intel bug, but before we submit a bug report I wanted to see if anyone > else had similar experiences? > > > We have basically gone back and replaced our calls to > DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers > into a ?local? subroutine that works fine. > > > > > > In case anyone is curious, the attached test code shows this behavior > when PETSc is compiled with the following options: > > > > > > ./configure \ > > > --with-clean=1 \ > > > --with-debugging=0 \ > > > --with-fortran=1 \ > > > --with-64-bit-indices \ > > > --download-mpich=../mpich-3.3a2.tar.gz \ > > > --with-blas-lapack-dir=/opt/intel/mkl \ > > > --with-cc=icc \ > > > --with-fc=ifort \ > > > --with-cxx=icc \ > > > --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > > > --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > > > --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2? \ > > > > > > > > > > > > Thanks, Randy M. > > > > > > > > -- Jeff Hammond jeff.science at gmail.com http://jeffhammond.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Apr 10 18:39:55 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 10 Apr 2018 18:39:55 -0500 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> Message-ID: On Tue, 10 Apr 2018, Jeff Hammond wrote: > This should generate an SSE2 binary: > > 'COPTFLAGS=-g', > 'FOPTFLAGS=-g', > > This should generate a KNL binary: > > 'COPTFLAGS=-g -xMIC-AVX512 -O3', > 'FOPTFLAGS=-g -xMIC-AVX512 -O3', > > This should generate a SSE2 binary that also supports CORE-AVX2 dispatch. > > '--COPTFLAGS=-g -axcore-avx2', > '--FOPTFLAGS=-g -axcore-avx2', > > I don't see a good reason for the third option to fail. Please report this > bug to Intel. > > You might also verify that this works: > > '--COPTFLAGS=-g -xCORE-AVX2', > '--FOPTFLAGS=-g -xCORE-AVX2', This fails the same way as -axcore-avx2 > > In general, one should avoid compiling for SSE on KNL, because SSE-AVX > transition penalties need to be avoided (google should find the details). > Are you trying to generate a single binary that is portable to ancient > Core/Xeon and KNL? My usage here is to reproduce this issue reported by Randy - assumed the knl box we have is the easiest way.. Satish > I recommend that you use AVX (Sandy Bridge) - > preferably AVX2 (Haswell) - as your oldest ISA target when generating a > portable binary that includes KNL support. > > Jeff > > On Tue, Apr 10, 2018 at 2:23 PM, Satish Balay wrote: > > > I tried a few builds with: > > > > '--with-64-bit-indices=1', > > '--with-memalign=64', > > '--with-blaslapack-dir=/home/intel/18/compilers_and_ > > libraries_2018.0.128/linux/mkl', > > '--with-cc=icc', > > '--with-fc=ifort', > > '--with-cxx=0', > > '--with-debugging=0', > > '--with-mpi=0', > > > > And then changed the OPTFLAGS: > > > > 1. 'basic -g' - works fine > > > > 'COPTFLAGS=-g', > > 'FOPTFLAGS=-g', > > > > 2. 'avx512' - works fine > > > > 'COPTFLAGS=-g -xMIC-AVX512 -O3', > > 'FOPTFLAGS=-g -xMIC-AVX512 -O3', > > > > 3. 'avx2' - breaks. > > > > '--COPTFLAGS=-g -axcore-avx2', > > '--FOPTFLAGS=-g -axcore-avx2', > > > > with a breakpoint at dmdavecrestorearrayf903_() in gdb - I see - the > > stack is fine during the first call to dmdavecrestorearrayf903_() - > > but is corrupted when it goes to the second call to > > dmdavecrestorearrayf903_() i.e ierr=0x7fffffffb4a0 changes to > > ierr=0x0] > > > > >>>>>>>>>> > > > > Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 , > > v=0x6030c0 , a=0x401abd , > > ierr=0x7fffffffb4a0) at /home/petsc/petsc.barry-test/ > > src/dm/impls/da/f90-custom/zda1f90.c:153 > > 153 { > > (gdb) where > > #0 dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030c0 > > , a=0x401abd , ierr=0x7fffffffb4a0) > > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/ > > zda1f90.c:153 > > #1 0x0000000000401abd in test () at ex1f.F90:80 > > #2 0x00000000004011ae in main () > > #3 0x00007fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6 > > #4 0x00000000004010b9 in _start () > > (gdb) c > > Continuing. > > > > Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 , > > v=0x6030b8 , a=0x401ada , ierr=0x0) > > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/ > > zda1f90.c:153 > > 153 { > > (gdb) where > > #0 dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030b8 > > , a=0x401ada , ierr=0x0) > > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/ > > zda1f90.c:153 > > #1 0x0000000000401ada in test () at ex1f.F90:81 > > #2 0x00000000004011ae in main () > > #3 0x00007fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6 > > #4 0x00000000004010b9 in _start () > > (gdb) > > > > >>>>>>>>> > > > > Its not clear to me why this happens. [and why it would work with > > -xMIC-AVX512 but breaks with -axcore-avx2]. > > > > Perhaps Richard, Jeff have better insight on this. > > > > BTW: The above run is with: > > > > bash-4.2$ icc --version > > icc (ICC) 18.0.0 20170811 > > > > Satish > > > > On Mon, 9 Apr 2018, Satish Balay wrote: > > > > > I'm able to reproduce this problem on knl box [with the attached test > > code]. But it goes away if I rebuild without the option > > --with-64-bit-indices. > > > > > > Will have to check further.. > > > > > > Satish > > > > > > > > > On Thu, 5 Apr 2018, Randall Mackie wrote: > > > > > > > Dear PETSc users, > > > > > > > > I?m curious if anyone else experiences problems using > > DMDAVecGetArrayF90 in conjunction with Intel compilers? > > > > We have had many problems (typically 11 SEGV segmentation violations) > > when PETSc is compiled in optimize mode (with various combinations of > > options). > > > > These same codes run valgrind clean with gfortran, so I assume this is > > an Intel bug, but before we submit a bug report I wanted to see if anyone > > else had similar experiences? > > > > We have basically gone back and replaced our calls to > > DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers > > into a ?local? subroutine that works fine. > > > > > > > > In case anyone is curious, the attached test code shows this behavior > > when PETSc is compiled with the following options: > > > > > > > > ./configure \ > > > > --with-clean=1 \ > > > > --with-debugging=0 \ > > > > --with-fortran=1 \ > > > > --with-64-bit-indices \ > > > > --download-mpich=../mpich-3.3a2.tar.gz \ > > > > --with-blas-lapack-dir=/opt/intel/mkl \ > > > > --with-cc=icc \ > > > > --with-fc=ifort \ > > > > --with-cxx=icc \ > > > > --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > > > > --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > > > > --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2? \ > > > > > > > > > > > > > > > > Thanks, Randy M. > > > > > > > > > > > > > > > > From jeff.science at gmail.com Tue Apr 10 18:57:05 2018 From: jeff.science at gmail.com (Jeff Hammond) Date: Tue, 10 Apr 2018 16:57:05 -0700 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> Message-ID: On Tue, Apr 10, 2018 at 4:39 PM, Satish Balay wrote: > On Tue, 10 Apr 2018, Jeff Hammond wrote: > > > This should generate an SSE2 binary: > > > > 'COPTFLAGS=-g', > > 'FOPTFLAGS=-g', > > > > This should generate a KNL binary: > > > > 'COPTFLAGS=-g -xMIC-AVX512 -O3', > > 'FOPTFLAGS=-g -xMIC-AVX512 -O3', > > > > This should generate a SSE2 binary that also supports CORE-AVX2 dispatch. > > > > '--COPTFLAGS=-g -axcore-avx2', > > '--FOPTFLAGS=-g -axcore-avx2', > > > > I don't see a good reason for the third option to fail. Please report > this > > bug to Intel. > > > > You might also verify that this works: > > > > '--COPTFLAGS=-g -xCORE-AVX2', > > '--FOPTFLAGS=-g -xCORE-AVX2', > > This fails the same way as -axcore-avx2 > > Can you try on a non-KNL host? It's a bug either way but I want to determine if KNL host is the issue. > > > > In general, one should avoid compiling for SSE on KNL, because SSE-AVX > > transition penalties need to be avoided (google should find the details). > > Are you trying to generate a single binary that is portable to ancient > > Core/Xeon and KNL? > > My usage here is to reproduce this issue reported by Randy - assumed the > knl box we have is the easiest way.. > > Based only what I see below, Randy doesn't seem to be reporting a KNL-specific issue. Is that incorrect? I strongly recommend generating KNL-specific binaries for KNL, in which case, the original issue should be investigated on non-KNL systems. Again, there is clearly a bug here, but it helps to localize the problem as much as possible. Jeff > Satish > > > I recommend that you use AVX (Sandy Bridge) - > > preferably AVX2 (Haswell) - as your oldest ISA target when generating a > > portable binary that includes KNL support. > > > > Jeff > > > > On Tue, Apr 10, 2018 at 2:23 PM, Satish Balay wrote: > > > > > I tried a few builds with: > > > > > > '--with-64-bit-indices=1', > > > '--with-memalign=64', > > > '--with-blaslapack-dir=/home/intel/18/compilers_and_ > > > libraries_2018.0.128/linux/mkl', > > > '--with-cc=icc', > > > '--with-fc=ifort', > > > '--with-cxx=0', > > > '--with-debugging=0', > > > '--with-mpi=0', > > > > > > And then changed the OPTFLAGS: > > > > > > 1. 'basic -g' - works fine > > > > > > 'COPTFLAGS=-g', > > > 'FOPTFLAGS=-g', > > > > > > 2. 'avx512' - works fine > > > > > > 'COPTFLAGS=-g -xMIC-AVX512 -O3', > > > 'FOPTFLAGS=-g -xMIC-AVX512 -O3', > > > > > > 3. 'avx2' - breaks. > > > > > > '--COPTFLAGS=-g -axcore-avx2', > > > '--FOPTFLAGS=-g -axcore-avx2', > > > > > > with a breakpoint at dmdavecrestorearrayf903_() in gdb - I see - the > > > stack is fine during the first call to dmdavecrestorearrayf903_() - > > > but is corrupted when it goes to the second call to > > > dmdavecrestorearrayf903_() i.e ierr=0x7fffffffb4a0 changes to > > > ierr=0x0] > > > > > > >>>>>>>>>> > > > > > > Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 , > > > v=0x6030c0 , a=0x401abd , > > > ierr=0x7fffffffb4a0) at /home/petsc/petsc.barry-test/ > > > src/dm/impls/da/f90-custom/zda1f90.c:153 > > > 153 { > > > (gdb) where > > > #0 dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030c0 > > > , a=0x401abd , ierr=0x7fffffffb4a0) > > > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/ > > > zda1f90.c:153 > > > #1 0x0000000000401abd in test () at ex1f.F90:80 > > > #2 0x00000000004011ae in main () > > > #3 0x00007fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6 > > > #4 0x00000000004010b9 in _start () > > > (gdb) c > > > Continuing. > > > > > > Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 , > > > v=0x6030b8 , a=0x401ada , ierr=0x0) > > > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/ > > > zda1f90.c:153 > > > 153 { > > > (gdb) where > > > #0 dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030b8 > > > , a=0x401ada , ierr=0x0) > > > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/ > > > zda1f90.c:153 > > > #1 0x0000000000401ada in test () at ex1f.F90:81 > > > #2 0x00000000004011ae in main () > > > #3 0x00007fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6 > > > #4 0x00000000004010b9 in _start () > > > (gdb) > > > > > > >>>>>>>>> > > > > > > Its not clear to me why this happens. [and why it would work with > > > -xMIC-AVX512 but breaks with -axcore-avx2]. > > > > > > Perhaps Richard, Jeff have better insight on this. > > > > > > BTW: The above run is with: > > > > > > bash-4.2$ icc --version > > > icc (ICC) 18.0.0 20170811 > > > > > > Satish > > > > > > On Mon, 9 Apr 2018, Satish Balay wrote: > > > > > > > I'm able to reproduce this problem on knl box [with the attached test > > > code]. But it goes away if I rebuild without the option > > > --with-64-bit-indices. > > > > > > > > Will have to check further.. > > > > > > > > Satish > > > > > > > > > > > > On Thu, 5 Apr 2018, Randall Mackie wrote: > > > > > > > > > Dear PETSc users, > > > > > > > > > > I?m curious if anyone else experiences problems using > > > DMDAVecGetArrayF90 in conjunction with Intel compilers? > > > > > We have had many problems (typically 11 SEGV segmentation > violations) > > > when PETSc is compiled in optimize mode (with various combinations of > > > options). > > > > > These same codes run valgrind clean with gfortran, so I assume > this is > > > an Intel bug, but before we submit a bug report I wanted to see if > anyone > > > else had similar experiences? > > > > > We have basically gone back and replaced our calls to > > > DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers > > > into a ?local? subroutine that works fine. > > > > > > > > > > In case anyone is curious, the attached test code shows this > behavior > > > when PETSc is compiled with the following options: > > > > > > > > > > ./configure \ > > > > > --with-clean=1 \ > > > > > --with-debugging=0 \ > > > > > --with-fortran=1 \ > > > > > --with-64-bit-indices \ > > > > > --download-mpich=../mpich-3.3a2.tar.gz \ > > > > > --with-blas-lapack-dir=/opt/intel/mkl \ > > > > > --with-cc=icc \ > > > > > --with-fc=ifort \ > > > > > --with-cxx=icc \ > > > > > --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > > > > > --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > > > > > --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2? \ > > > > > > > > > > > > > > > > > > > > Thanks, Randy M. > > > > > > > > > > > > > > > > > > > > > > > > > -- Jeff Hammond jeff.science at gmail.com http://jeffhammond.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Apr 10 19:40:10 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 10 Apr 2018 19:40:10 -0500 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> Message-ID: On Tue, 10 Apr 2018, Jeff Hammond wrote: > Can you try on a non-KNL host? It's a bug either way but I want to > determine if KNL host is the issue. Breaks on 'E5-2695 v4' aswell (bebop.lcrc) with '-axcore-avx2' and 'icc (ICC) 18.0.1 20171018' > Based only what I see below, Randy doesn't seem to be reporting a > KNL-specific issue. Is that incorrect? Hardware details weren't mentioned in this thread. > Again, there is clearly a bug here, but it helps to localize the problem as > much as possible. > > > > > On Thu, 5 Apr 2018, Randall Mackie wrote: > > > > so I assume this is an Intel bug, but before we submit a bug > > > > report I wanted to see if anyone else had similar experiences? Randy, I'll leave this to you to file a report with Intel. Thanks, Satish From rlmackie862 at gmail.com Wed Apr 11 02:12:10 2018 From: rlmackie862 at gmail.com (Randall Mackie) Date: Wed, 11 Apr 2018 09:12:10 +0200 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> Message-ID: <9D37AE46-A179-409E-8476-C2125346B7F9@gmail.com> Hi Satish, Thanks to you and Jeff for taking a look at this. We will submit a bug report to Intel. Randy > On Apr 11, 2018, at 2:40 AM, Satish Balay wrote: > > On Tue, 10 Apr 2018, Jeff Hammond wrote: > >> Can you try on a non-KNL host? It's a bug either way but I want to >> determine if KNL host is the issue. > > Breaks on 'E5-2695 v4' aswell (bebop.lcrc) with '-axcore-avx2' and 'icc (ICC) 18.0.1 20171018' > >> Based only what I see below, Randy doesn't seem to be reporting a >> KNL-specific issue. Is that incorrect? > > Hardware details weren't mentioned in this thread. > >> Again, there is clearly a bug here, but it helps to localize the problem as >> much as possible. > >>>>>> On Thu, 5 Apr 2018, Randall Mackie wrote: > >>>>> so I assume this is an Intel bug, but before we submit a bug >>>>> report I wanted to see if anyone else had similar experiences? > > Randy, > > I'll leave this to you to file a report with Intel. > > Thanks, > Satish From bernardomartinsrocha at gmail.com Wed Apr 11 08:06:19 2018 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Wed, 11 Apr 2018 10:06:19 -0300 Subject: [petsc-users] PETSc version 3.9 with Fortran and DMPlex questions In-Reply-To: References: Message-ID: OK Matthew, thanks a lot for your feedback. Reading the code for the first two faces I noted you used the standard numbering for the faces. face 0 (bottom) -> nodes 0 1 2 3 face 1 (top) -> nodes 4 5 6 7 When this is defined, all the other faces are fixed. Therefore, I noticed some indexes errors in the code: //facesTmp[8] = cone[0]; facesTmp[9] = cone[3]; facesTmp[10] = cone[5]; facesTmp[11] = cone[4]; /* Front */ //facesTmp[12] = cone[2]; facesTmp[13] = cone[1]; facesTmp[14] = cone[7]; facesTmp[15] = cone[6]; /* Back */ //facesTmp[16] = cone[3]; facesTmp[17] = cone[2]; facesTmp[18] = cone[6]; facesTmp[19] = cone[5]; /* Right */ //facesTmp[20] = cone[0]; facesTmp[21] = cone[4]; facesTmp[22] = cone[7]; facesTmp[23] = cone[1]; /* Left */ It should be fixed to (basically swap cone[5] with cone[7]): facesTmp[8] = cone[0]; facesTmp[9] = cone[3]; facesTmp[10] = cone[7]; facesTmp[11] = cone[4]; /* Front */ facesTmp[12] = cone[2]; facesTmp[13] = cone[1]; facesTmp[14] = cone[5]; facesTmp[15] = cone[6]; /* Back */ facesTmp[16] = cone[3]; facesTmp[17] = cone[2]; facesTmp[18] = cone[6]; facesTmp[19] = cone[7]; /* Right */ facesTmp[20] = cone[0]; facesTmp[21] = cone[4]; facesTmp[22] = cone[5]; facesTmp[23] = cone[1]; /* Left */ I did it like this, recompiled the code and at least the number of faces and edge are correct for me now when using DMPlexCreateFromCellList. I don't know if this change has some impact in other parts of the code. I hope not. Best regards, Bernardo M. Rocha On Tue, Apr 10, 2018 at 5:14 PM, Matthew Knepley wrote: > On Tue, Apr 10, 2018 at 2:19 PM, Bernardo Rocha < > bernardomartinsrocha at gmail.com> wrote: > >> Hi everyone, >> >> I have two questions. >> >> (Q1) >> The first one is related to the new PETSc 3.9. >> I've been using Fortran and using the "userctx" to wrap some data >> for the function calls within my program. >> >> Just like : call func(...,userctx) >> and retrieving data as: >> PetscFortranAddr userctx(*) >> b = userctx(2) >> >> With the new version I got the following error: >> b = userctx(2) >> Error: Can't convert INTEGER(8) to TYPE(tvec) at (1) >> >> Is there a way to keep using the trick with "userctx", instead of passing >> all the arguments to the functions separately? >> >> (Q2) >> The second is related to DMPlex. >> When I use the CreateBoxMesh with 2x1x1 cells, I get the expected output: >> >> DM Object: 1 MPI processes >> type: plex >> DM_0x84000000_0 in 3 dimensions: >> 0-cells: 12 >> * 1-cells: 20* >> *2-cells: 11* >> 3-cells: 2 >> Labels: >> depth: 4 strata with value/size (0 (12), 1 (20), 2 (11), 3 (2)) >> >> However, If I try to create that manually to reproduce the same case >> 2x1x1 cells (creating this simple element connectivity by hand) using >> DMPlexCreateFromCellList I get the following: >> >> DM Object: DM_0x84000000_0 1 MPI processes >> type: plex >> DM_0x84000000_0 in 3 dimensions: >> 0-cells: 12 >> *1-cells: 22* >> * 2-cells: 12* >> 3-cells: 2 >> Labels: >> depth: 4 strata with value/size (0 (12), 1 (22), 2 (12), 3 (2)) >> >> That is, it seems the shared face between the two cells is counted twice. >> Is that right? What am I missing here? >> > > Since the input is cell-vertex, my guess is that the order of the vertices > on a cell is not what I expect. > I have readers for common formats, but FromCellList() is my own. You can > see what vertices I think > constitute the faces: > > https://bitbucket.org/petsc/petsc/src/05d412bc580c652749a091cc13ab75 > 6bb8816ec1/src/dm/impls/plex/plexinterpolate.c?at=master& > fileviewer=file-view-default#plexinterpolate.c-113 > > Sorry, this was confusing. Most people are using the input formats, mesh > generators, or predefined shapes, > so I do not get a lot of feedback on this interface. > > Thanks, > > Matt > > >> Best regards, >> Bernardo >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 11 08:15:02 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 11 Apr 2018 09:15:02 -0400 Subject: [petsc-users] PETSc version 3.9 with Fortran and DMPlex questions In-Reply-To: References: Message-ID: On Wed, Apr 11, 2018 at 9:06 AM, Bernardo Rocha < bernardomartinsrocha at gmail.com> wrote: > OK Matthew, thanks a lot for your feedback. > > Reading the code for the first two faces I noted you used the standard > numbering for the faces. > face 0 (bottom) -> nodes 0 1 2 3 > face 1 (top) -> nodes 4 5 6 7 > Hi Bernardo, No, you are misinterpreting the numbering. I want all cell faces to have outward facing normal, so the unit cell looks like 7------6 /| /| / | / | 4------5 | | | | | | | | | | 1------2 | / | / | / |/ 0------3 Does it make more sense now? Thanks, Matt When this is defined, all the other faces are fixed. > Therefore, I noticed some indexes errors in the code: > > //facesTmp[8] = cone[0]; facesTmp[9] = cone[3]; facesTmp[10] = > cone[5]; facesTmp[11] = cone[4]; /* Front */ > //facesTmp[12] = cone[2]; facesTmp[13] = cone[1]; facesTmp[14] = > cone[7]; facesTmp[15] = cone[6]; /* Back */ > //facesTmp[16] = cone[3]; facesTmp[17] = cone[2]; facesTmp[18] = > cone[6]; facesTmp[19] = cone[5]; /* Right */ > //facesTmp[20] = cone[0]; facesTmp[21] = cone[4]; facesTmp[22] = > cone[7]; facesTmp[23] = cone[1]; /* Left */ > > It should be fixed to (basically swap cone[5] with cone[7]): > > facesTmp[8] = cone[0]; facesTmp[9] = cone[3]; facesTmp[10] = > cone[7]; facesTmp[11] = cone[4]; /* Front */ > facesTmp[12] = cone[2]; facesTmp[13] = cone[1]; facesTmp[14] = > cone[5]; facesTmp[15] = cone[6]; /* Back */ > facesTmp[16] = cone[3]; facesTmp[17] = cone[2]; facesTmp[18] = > cone[6]; facesTmp[19] = cone[7]; /* Right */ > facesTmp[20] = cone[0]; facesTmp[21] = cone[4]; facesTmp[22] = > cone[5]; facesTmp[23] = cone[1]; /* Left */ > > I did it like this, recompiled the code and at least the number of faces > and edge are correct for me now > when using DMPlexCreateFromCellList. > > I don't know if this change has some impact in other parts of the code. > I hope not. > > Best regards, > Bernardo M. Rocha > > > > On Tue, Apr 10, 2018 at 5:14 PM, Matthew Knepley > wrote: > >> On Tue, Apr 10, 2018 at 2:19 PM, Bernardo Rocha < >> bernardomartinsrocha at gmail.com> wrote: >> >>> Hi everyone, >>> >>> I have two questions. >>> >>> (Q1) >>> The first one is related to the new PETSc 3.9. >>> I've been using Fortran and using the "userctx" to wrap some data >>> for the function calls within my program. >>> >>> Just like : call func(...,userctx) >>> and retrieving data as: >>> PetscFortranAddr userctx(*) >>> b = userctx(2) >>> >>> With the new version I got the following error: >>> b = userctx(2) >>> Error: Can't convert INTEGER(8) to TYPE(tvec) at (1) >>> >>> Is there a way to keep using the trick with "userctx", instead of passing >>> all the arguments to the functions separately? >>> >>> (Q2) >>> The second is related to DMPlex. >>> When I use the CreateBoxMesh with 2x1x1 cells, I get the expected output: >>> >>> DM Object: 1 MPI processes >>> type: plex >>> DM_0x84000000_0 in 3 dimensions: >>> 0-cells: 12 >>> * 1-cells: 20* >>> *2-cells: 11* >>> 3-cells: 2 >>> Labels: >>> depth: 4 strata with value/size (0 (12), 1 (20), 2 (11), 3 (2)) >>> >>> However, If I try to create that manually to reproduce the same case >>> 2x1x1 cells (creating this simple element connectivity by hand) using >>> DMPlexCreateFromCellList I get the following: >>> >>> DM Object: DM_0x84000000_0 1 MPI processes >>> type: plex >>> DM_0x84000000_0 in 3 dimensions: >>> 0-cells: 12 >>> *1-cells: 22* >>> * 2-cells: 12* >>> 3-cells: 2 >>> Labels: >>> depth: 4 strata with value/size (0 (12), 1 (22), 2 (12), 3 (2)) >>> >>> That is, it seems the shared face between the two cells is counted twice. >>> Is that right? What am I missing here? >>> >> >> Since the input is cell-vertex, my guess is that the order of the >> vertices on a cell is not what I expect. >> I have readers for common formats, but FromCellList() is my own. You can >> see what vertices I think >> constitute the faces: >> >> https://bitbucket.org/petsc/petsc/src/05d412bc580c652749a0 >> 91cc13ab756bb8816ec1/src/dm/impls/plex/plexinterpolate.c? >> at=master&fileviewer=file-view-default#plexinterpolate.c-113 >> >> Sorry, this was confusing. Most people are using the input formats, mesh >> generators, or predefined shapes, >> so I do not get a lot of feedback on this interface. >> >> Thanks, >> >> Matt >> >> >>> Best regards, >>> Bernardo >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernardomartinsrocha at gmail.com Wed Apr 11 08:28:59 2018 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Wed, 11 Apr 2018 10:28:59 -0300 Subject: [petsc-users] PETSc version 3.9 with Fortran and DMPlex questions In-Reply-To: References: Message-ID: Uhmmm. Indeed. Sorry for the mistake. Just fixed my ordering before sending them to DMPlexCreateFromCellList and it worked fine. Thanks. On Wed, Apr 11, 2018 at 10:15 AM, Matthew Knepley wrote: > On Wed, Apr 11, 2018 at 9:06 AM, Bernardo Rocha < > bernardomartinsrocha at gmail.com> wrote: > >> OK Matthew, thanks a lot for your feedback. >> >> Reading the code for the first two faces I noted you used the standard >> numbering for the faces. >> face 0 (bottom) -> nodes 0 1 2 3 >> face 1 (top) -> nodes 4 5 6 7 >> > > Hi Bernardo, > > No, you are misinterpreting the numbering. I want all cell faces to have > outward facing normal, so > the unit cell looks like > > 7------6 > /| /| > / | / | > 4------5 | > | | | | > | | | | > | 1------2 > | / | / > | / |/ > 0------3 > > Does it make more sense now? > > Thanks, > > Matt > > When this is defined, all the other faces are fixed. >> Therefore, I noticed some indexes errors in the code: >> >> //facesTmp[8] = cone[0]; facesTmp[9] = cone[3]; facesTmp[10] = >> cone[5]; facesTmp[11] = cone[4]; /* Front */ >> //facesTmp[12] = cone[2]; facesTmp[13] = cone[1]; facesTmp[14] = >> cone[7]; facesTmp[15] = cone[6]; /* Back */ >> //facesTmp[16] = cone[3]; facesTmp[17] = cone[2]; facesTmp[18] = >> cone[6]; facesTmp[19] = cone[5]; /* Right */ >> //facesTmp[20] = cone[0]; facesTmp[21] = cone[4]; facesTmp[22] = >> cone[7]; facesTmp[23] = cone[1]; /* Left */ >> >> It should be fixed to (basically swap cone[5] with cone[7]): >> >> facesTmp[8] = cone[0]; facesTmp[9] = cone[3]; facesTmp[10] = >> cone[7]; facesTmp[11] = cone[4]; /* Front */ >> facesTmp[12] = cone[2]; facesTmp[13] = cone[1]; facesTmp[14] = >> cone[5]; facesTmp[15] = cone[6]; /* Back */ >> facesTmp[16] = cone[3]; facesTmp[17] = cone[2]; facesTmp[18] = >> cone[6]; facesTmp[19] = cone[7]; /* Right */ >> facesTmp[20] = cone[0]; facesTmp[21] = cone[4]; facesTmp[22] = >> cone[5]; facesTmp[23] = cone[1]; /* Left */ >> >> I did it like this, recompiled the code and at least the number of faces >> and edge are correct for me now >> when using DMPlexCreateFromCellList. >> >> I don't know if this change has some impact in other parts of the code. >> I hope not. >> >> Best regards, >> Bernardo M. Rocha >> >> >> >> On Tue, Apr 10, 2018 at 5:14 PM, Matthew Knepley >> wrote: >> >>> On Tue, Apr 10, 2018 at 2:19 PM, Bernardo Rocha < >>> bernardomartinsrocha at gmail.com> wrote: >>> >>>> Hi everyone, >>>> >>>> I have two questions. >>>> >>>> (Q1) >>>> The first one is related to the new PETSc 3.9. >>>> I've been using Fortran and using the "userctx" to wrap some data >>>> for the function calls within my program. >>>> >>>> Just like : call func(...,userctx) >>>> and retrieving data as: >>>> PetscFortranAddr userctx(*) >>>> b = userctx(2) >>>> >>>> With the new version I got the following error: >>>> b = userctx(2) >>>> Error: Can't convert INTEGER(8) to TYPE(tvec) at (1) >>>> >>>> Is there a way to keep using the trick with "userctx", instead of >>>> passing >>>> all the arguments to the functions separately? >>>> >>>> (Q2) >>>> The second is related to DMPlex. >>>> When I use the CreateBoxMesh with 2x1x1 cells, I get the expected >>>> output: >>>> >>>> DM Object: 1 MPI processes >>>> type: plex >>>> DM_0x84000000_0 in 3 dimensions: >>>> 0-cells: 12 >>>> * 1-cells: 20* >>>> *2-cells: 11* >>>> 3-cells: 2 >>>> Labels: >>>> depth: 4 strata with value/size (0 (12), 1 (20), 2 (11), 3 (2)) >>>> >>>> However, If I try to create that manually to reproduce the same case >>>> 2x1x1 cells (creating this simple element connectivity by hand) using >>>> DMPlexCreateFromCellList I get the following: >>>> >>>> DM Object: DM_0x84000000_0 1 MPI processes >>>> type: plex >>>> DM_0x84000000_0 in 3 dimensions: >>>> 0-cells: 12 >>>> *1-cells: 22* >>>> * 2-cells: 12* >>>> 3-cells: 2 >>>> Labels: >>>> depth: 4 strata with value/size (0 (12), 1 (22), 2 (12), 3 (2)) >>>> >>>> That is, it seems the shared face between the two cells is counted >>>> twice. >>>> Is that right? What am I missing here? >>>> >>> >>> Since the input is cell-vertex, my guess is that the order of the >>> vertices on a cell is not what I expect. >>> I have readers for common formats, but FromCellList() is my own. You can >>> see what vertices I think >>> constitute the faces: >>> >>> https://bitbucket.org/petsc/petsc/src/05d412bc580c652749a0 >>> 91cc13ab756bb8816ec1/src/dm/impls/plex/plexinterpolate.c?at= >>> master&fileviewer=file-view-default#plexinterpolate.c-113 >>> >>> Sorry, this was confusing. Most people are using the input formats, mesh >>> generators, or predefined shapes, >>> so I do not get a lot of feedback on this interface. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Best regards, >>>> Bernardo >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Wed Apr 11 10:07:25 2018 From: hongzhang at anl.gov (Zhang, Hong) Date: Wed, 11 Apr 2018 15:07:25 +0000 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> Message-ID: <1CC8CD4E-81EA-4D81-8B65-2A58F5D5B901@anl.gov> Performance wise, I would suggest to use "-xAVX" instead of "-axcore-avx2". Based on our experience with running PETSc on a variety of Xeon processors (including KNL), using AVX2 yields comparable and sometimes worse performance than using AVX. But if your machine supports AVX-512, it is definitely beneficial to use AVX-512. Hong (Mr.) > On Apr 5, 2018, at 10:03 AM, Randall Mackie wrote: > > Dear PETSc users, > > I?m curious if anyone else experiences problems using DMDAVecGetArrayF90 in conjunction with Intel compilers? > We have had many problems (typically 11 SEGV segmentation violations) when PETSc is compiled in optimize mode (with various combinations of options). > These same codes run valgrind clean with gfortran, so I assume this is an Intel bug, but before we submit a bug report I wanted to see if anyone else had similar experiences? > We have basically gone back and replaced our calls to DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers into a ?local? subroutine that works fine. > > In case anyone is curious, the attached test code shows this behavior when PETSc is compiled with the following options: > > ./configure \ > --with-clean=1 \ > --with-debugging=0 \ > --with-fortran=1 \ > --with-64-bit-indices \ > --download-mpich=../mpich-3.3a2.tar.gz \ > --with-blas-lapack-dir=/opt/intel/mkl \ > --with-cc=icc \ > --with-fc=ifort \ > --with-cxx=icc \ > --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2? \ > > > > Thanks, Randy M. > > From balay at mcs.anl.gov Wed Apr 11 17:08:24 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 11 Apr 2018 17:08:24 -0500 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: <1CC8CD4E-81EA-4D81-8B65-2A58F5D5B901@anl.gov> References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> <1CC8CD4E-81EA-4D81-8B65-2A58F5D5B901@anl.gov> Message-ID: I see this error with "-xAVX" build aswell. Satish On Wed, 11 Apr 2018, Zhang, Hong wrote: > Performance wise, I would suggest to use "-xAVX" instead of "-axcore-avx2". Based on our experience with running PETSc on a variety of Xeon processors (including KNL), using AVX2 yields comparable and sometimes worse performance than using AVX. But if your machine supports AVX-512, it is definitely beneficial to use AVX-512. > > Hong (Mr.) > > > On Apr 5, 2018, at 10:03 AM, Randall Mackie wrote: > > > > Dear PETSc users, > > > > I?m curious if anyone else experiences problems using DMDAVecGetArrayF90 in conjunction with Intel compilers? > > We have had many problems (typically 11 SEGV segmentation violations) when PETSc is compiled in optimize mode (with various combinations of options). > > These same codes run valgrind clean with gfortran, so I assume this is an Intel bug, but before we submit a bug report I wanted to see if anyone else had similar experiences? > > We have basically gone back and replaced our calls to DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers into a ?local? subroutine that works fine. > > > > In case anyone is curious, the attached test code shows this behavior when PETSc is compiled with the following options: > > > > ./configure \ > > --with-clean=1 \ > > --with-debugging=0 \ > > --with-fortran=1 \ > > --with-64-bit-indices \ > > --download-mpich=../mpich-3.3a2.tar.gz \ > > --with-blas-lapack-dir=/opt/intel/mkl \ > > --with-cc=icc \ > > --with-fc=ifort \ > > --with-cxx=icc \ > > --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > > --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ > > --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2? \ > > > > > > > > Thanks, Randy M. > > > > > > From rongliang.chan at gmail.com Wed Apr 11 18:17:43 2018 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Thu, 12 Apr 2018 07:17:43 +0800 Subject: [petsc-users] PetscPrintf Message-ID: <3955a17e-4dd9-ac23-0e89-0e9a8a3145ac@gmail.com> Dear All, When I tried to print a long string using PetscPrintf() I found that it truncated the string. Attached is a simple example for this (run with single processor). I used PetscPrintf() and printf() to print the same string and the printf() seems OK. I am using petsc-3.8.4. Best, Rongliang -------------- next part -------------- A non-text attachment was scrubbed... Name: ex111.c Type: text/x-csrc Size: 1128 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Apr 11 18:22:26 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 11 Apr 2018 23:22:26 +0000 Subject: [petsc-users] PetscPrintf In-Reply-To: <3955a17e-4dd9-ac23-0e89-0e9a8a3145ac@gmail.com> References: <3955a17e-4dd9-ac23-0e89-0e9a8a3145ac@gmail.com> Message-ID: <0F253F9E-0CB9-4E51-BC8C-903376B82CE7@anl.gov> Yes, PetscPrintf() and related functions have a maximum string length of about 8000 characters. Barry > On Apr 11, 2018, at 6:17 PM, Rongliang Chen wrote: > > Dear All, > > > When I tried to print a long string using PetscPrintf() I found that it truncated the string. Attached is a simple example for this (run with single processor). I used PetscPrintf() and printf() to print the same string and the printf() seems OK. I am using petsc-3.8.4. > > > Best, > > Rongliang > > From rongliang.chan at gmail.com Wed Apr 11 19:16:56 2018 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Thu, 12 Apr 2018 08:16:56 +0800 Subject: [petsc-users] PetscPrintf In-Reply-To: <0F253F9E-0CB9-4E51-BC8C-903376B82CE7@anl.gov> References: <3955a17e-4dd9-ac23-0e89-0e9a8a3145ac@gmail.com> <0F253F9E-0CB9-4E51-BC8C-903376B82CE7@anl.gov> Message-ID: Thanks Barry. I found petsc-3.6 and older versions did not have this restriction. Best, Rongliang On 04/12/2018 07:22 AM, Smith, Barry F. wrote: > Yes, PetscPrintf() and related functions have a maximum string length of about 8000 characters. > > Barry > > >> On Apr 11, 2018, at 6:17 PM, Rongliang Chen wrote: >> >> Dear All, >> >> >> When I tried to print a long string using PetscPrintf() I found that it truncated the string. Attached is a simple example for this (run with single processor). I used PetscPrintf() and printf() to print the same string and the printf() seems OK. I am using petsc-3.8.4. >> >> >> Best, >> >> Rongliang >> >> From rlmackie862 at gmail.com Thu Apr 12 00:04:04 2018 From: rlmackie862 at gmail.com (Randall Mackie) Date: Thu, 12 Apr 2018 07:04:04 +0200 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> <1CC8CD4E-81EA-4D81-8B65-2A58F5D5B901@anl.gov> Message-ID: Thanks Satish, we?ve submitted a bug report with Intel and are following up with them. Randy > On Apr 12, 2018, at 12:08 AM, Satish Balay wrote: > > I see this error with "-xAVX" build aswell. > > Satish > > On Wed, 11 Apr 2018, Zhang, Hong wrote: > >> Performance wise, I would suggest to use "-xAVX" instead of "-axcore-avx2". Based on our experience with running PETSc on a variety of Xeon processors (including KNL), using AVX2 yields comparable and sometimes worse performance than using AVX. But if your machine supports AVX-512, it is definitely beneficial to use AVX-512. >> >> Hong (Mr.) >> >>> On Apr 5, 2018, at 10:03 AM, Randall Mackie wrote: >>> >>> Dear PETSc users, >>> >>> I?m curious if anyone else experiences problems using DMDAVecGetArrayF90 in conjunction with Intel compilers? >>> We have had many problems (typically 11 SEGV segmentation violations) when PETSc is compiled in optimize mode (with various combinations of options). >>> These same codes run valgrind clean with gfortran, so I assume this is an Intel bug, but before we submit a bug report I wanted to see if anyone else had similar experiences? >>> We have basically gone back and replaced our calls to DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers into a ?local? subroutine that works fine. >>> >>> In case anyone is curious, the attached test code shows this behavior when PETSc is compiled with the following options: >>> >>> ./configure \ >>> --with-clean=1 \ >>> --with-debugging=0 \ >>> --with-fortran=1 \ >>> --with-64-bit-indices \ >>> --download-mpich=../mpich-3.3a2.tar.gz \ >>> --with-blas-lapack-dir=/opt/intel/mkl \ >>> --with-cc=icc \ >>> --with-fc=ifort \ >>> --with-cxx=icc \ >>> --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ >>> --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ >>> --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2? \ >>> >>> >>> >>> Thanks, Randy M. >>> >>> >> >> From eijkhout at tacc.utexas.edu Thu Apr 12 01:27:09 2018 From: eijkhout at tacc.utexas.edu (Victor Eijkhout) Date: Thu, 12 Apr 2018 06:27:09 +0000 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: <9D37AE46-A179-409E-8476-C2125346B7F9@gmail.com> References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> <9D37AE46-A179-409E-8476-C2125346B7F9@gmail.com> Message-ID: > On Apr 11, 2018, at 9:12 AM, Randall Mackie wrote: > > Hi Satish, > > Thanks to you and Jeff for taking a look at this. We will submit a bug report to Intel. Intel has released 18 update 2 which seems to fix several of these problems, though I have not seen this exact problem with earlier versions. Victor. From aliberkkahraman at yahoo.com Thu Apr 12 07:47:12 2018 From: aliberkkahraman at yahoo.com (Ali Berk Kahraman) Date: Thu, 12 Apr 2018 15:47:12 +0300 Subject: [petsc-users] Finding NullSpaces of Jacobian and TransposeJacobian Message-ID: <4dc98b7e-d785-ab14-da44-3dccea879b34@yahoo.com> Dear All, I am trying to solve a Poisson problem with a known source function with all Neumann BCs. Hence, I know that my problem does not always have a solution, at those times it has a transpose nullspace of one vector. If I know that vector, I can inform petsc of its existence using a MatNullSpace object. Currently, I am trying to find the nullspace vector by solving a linear system where TrJacobian*nullvec = e, but an arbitrary row of the TrJacobian (ith row) is replaced with all zeros except for the ith column, and every entry of e is 0 except for the ith entry which is 1. However, sometimes the KSP that solves this either does not converge, or even if it converges the nullvec it creates does not help with the original problem's ksp. Is there a more reliable way to find the transpose nullspace of a matrix? Best Regards, Ali Berk Kahraman M.Sc. Student, Mechanical Eng. Bogazici Uni. Istanbul, Turkey From hbuesing at eonerc.rwth-aachen.de Thu Apr 12 08:26:05 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Thu, 12 Apr 2018 13:26:05 +0000 Subject: [petsc-users] Optimal coloring using -snes_fd_color with DMDACreate Message-ID: Dear all, When I use -snes_fd_color with my da created by DMDACreate3d, will it use an optimal coloring algorithm (see [1]) to estimate the Jacobian? Thank you! Henrik [1] Goldfarb/Toint: Optimal estimation of Jacobian and Hessian matrices that arise in finite difference calculations https://doi.org/10.1090/S0025-5718-1984-0744925-5 -- Dipl.-Math. Henrik B?sing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing at eonerc.rwth-aachen.de ------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Apr 12 08:30:36 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 12 Apr 2018 08:30:36 -0500 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> <9D37AE46-A179-409E-8476-C2125346B7F9@gmail.com> Message-ID: On Thu, 12 Apr 2018, Victor Eijkhout wrote: > > > > On Apr 11, 2018, at 9:12 AM, Randall Mackie wrote: > > > > Hi Satish, > > > > Thanks to you and Jeff for taking a look at this. We will submit a bug report to Intel. > > Intel has released 18 update 2 which seems to fix several of these problems, though I have not seen this exact problem with earlier versions. I can reproudce this issue with 'icc (ICC) 18.0.2 20180210'. Is there a newer version? Satish From eijkhout at tacc.utexas.edu Thu Apr 12 08:40:59 2018 From: eijkhout at tacc.utexas.edu (Victor Eijkhout) Date: Thu, 12 Apr 2018 13:40:59 +0000 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> <9D37AE46-A179-409E-8476-C2125346B7F9@gmail.com> Message-ID: On Apr 12, 2018, at 3:30 PM, Satish Balay > wrote: I can reproudce this issue with 'icc (ICC) 18.0.2 20180210'. Is there a newer version? Probably not. But I can?t figure out Intel version numbers. Here?s what I find: compilers_and_libraries_2018.2.199/ Victor. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Apr 12 08:47:48 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 12 Apr 2018 08:47:48 -0500 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> <9D37AE46-A179-409E-8476-C2125346B7F9@gmail.com> Message-ID: On Thu, 12 Apr 2018, Victor Eijkhout wrote: > > > On Apr 12, 2018, at 3:30 PM, Satish Balay > wrote: > > I can reproudce this issue with 'icc (ICC) 18.0.2 20180210'. Is there a newer version? > > Probably not. But I can?t figure out Intel version numbers. Here?s what I find: > > compilers_and_libraries_2018.2.199/ bash-4.2$ which icc /homes/intel/18u2/compilers_and_libraries_2018.2.199/linux/bin/intel64/icc Ok - its the same version. Satish From bsmith at mcs.anl.gov Thu Apr 12 09:37:42 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 12 Apr 2018 14:37:42 +0000 Subject: [petsc-users] Finding NullSpaces of Jacobian and TransposeJacobian In-Reply-To: <4dc98b7e-d785-ab14-da44-3dccea879b34@yahoo.com> References: <4dc98b7e-d785-ab14-da44-3dccea879b34@yahoo.com> Message-ID: A few thoughts, > On Apr 12, 2018, at 7:47 AM, Ali Berk Kahraman wrote: > > Dear All, > > I am trying to solve a Poisson problem with a known source function with all Neumann BCs. If you are using finite elements then the matrix should be symmetric and hence its transpose nullspace is the same as the original matrix. Presumably you want to know the transpose null space so you can remove it from the right hand side to get a consistent linear system? With GMRES you don't need to actually remove it from the right hand side. Just apply GMRES to the inconsistent system, what will happen, if you use a good preconditioner is that GMRES will converge to a solution that minimizes the residual. So the residual norm will not be zero but the solution to GMRES is a valid solution to the linear system. In fact the "left over" residual will be in the null space of the transpose; in your case it will be the null space (since it is one dimensional). Barry > Hence, I know that my problem does not always have a solution, at those times it has a transpose nullspace of one vector. If I know that vector, I can inform petsc of its existence using a MatNullSpace object. > > Currently, I am trying to find the nullspace vector by solving a linear system where TrJacobian*nullvec = e, but an arbitrary row of the TrJacobian (ith row) is replaced with all zeros except for the ith column, and every entry of e is 0 except for the ith entry which is 1. However, sometimes the KSP that solves this either does not converge, or even if it converges the nullvec it creates does not help with the original problem's ksp. > > Is there a more reliable way to find the transpose nullspace of a matrix? > > Best Regards, > > Ali Berk Kahraman > M.Sc. Student, Mechanical Eng. > Bogazici Uni. Istanbul, Turkey > From bsmith at mcs.anl.gov Thu Apr 12 09:41:37 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 12 Apr 2018 14:41:37 +0000 Subject: [petsc-users] Optimal coloring using -snes_fd_color with DMDACreate In-Reply-To: References: Message-ID: I'm not familiar with that paper so I am not sure if it produces exactly the same colorings. It is based on the same principle. The coloring is done in /src/dm/impls/da/fdda.c DMCreateColoring_DA_3d_MPIAIJ() so you could compare. Note if they have more efficient special cases in the paper they could be added to the code without too much difficulty. Barry > On Apr 12, 2018, at 8:26 AM, Buesing, Henrik wrote: > > Dear all, > > When I use ?snes_fd_color with my da created by DMDACreate3d, will it use an optimal coloring algorithm (see [1]) to estimate the Jacobian? > > Thank you! > Henrik > > > > [1] > Goldfarb/Toint: Optimal estimation of Jacobian and Hessian matrices that arise in finite difference calculations > https://doi.org/10.1090/S0025-5718-1984-0744925-5 > > > -- > Dipl.-Math. Henrik B?sing > Institute for Applied Geophysics and Geothermal Energy > E.ON Energy Research Center > RWTH Aachen University > ------------------------------------------------------ > Mathieustr. 10 | Tel +49 (0)241 80 49907 > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > ------------------------------------------------------ > http://www.eonerc.rwth-aachen.de/GGE > hbuesing at eonerc.rwth-aachen.de > ------------------------------------------------------ From aliberkkahraman at yahoo.com Thu Apr 12 10:43:03 2018 From: aliberkkahraman at yahoo.com (Ali Berk Kahraman) Date: Thu, 12 Apr 2018 18:43:03 +0300 Subject: [petsc-users] Finding NullSpaces of Jacobian and TransposeJacobian In-Reply-To: References: <4dc98b7e-d785-ab14-da44-3dccea879b34@yahoo.com> Message-ID: <6359a517-6342-6b98-e7f4-4a8c8138a854@yahoo.com> Dear Barry, Thank you very very much for your response. Unfortunately, I use finite differences on an unstructured grid, so my matrix is not symmetric. That is exactly why I want to know the null space, so I can remove it from the RHS vector. Or as I understand it from the online documentation, so petsc can solve a least-squares problem using a MatNullSpace object. You suggestion should also work. I am fine with all of these methods. The problem is, I still have to know the left null space, or its norm in your suggestion, to use these methods. On 12-04-2018 17:37, Smith, Barry F. wrote: > A few thoughts, > > >> On Apr 12, 2018, at 7:47 AM, Ali Berk Kahraman wrote: >> >> Dear All, >> >> I am trying to solve a Poisson problem with a known source function with all Neumann BCs. > If you are using finite elements then the matrix should be symmetric and hence its transpose nullspace is the same as the original matrix. > > Presumably you want to know the transpose null space so you can remove it from the right hand side to get a consistent linear system? With > GMRES you don't need to actually remove it from the right hand side. Just apply GMRES to the inconsistent system, what will happen, if you > use a good preconditioner is that GMRES will converge to a solution that minimizes the residual. So the residual norm will not be zero but the solution to GMRES is a valid solution to the linear system. In fact the "left over" residual will be in the null space of the transpose; in your case it will be the null space (since it is one dimensional). > > Barry > > >> Hence, I know that my problem does not always have a solution, at those times it has a transpose nullspace of one vector. If I know that vector, I can inform petsc of its existence using a MatNullSpace object. >> >> Currently, I am trying to find the nullspace vector by solving a linear system where TrJacobian*nullvec = e, but an arbitrary row of the TrJacobian (ith row) is replaced with all zeros except for the ith column, and every entry of e is 0 except for the ith entry which is 1. However, sometimes the KSP that solves this either does not converge, or even if it converges the nullvec it creates does not help with the original problem's ksp. >> >> Is there a more reliable way to find the transpose nullspace of a matrix? >> >> Best Regards, >> >> Ali Berk Kahraman >> M.Sc. Student, Mechanical Eng. >> Bogazici Uni. Istanbul, Turkey >> From rlmackie862 at gmail.com Thu Apr 12 10:49:55 2018 From: rlmackie862 at gmail.com (Randall Mackie) Date: Thu, 12 Apr 2018 17:49:55 +0200 Subject: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel In-Reply-To: References: <4B547C08-A6D3-428B-89B7-48AA59BA697F@gmail.com> <9D37AE46-A179-409E-8476-C2125346B7F9@gmail.com> Message-ID: <11D3EBDC-EB14-473A-9827-386F8C4161BF@gmail.com> > On Apr 12, 2018, at 3:47 PM, Satish Balay wrote: > > On Thu, 12 Apr 2018, Victor Eijkhout wrote: > >> >> >> On Apr 12, 2018, at 3:30 PM, Satish Balay > wrote: >> >> I can reproudce this issue with 'icc (ICC) 18.0.2 20180210'. Is there a newer version? >> >> Probably not. But I can?t figure out Intel version numbers. Here?s what I find: >> >> compilers_and_libraries_2018.2.199/ > > bash-4.2$ which icc > /homes/intel/18u2/compilers_and_libraries_2018.2.199/linux/bin/intel64/icc > > Ok - its the same version. > > Satish Thanks Satish, we can also confirm the bug still exists in update 2 of the Intel compiler, and we have updated our bug report with Intel to indicate this. Randy M. From bsmith at mcs.anl.gov Thu Apr 12 10:54:13 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 12 Apr 2018 15:54:13 +0000 Subject: [petsc-users] Finding NullSpaces of Jacobian and TransposeJacobian In-Reply-To: <6359a517-6342-6b98-e7f4-4a8c8138a854@yahoo.com> References: <4dc98b7e-d785-ab14-da44-3dccea879b34@yahoo.com> <6359a517-6342-6b98-e7f4-4a8c8138a854@yahoo.com> Message-ID: <7E5D7BFC-954F-41EC-A457-9FB8D61C88ED@anl.gov> > On Apr 12, 2018, at 10:43 AM, Ali Berk Kahraman wrote: > > Dear Barry, > > Thank you very very much for your response. > > Unfortunately, I use finite differences on an unstructured grid, so my matrix is not symmetric. > > That is exactly why I want to know the null space, so I can remove it from the RHS vector. Or as I understand it from the online documentation, so petsc can solve a least-squares problem using a MatNullSpace object. You suggestion should also work. I am fine with all of these methods. The problem is, I still have to know the left null space, or its norm in your suggestion, to use these methods. I don't think you need to know the null space. As I said, just run GMRES until the residual norm becomes fixed (but non-zero) and you have your solution. No need to remove the transpose null space from the right hand side initially. Barry > > > On 12-04-2018 17:37, Smith, Barry F. wrote: >> A few thoughts, >> >> >>> On Apr 12, 2018, at 7:47 AM, Ali Berk Kahraman wrote: >>> >>> Dear All, >>> >>> I am trying to solve a Poisson problem with a known source function with all Neumann BCs. >> If you are using finite elements then the matrix should be symmetric and hence its transpose nullspace is the same as the original matrix. >> >> Presumably you want to know the transpose null space so you can remove it from the right hand side to get a consistent linear system? With >> GMRES you don't need to actually remove it from the right hand side. Just apply GMRES to the inconsistent system, what will happen, if you >> use a good preconditioner is that GMRES will converge to a solution that minimizes the residual. So the residual norm will not be zero but the solution to GMRES is a valid solution to the linear system. In fact the "left over" residual will be in the null space of the transpose; in your case it will be the null space (since it is one dimensional). >> >> Barry >> >> >>> Hence, I know that my problem does not always have a solution, at those times it has a transpose nullspace of one vector. If I know that vector, I can inform petsc of its existence using a MatNullSpace object. >>> >>> Currently, I am trying to find the nullspace vector by solving a linear system where TrJacobian*nullvec = e, but an arbitrary row of the TrJacobian (ith row) is replaced with all zeros except for the ith column, and every entry of e is 0 except for the ith entry which is 1. However, sometimes the KSP that solves this either does not converge, or even if it converges the nullvec it creates does not help with the original problem's ksp. >>> >>> Is there a more reliable way to find the transpose nullspace of a matrix? >>> >>> Best Regards, >>> >>> Ali Berk Kahraman >>> M.Sc. Student, Mechanical Eng. >>> Bogazici Uni. Istanbul, Turkey >>> > From aliberkkahraman at yahoo.com Thu Apr 12 10:56:29 2018 From: aliberkkahraman at yahoo.com (Ali Berk Kahraman) Date: Thu, 12 Apr 2018 18:56:29 +0300 Subject: [petsc-users] Finding NullSpaces of Jacobian and TransposeJacobian In-Reply-To: <7E5D7BFC-954F-41EC-A457-9FB8D61C88ED@anl.gov> References: <4dc98b7e-d785-ab14-da44-3dccea879b34@yahoo.com> <6359a517-6342-6b98-e7f4-4a8c8138a854@yahoo.com> <7E5D7BFC-954F-41EC-A457-9FB8D61C88ED@anl.gov> Message-ID: <756a56c7-b969-25de-cd56-b1b64d7c8c72@yahoo.com> How do I tell that to the KSP object? Is there a direct command or a function? On 12-04-2018 18:54, Smith, Barry F. wrote: > >> On Apr 12, 2018, at 10:43 AM, Ali Berk Kahraman wrote: >> >> Dear Barry, >> >> Thank you very very much for your response. >> >> Unfortunately, I use finite differences on an unstructured grid, so my matrix is not symmetric. >> >> That is exactly why I want to know the null space, so I can remove it from the RHS vector. Or as I understand it from the online documentation, so petsc can solve a least-squares problem using a MatNullSpace object. You suggestion should also work. I am fine with all of these methods. The problem is, I still have to know the left null space, or its norm in your suggestion, to use these methods. > I don't think you need to know the null space. As I said, just run GMRES until the residual norm becomes fixed (but non-zero) and you have your solution. No need to remove the transpose null space from the right hand side initially. > > Barry > >> >> On 12-04-2018 17:37, Smith, Barry F. wrote: >>> A few thoughts, >>> >>> >>>> On Apr 12, 2018, at 7:47 AM, Ali Berk Kahraman wrote: >>>> >>>> Dear All, >>>> >>>> I am trying to solve a Poisson problem with a known source function with all Neumann BCs. >>> If you are using finite elements then the matrix should be symmetric and hence its transpose nullspace is the same as the original matrix. >>> >>> Presumably you want to know the transpose null space so you can remove it from the right hand side to get a consistent linear system? With >>> GMRES you don't need to actually remove it from the right hand side. Just apply GMRES to the inconsistent system, what will happen, if you >>> use a good preconditioner is that GMRES will converge to a solution that minimizes the residual. So the residual norm will not be zero but the solution to GMRES is a valid solution to the linear system. In fact the "left over" residual will be in the null space of the transpose; in your case it will be the null space (since it is one dimensional). >>> >>> Barry >>> >>> >>>> Hence, I know that my problem does not always have a solution, at those times it has a transpose nullspace of one vector. If I know that vector, I can inform petsc of its existence using a MatNullSpace object. >>>> >>>> Currently, I am trying to find the nullspace vector by solving a linear system where TrJacobian*nullvec = e, but an arbitrary row of the TrJacobian (ith row) is replaced with all zeros except for the ith column, and every entry of e is 0 except for the ith entry which is 1. However, sometimes the KSP that solves this either does not converge, or even if it converges the nullvec it creates does not help with the original problem's ksp. >>>> >>>> Is there a more reliable way to find the transpose nullspace of a matrix? >>>> >>>> Best Regards, >>>> >>>> Ali Berk Kahraman >>>> M.Sc. Student, Mechanical Eng. >>>> Bogazici Uni. Istanbul, Turkey >>>> From bsmith at mcs.anl.gov Thu Apr 12 11:00:32 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 12 Apr 2018 16:00:32 +0000 Subject: [petsc-users] Finding NullSpaces of Jacobian and TransposeJacobian In-Reply-To: <756a56c7-b969-25de-cd56-b1b64d7c8c72@yahoo.com> References: <4dc98b7e-d785-ab14-da44-3dccea879b34@yahoo.com> <6359a517-6342-6b98-e7f4-4a8c8138a854@yahoo.com> <7E5D7BFC-954F-41EC-A457-9FB8D61C88ED@anl.gov> <756a56c7-b969-25de-cd56-b1b64d7c8c72@yahoo.com> Message-ID: <0632EF1C-D155-4D23-B354-C3C410D9A90F@mcs.anl.gov> > On Apr 12, 2018, at 10:56 AM, Ali Berk Kahraman wrote: > > How do I tell that to the KSP object? Is there a direct command or a function? Tell the KSP object what? What happens when you run with -ksp_monitor_true_residual ? We may not have a good convergence test for this case, since the default convergence test wants a residual norm near 0. Barry > > > On 12-04-2018 18:54, Smith, Barry F. wrote: >> >>> On Apr 12, 2018, at 10:43 AM, Ali Berk Kahraman wrote: >>> >>> Dear Barry, >>> >>> Thank you very very much for your response. >>> >>> Unfortunately, I use finite differences on an unstructured grid, so my matrix is not symmetric. >>> >>> That is exactly why I want to know the null space, so I can remove it from the RHS vector. Or as I understand it from the online documentation, so petsc can solve a least-squares problem using a MatNullSpace object. You suggestion should also work. I am fine with all of these methods. The problem is, I still have to know the left null space, or its norm in your suggestion, to use these methods. >> I don't think you need to know the null space. As I said, just run GMRES until the residual norm becomes fixed (but non-zero) and you have your solution. No need to remove the transpose null space from the right hand side initially. >> >> Barry >> >>> >>> On 12-04-2018 17:37, Smith, Barry F. wrote: >>>> A few thoughts, >>>> >>>> >>>>> On Apr 12, 2018, at 7:47 AM, Ali Berk Kahraman wrote: >>>>> >>>>> Dear All, >>>>> >>>>> I am trying to solve a Poisson problem with a known source function with all Neumann BCs. >>>> If you are using finite elements then the matrix should be symmetric and hence its transpose nullspace is the same as the original matrix. >>>> >>>> Presumably you want to know the transpose null space so you can remove it from the right hand side to get a consistent linear system? With >>>> GMRES you don't need to actually remove it from the right hand side. Just apply GMRES to the inconsistent system, what will happen, if you >>>> use a good preconditioner is that GMRES will converge to a solution that minimizes the residual. So the residual norm will not be zero but the solution to GMRES is a valid solution to the linear system. In fact the "left over" residual will be in the null space of the transpose; in your case it will be the null space (since it is one dimensional). >>>> >>>> Barry >>>> >>>> >>>>> Hence, I know that my problem does not always have a solution, at those times it has a transpose nullspace of one vector. If I know that vector, I can inform petsc of its existence using a MatNullSpace object. >>>>> >>>>> Currently, I am trying to find the nullspace vector by solving a linear system where TrJacobian*nullvec = e, but an arbitrary row of the TrJacobian (ith row) is replaced with all zeros except for the ith column, and every entry of e is 0 except for the ith entry which is 1. However, sometimes the KSP that solves this either does not converge, or even if it converges the nullvec it creates does not help with the original problem's ksp. >>>>> >>>>> Is there a more reliable way to find the transpose nullspace of a matrix? >>>>> >>>>> Best Regards, >>>>> >>>>> Ali Berk Kahraman >>>>> M.Sc. Student, Mechanical Eng. >>>>> Bogazici Uni. Istanbul, Turkey >>>>> > From aliberkkahraman at yahoo.com Thu Apr 12 11:13:55 2018 From: aliberkkahraman at yahoo.com (Ali Berk Kahraman) Date: Thu, 12 Apr 2018 19:13:55 +0300 Subject: [petsc-users] Finding NullSpaces of Jacobian and TransposeJacobian In-Reply-To: <0632EF1C-D155-4D23-B354-C3C410D9A90F@mcs.anl.gov> References: <4dc98b7e-d785-ab14-da44-3dccea879b34@yahoo.com> <6359a517-6342-6b98-e7f4-4a8c8138a854@yahoo.com> <7E5D7BFC-954F-41EC-A457-9FB8D61C88ED@anl.gov> <756a56c7-b969-25de-cd56-b1b64d7c8c72@yahoo.com> <0632EF1C-D155-4D23-B354-C3C410D9A90F@mcs.anl.gov> Message-ID: <354b488d-f74d-bfce-1bff-741dd815c537@yahoo.com> That was I was trying to ask, sorry for my vagueness. What kind of a convergence test should I conduct? Correct me if I am mistaken, the KSP object compares the current residual norm with iteration 0. If it reaches rtol, it declares convergence. What should I do to make it compare the residual norm with the previous iteration, and not iteration 0? On 12-04-2018 19:00, Smith, Barry F. wrote: > >> On Apr 12, 2018, at 10:56 AM, Ali Berk Kahraman wrote: >> >> How do I tell that to the KSP object? Is there a direct command or a function? > Tell the KSP object what? > > What happens when you run with -ksp_monitor_true_residual ? > > We may not have a good convergence test for this case, since the default convergence test wants a residual norm near 0. > > Barry > >> >> On 12-04-2018 18:54, Smith, Barry F. wrote: >>>> On Apr 12, 2018, at 10:43 AM, Ali Berk Kahraman wrote: >>>> >>>> Dear Barry, >>>> >>>> Thank you very very much for your response. >>>> >>>> Unfortunately, I use finite differences on an unstructured grid, so my matrix is not symmetric. >>>> >>>> That is exactly why I want to know the null space, so I can remove it from the RHS vector. Or as I understand it from the online documentation, so petsc can solve a least-squares problem using a MatNullSpace object. You suggestion should also work. I am fine with all of these methods. The problem is, I still have to know the left null space, or its norm in your suggestion, to use these methods. >>> I don't think you need to know the null space. As I said, just run GMRES until the residual norm becomes fixed (but non-zero) and you have your solution. No need to remove the transpose null space from the right hand side initially. >>> >>> Barry >>> >>>> On 12-04-2018 17:37, Smith, Barry F. wrote: >>>>> A few thoughts, >>>>> >>>>> >>>>>> On Apr 12, 2018, at 7:47 AM, Ali Berk Kahraman wrote: >>>>>> >>>>>> Dear All, >>>>>> >>>>>> I am trying to solve a Poisson problem with a known source function with all Neumann BCs. >>>>> If you are using finite elements then the matrix should be symmetric and hence its transpose nullspace is the same as the original matrix. >>>>> >>>>> Presumably you want to know the transpose null space so you can remove it from the right hand side to get a consistent linear system? With >>>>> GMRES you don't need to actually remove it from the right hand side. Just apply GMRES to the inconsistent system, what will happen, if you >>>>> use a good preconditioner is that GMRES will converge to a solution that minimizes the residual. So the residual norm will not be zero but the solution to GMRES is a valid solution to the linear system. In fact the "left over" residual will be in the null space of the transpose; in your case it will be the null space (since it is one dimensional). >>>>> >>>>> Barry >>>>> >>>>> >>>>>> Hence, I know that my problem does not always have a solution, at those times it has a transpose nullspace of one vector. If I know that vector, I can inform petsc of its existence using a MatNullSpace object. >>>>>> >>>>>> Currently, I am trying to find the nullspace vector by solving a linear system where TrJacobian*nullvec = e, but an arbitrary row of the TrJacobian (ith row) is replaced with all zeros except for the ith column, and every entry of e is 0 except for the ith entry which is 1. However, sometimes the KSP that solves this either does not converge, or even if it converges the nullvec it creates does not help with the original problem's ksp. >>>>>> >>>>>> Is there a more reliable way to find the transpose nullspace of a matrix? >>>>>> >>>>>> Best Regards, >>>>>> >>>>>> Ali Berk Kahraman >>>>>> M.Sc. Student, Mechanical Eng. >>>>>> Bogazici Uni. Istanbul, Turkey >>>>>> From bsmith at mcs.anl.gov Thu Apr 12 13:33:40 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 12 Apr 2018 18:33:40 +0000 Subject: [petsc-users] Finding NullSpaces of Jacobian and TransposeJacobian In-Reply-To: <354b488d-f74d-bfce-1bff-741dd815c537@yahoo.com> References: <4dc98b7e-d785-ab14-da44-3dccea879b34@yahoo.com> <6359a517-6342-6b98-e7f4-4a8c8138a854@yahoo.com> <7E5D7BFC-954F-41EC-A457-9FB8D61C88ED@anl.gov> <756a56c7-b969-25de-cd56-b1b64d7c8c72@yahoo.com> <0632EF1C-D155-4D23-B354-C3C410D9A90F@mcs.anl.gov> <354b488d-f74d-bfce-1bff-741dd815c537@yahoo.com> Message-ID: <6C4AF03B-2BEB-4DCA-B46F-B3EF4A50F4C0@mcs.anl.gov> Unfortunately I don't have a good answer for you. The manual page for KSPCGNE says one should use KSPLSQR for least squares problems. This one hopefully has the correct convergence test for inconsistent systems built in. Please try this solver and let us know how it goes. Barry > On Apr 12, 2018, at 11:13 AM, Ali Berk Kahraman wrote: > > That was I was trying to ask, sorry for my vagueness. What kind of a convergence test should I conduct? Correct me if I am mistaken, the KSP object compares the current residual norm with iteration 0. If it reaches rtol, it declares convergence. What should I do to make it compare the residual norm with the previous iteration, and not iteration 0? > > On 12-04-2018 19:00, Smith, Barry F. wrote: >> >>> On Apr 12, 2018, at 10:56 AM, Ali Berk Kahraman wrote: >>> >>> How do I tell that to the KSP object? Is there a direct command or a function? >> Tell the KSP object what? >> >> What happens when you run with -ksp_monitor_true_residual ? >> >> We may not have a good convergence test for this case, since the default convergence test wants a residual norm near 0. >> >> Barry >> >>> >>> On 12-04-2018 18:54, Smith, Barry F. wrote: >>>>> On Apr 12, 2018, at 10:43 AM, Ali Berk Kahraman wrote: >>>>> >>>>> Dear Barry, >>>>> >>>>> Thank you very very much for your response. >>>>> >>>>> Unfortunately, I use finite differences on an unstructured grid, so my matrix is not symmetric. >>>>> >>>>> That is exactly why I want to know the null space, so I can remove it from the RHS vector. Or as I understand it from the online documentation, so petsc can solve a least-squares problem using a MatNullSpace object. You suggestion should also work. I am fine with all of these methods. The problem is, I still have to know the left null space, or its norm in your suggestion, to use these methods. >>>> I don't think you need to know the null space. As I said, just run GMRES until the residual norm becomes fixed (but non-zero) and you have your solution. No need to remove the transpose null space from the right hand side initially. >>>> >>>> Barry >>>> >>>>> On 12-04-2018 17:37, Smith, Barry F. wrote: >>>>>> A few thoughts, >>>>>> >>>>>> >>>>>>> On Apr 12, 2018, at 7:47 AM, Ali Berk Kahraman wrote: >>>>>>> >>>>>>> Dear All, >>>>>>> >>>>>>> I am trying to solve a Poisson problem with a known source function with all Neumann BCs. >>>>>> If you are using finite elements then the matrix should be symmetric and hence its transpose nullspace is the same as the original matrix. >>>>>> >>>>>> Presumably you want to know the transpose null space so you can remove it from the right hand side to get a consistent linear system? With >>>>>> GMRES you don't need to actually remove it from the right hand side. Just apply GMRES to the inconsistent system, what will happen, if you >>>>>> use a good preconditioner is that GMRES will converge to a solution that minimizes the residual. So the residual norm will not be zero but the solution to GMRES is a valid solution to the linear system. In fact the "left over" residual will be in the null space of the transpose; in your case it will be the null space (since it is one dimensional). >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>>> Hence, I know that my problem does not always have a solution, at those times it has a transpose nullspace of one vector. If I know that vector, I can inform petsc of its existence using a MatNullSpace object. >>>>>>> >>>>>>> Currently, I am trying to find the nullspace vector by solving a linear system where TrJacobian*nullvec = e, but an arbitrary row of the TrJacobian (ith row) is replaced with all zeros except for the ith column, and every entry of e is 0 except for the ith entry which is 1. However, sometimes the KSP that solves this either does not converge, or even if it converges the nullvec it creates does not help with the original problem's ksp. >>>>>>> >>>>>>> Is there a more reliable way to find the transpose nullspace of a matrix? >>>>>>> >>>>>>> Best Regards, >>>>>>> >>>>>>> Ali Berk Kahraman >>>>>>> M.Sc. Student, Mechanical Eng. >>>>>>> Bogazici Uni. Istanbul, Turkey >>>>>>> > From k_burkart at yahoo.com Thu Apr 12 13:54:05 2018 From: k_burkart at yahoo.com (Klaus Burkart) Date: Thu, 12 Apr 2018 18:54:05 +0000 (UTC) Subject: [petsc-users] Which matrix object type should I use? References: <371887203.2313240.1523559245624.ref@mail.yahoo.com> Message-ID: <371887203.2313240.1523559245624@mail.yahoo.com> I am confused with matrix object types and which one to use in my case: I want to solve a linear system using multiple processes. The steps I am struggling with is the import of the global matrix and global rhs vector into applicable PETSc objects. The global matrix is currently stored in CSR format using 3 arrays. Array one contains all the row indices, array two all the column indices, array three all the values of the global matrix. In addition to that, there's a vector object containing the rhs vector. I think for the vector object I should use VectorCreateMPI with m set to PETSC_DECIDE? What's the matrix object I should use in my case which also takes care of the allocation local matrices to the processes at a later stage i.e. creates the local matrices in line with the rhs vector? Klaus -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Apr 12 14:10:08 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 12 Apr 2018 19:10:08 +0000 Subject: [petsc-users] Which matrix object type should I use? In-Reply-To: <371887203.2313240.1523559245624@mail.yahoo.com> References: <371887203.2313240.1523559245624.ref@mail.yahoo.com> <371887203.2313240.1523559245624@mail.yahoo.com> Message-ID: <624C893D-7D94-46A2-94CE-0536491E797F@anl.gov> Are your matrix entries stored in parallel or is your current code that generates the rows, columns, and values sequential. Does your matrix have a pointwise-block structure (like from having multiple degrees of freedom per grid point) or is it a scalar PDE? If scalar than you use AIJ matrices if point wise-block structure then use BAIJ. If symmetric than use SBAIJ. How are your matrix entries generated? Via the finite element method, finite differences, something else? Is there a mesh, how do you handle the mesh? Barry > On Apr 12, 2018, at 1:54 PM, Klaus Burkart wrote: > > I am confused with matrix object types and which one to use in my case: > > > I want to solve a linear system using multiple processes. The steps I am struggling with is the import of the global matrix and global rhs vector into applicable PETSc objects. > > The global matrix is currently stored in CSR format using 3 arrays. Array one contains all the row indices, array two all the column indices, array three all the values of the global matrix. In addition to that, there's a vector object containing the rhs vector. > > I think for the vector object I should use VectorCreateMPI with m set to PETSC_DECIDE? > > > What's the matrix object I should use in my case which also takes care of the allocation local matrices to the processes at a later stage i.e. creates the local matrices in line with the rhs vector? > > Klaus From bsmith at mcs.anl.gov Thu Apr 12 14:46:33 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 12 Apr 2018 19:46:33 +0000 Subject: [petsc-users] Which matrix object type should I use? In-Reply-To: <94486403.2336635.1523561529357@mail.yahoo.com> References: <371887203.2313240.1523559245624.ref@mail.yahoo.com> <371887203.2313240.1523559245624@mail.yahoo.com> <624C893D-7D94-46A2-94CE-0536491E797F@anl.gov> <94486403.2336635.1523561529357@mail.yahoo.com> Message-ID: So you want to have a parallel linear? solver and the rest of the code including Jacobian generation sequential? Barry > On Apr 12, 2018, at 2:32 PM, Klaus Burkart wrote: > > Are your matrix entries stored in parallel or is your current code that generates the rows, columns, and values sequential. > > >> The three arrays are the result of a matrix format conversion to CSR format in my cfd application interface, the code is sequential. > > > Does your matrix have a pointwise-block structure (like from having multiple degrees of freedom per grid point) or is it a scalar PDE? If scalar than you use AIJ matrices if point wise-block structure then use BAIJ. If symmetric than use SBAIJ. > > >> I assume it's a case for AIJ > > > How are your matrix entries generated? Via the finite element method, finite differences, something else? Is there a mesh, how do you handle the mesh? > > >> Via finite volume, yes there is a mesh but I am only replacing the matrix solver (as I've done with another LA library before) afterwards returning the solution vector to the application where all the other steps are handled > > Klaus > > > Am Donnerstag, 12. April 2018, 21:10:12 MESZ hat Smith, Barry F. Folgendes geschrieben: > > > > Are your matrix entries stored in parallel or is your current code that generates the rows, columns, and values sequential. > > Does your matrix have a pointwise-block structure (like from having multiple degrees of freedom per grid point) or is it a scalar PDE? If scalar than you use AIJ matrices if point wise-block structure then use BAIJ. If symmetric than use SBAIJ. > > How are your matrix entries generated? Via the finite element method, finite differences, something else? Is there a mesh, how do you handle the mesh? > > Barry > > > > On Apr 12, 2018, at 1:54 PM, Klaus Burkart wrote: > > > > I am confused with matrix object types and which one to use in my case: > > > > > > I want to solve a linear system using multiple processes. The steps I am struggling with is the import of the global matrix and global rhs vector into applicable PETSc objects. > > > > The global matrix is currently stored in CSR format using 3 arrays. Array one contains all the row indices, array two all the column indices, array three all the values of the global matrix. In addition to that, there's a vector object containing the rhs vector. > > > > I think for the vector object I should use VectorCreateMPI with m set to PETSC_DECIDE? > > > > > > What's the matrix object I should use in my case which also takes care of the allocation local matrices to the processes at a later stage i.e. creates the local matrices in line with the rhs vector? > > > > Klaus From aliberkkahraman at yahoo.com Thu Apr 12 14:50:33 2018 From: aliberkkahraman at yahoo.com (Ali Berk Kahraman) Date: Thu, 12 Apr 2018 22:50:33 +0300 Subject: [petsc-users] Finding NullSpaces of Jacobian and TransposeJacobian In-Reply-To: <6C4AF03B-2BEB-4DCA-B46F-B3EF4A50F4C0@mcs.anl.gov> References: <4dc98b7e-d785-ab14-da44-3dccea879b34@yahoo.com> <6359a517-6342-6b98-e7f4-4a8c8138a854@yahoo.com> <7E5D7BFC-954F-41EC-A457-9FB8D61C88ED@anl.gov> <756a56c7-b969-25de-cd56-b1b64d7c8c72@yahoo.com> <0632EF1C-D155-4D23-B354-C3C410D9A90F@mcs.anl.gov> <354b488d-f74d-bfce-1bff-741dd815c537@yahoo.com> <6C4AF03B-2BEB-4DCA-B46F-B3EF4A50F4C0@mcs.anl.gov> Message-ID: In the very first e-mail, I have a scheme at hand. But sometimes it works, sometimes it does not. I have tried KSPLSQR with PCSOR, and it diverged with DIVERGED_BREAKDOWN where GMRES with PCSOR converges. When I tried it without going through the scheme in the first e-mail, again with PCSOR and default parallel PC, it diverged again with DIVERGED_BREAKDOWN. Thank you very much for your interest so far. Ali On 12-04-2018 21:33, Smith, Barry F. wrote: > Unfortunately I don't have a good answer for you. > > The manual page for KSPCGNE says one should use KSPLSQR for least squares problems. This one hopefully has the correct convergence test for inconsistent systems built in. Please try this solver and let us know how it goes. > > Barry > > > >> On Apr 12, 2018, at 11:13 AM, Ali Berk Kahraman wrote: >> >> That was I was trying to ask, sorry for my vagueness. What kind of a convergence test should I conduct? Correct me if I am mistaken, the KSP object compares the current residual norm with iteration 0. If it reaches rtol, it declares convergence. What should I do to make it compare the residual norm with the previous iteration, and not iteration 0? >> >> On 12-04-2018 19:00, Smith, Barry F. wrote: >>>> On Apr 12, 2018, at 10:56 AM, Ali Berk Kahraman wrote: >>>> >>>> How do I tell that to the KSP object? Is there a direct command or a function? >>> Tell the KSP object what? >>> >>> What happens when you run with -ksp_monitor_true_residual ? >>> >>> We may not have a good convergence test for this case, since the default convergence test wants a residual norm near 0. >>> >>> Barry >>> >>>> On 12-04-2018 18:54, Smith, Barry F. wrote: >>>>>> On Apr 12, 2018, at 10:43 AM, Ali Berk Kahraman wrote: >>>>>> >>>>>> Dear Barry, >>>>>> >>>>>> Thank you very very much for your response. >>>>>> >>>>>> Unfortunately, I use finite differences on an unstructured grid, so my matrix is not symmetric. >>>>>> >>>>>> That is exactly why I want to know the null space, so I can remove it from the RHS vector. Or as I understand it from the online documentation, so petsc can solve a least-squares problem using a MatNullSpace object. You suggestion should also work. I am fine with all of these methods. The problem is, I still have to know the left null space, or its norm in your suggestion, to use these methods. >>>>> I don't think you need to know the null space. As I said, just run GMRES until the residual norm becomes fixed (but non-zero) and you have your solution. No need to remove the transpose null space from the right hand side initially. >>>>> >>>>> Barry >>>>> >>>>>> On 12-04-2018 17:37, Smith, Barry F. wrote: >>>>>>> A few thoughts, >>>>>>> >>>>>>> >>>>>>>> On Apr 12, 2018, at 7:47 AM, Ali Berk Kahraman wrote: >>>>>>>> >>>>>>>> Dear All, >>>>>>>> >>>>>>>> I am trying to solve a Poisson problem with a known source function with all Neumann BCs. >>>>>>> If you are using finite elements then the matrix should be symmetric and hence its transpose nullspace is the same as the original matrix. >>>>>>> >>>>>>> Presumably you want to know the transpose null space so you can remove it from the right hand side to get a consistent linear system? With >>>>>>> GMRES you don't need to actually remove it from the right hand side. Just apply GMRES to the inconsistent system, what will happen, if you >>>>>>> use a good preconditioner is that GMRES will converge to a solution that minimizes the residual. So the residual norm will not be zero but the solution to GMRES is a valid solution to the linear system. In fact the "left over" residual will be in the null space of the transpose; in your case it will be the null space (since it is one dimensional). >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>>> Hence, I know that my problem does not always have a solution, at those times it has a transpose nullspace of one vector. If I know that vector, I can inform petsc of its existence using a MatNullSpace object. >>>>>>>> >>>>>>>> Currently, I am trying to find the nullspace vector by solving a linear system where TrJacobian*nullvec = e, but an arbitrary row of the TrJacobian (ith row) is replaced with all zeros except for the ith column, and every entry of e is 0 except for the ith entry which is 1. However, sometimes the KSP that solves this either does not converge, or even if it converges the nullvec it creates does not help with the original problem's ksp. >>>>>>>> >>>>>>>> Is there a more reliable way to find the transpose nullspace of a matrix? >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> >>>>>>>> Ali Berk Kahraman >>>>>>>> M.Sc. Student, Mechanical Eng. >>>>>>>> Bogazici Uni. Istanbul, Turkey >>>>>>>> From bsmith at mcs.anl.gov Thu Apr 12 16:41:38 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 12 Apr 2018 21:41:38 +0000 Subject: [petsc-users] Finding NullSpaces of Jacobian and TransposeJacobian In-Reply-To: References: <4dc98b7e-d785-ab14-da44-3dccea879b34@yahoo.com> <6359a517-6342-6b98-e7f4-4a8c8138a854@yahoo.com> <7E5D7BFC-954F-41EC-A457-9FB8D61C88ED@anl.gov> <756a56c7-b969-25de-cd56-b1b64d7c8c72@yahoo.com> <0632EF1C-D155-4D23-B354-C3C410D9A90F@mcs.anl.gov> <354b488d-f74d-bfce-1bff-741dd815c537@yahoo.com> <6C4AF03B-2BEB-4DCA-B46F-B3EF4A50F4C0@mcs.anl.gov> Message-ID: Please run a case that breaks with LSQR using the option -ksp_view_mat binary -ksp_view_rhs binary and email to petsc-maint at mcs.anl.gov the file called binaryoutput Barry > On Apr 12, 2018, at 2:50 PM, Ali Berk Kahraman wrote: > > In the very first e-mail, I have a scheme at hand. But sometimes it works, sometimes it does not. I have tried KSPLSQR with PCSOR, and it diverged with DIVERGED_BREAKDOWN where GMRES with PCSOR converges. > > When I tried it without going through the scheme in the first e-mail, again with PCSOR and default parallel PC, it diverged again with DIVERGED_BREAKDOWN. > > Thank you very much for your interest so far. > > Ali > > > On 12-04-2018 21:33, Smith, Barry F. wrote: >> Unfortunately I don't have a good answer for you. >> >> The manual page for KSPCGNE says one should use KSPLSQR for least squares problems. This one hopefully has the correct convergence test for inconsistent systems built in. Please try this solver and let us know how it goes. >> >> Barry >> >> >> >>> On Apr 12, 2018, at 11:13 AM, Ali Berk Kahraman wrote: >>> >>> That was I was trying to ask, sorry for my vagueness. What kind of a convergence test should I conduct? Correct me if I am mistaken, the KSP object compares the current residual norm with iteration 0. If it reaches rtol, it declares convergence. What should I do to make it compare the residual norm with the previous iteration, and not iteration 0? >>> >>> On 12-04-2018 19:00, Smith, Barry F. wrote: >>>>> On Apr 12, 2018, at 10:56 AM, Ali Berk Kahraman wrote: >>>>> >>>>> How do I tell that to the KSP object? Is there a direct command or a function? >>>> Tell the KSP object what? >>>> >>>> What happens when you run with -ksp_monitor_true_residual ? >>>> >>>> We may not have a good convergence test for this case, since the default convergence test wants a residual norm near 0. >>>> >>>> Barry >>>> >>>>> On 12-04-2018 18:54, Smith, Barry F. wrote: >>>>>>> On Apr 12, 2018, at 10:43 AM, Ali Berk Kahraman wrote: >>>>>>> >>>>>>> Dear Barry, >>>>>>> >>>>>>> Thank you very very much for your response. >>>>>>> >>>>>>> Unfortunately, I use finite differences on an unstructured grid, so my matrix is not symmetric. >>>>>>> >>>>>>> That is exactly why I want to know the null space, so I can remove it from the RHS vector. Or as I understand it from the online documentation, so petsc can solve a least-squares problem using a MatNullSpace object. You suggestion should also work. I am fine with all of these methods. The problem is, I still have to know the left null space, or its norm in your suggestion, to use these methods. >>>>>> I don't think you need to know the null space. As I said, just run GMRES until the residual norm becomes fixed (but non-zero) and you have your solution. No need to remove the transpose null space from the right hand side initially. >>>>>> >>>>>> Barry >>>>>> >>>>>>> On 12-04-2018 17:37, Smith, Barry F. wrote: >>>>>>>> A few thoughts, >>>>>>>> >>>>>>>> >>>>>>>>> On Apr 12, 2018, at 7:47 AM, Ali Berk Kahraman wrote: >>>>>>>>> >>>>>>>>> Dear All, >>>>>>>>> >>>>>>>>> I am trying to solve a Poisson problem with a known source function with all Neumann BCs. >>>>>>>> If you are using finite elements then the matrix should be symmetric and hence its transpose nullspace is the same as the original matrix. >>>>>>>> >>>>>>>> Presumably you want to know the transpose null space so you can remove it from the right hand side to get a consistent linear system? With >>>>>>>> GMRES you don't need to actually remove it from the right hand side. Just apply GMRES to the inconsistent system, what will happen, if you >>>>>>>> use a good preconditioner is that GMRES will converge to a solution that minimizes the residual. So the residual norm will not be zero but the solution to GMRES is a valid solution to the linear system. In fact the "left over" residual will be in the null space of the transpose; in your case it will be the null space (since it is one dimensional). >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>>> Hence, I know that my problem does not always have a solution, at those times it has a transpose nullspace of one vector. If I know that vector, I can inform petsc of its existence using a MatNullSpace object. >>>>>>>>> >>>>>>>>> Currently, I am trying to find the nullspace vector by solving a linear system where TrJacobian*nullvec = e, but an arbitrary row of the TrJacobian (ith row) is replaced with all zeros except for the ith column, and every entry of e is 0 except for the ith entry which is 1. However, sometimes the KSP that solves this either does not converge, or even if it converges the nullvec it creates does not help with the original problem's ksp. >>>>>>>>> >>>>>>>>> Is there a more reliable way to find the transpose nullspace of a matrix? >>>>>>>>> >>>>>>>>> Best Regards, >>>>>>>>> >>>>>>>>> Ali Berk Kahraman >>>>>>>>> M.Sc. Student, Mechanical Eng. >>>>>>>>> Bogazici Uni. Istanbul, Turkey >>>>>>>>> > From hbuesing at eonerc.rwth-aachen.de Fri Apr 13 03:14:45 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Fri, 13 Apr 2018 08:14:45 +0000 Subject: [petsc-users] Optimal coloring using -snes_fd_color with DMDACreate In-Reply-To: References: Message-ID: Great! Thank you! I will have a look at the code and I will also check how much function evaluations are needed in my case. Henrik -- Dipl.-Math. Henrik B?sing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 http://www.eonerc.rwth-aachen.de/GGE hbuesing at eonerc.rwth-aachen.de -----Urspr?ngliche Nachricht----- Von: Smith, Barry F. Gesendet: Donnerstag, 12. April 2018 16:42 An: Buesing, Henrik Cc: petsc-users Betreff: Re: [petsc-users] Optimal coloring using -snes_fd_color with DMDACreate I'm not familiar with that paper so I am not sure if it produces exactly the same colorings. It is based on the same principle. The coloring is done in /src/dm/impls/da/fdda.c DMCreateColoring_DA_3d_MPIAIJ() so you could compare. Note if they have more efficient special cases in the paper they could be added to the code without too much difficulty. Barry > On Apr 12, 2018, at 8:26 AM, Buesing, Henrik wrote: > > Dear all, > > When I use ?snes_fd_color with my da created by DMDACreate3d, will it use an optimal coloring algorithm (see [1]) to estimate the Jacobian? > > Thank you! > Henrik > > > > [1] > Goldfarb/Toint: Optimal estimation of Jacobian and Hessian matrices that arise in finite difference calculations > https://doi.org/10.1090/S0025-5718-1984-0744925-5 > > > -- > Dipl.-Math. Henrik B?sing > Institute for Applied Geophysics and Geothermal Energy > E.ON Energy Research Center > RWTH Aachen University > ------------------------------------------------------ > Mathieustr. 10 | Tel +49 (0)241 80 49907 > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > ------------------------------------------------------ > http://www.eonerc.rwth-aachen.de/GGE > hbuesing at eonerc.rwth-aachen.de > ------------------------------------------------------ From dayedut123 at 163.com Fri Apr 13 04:45:29 2018 From: dayedut123 at 163.com (=?GBK?B?ztI=?=) Date: Fri, 13 Apr 2018 17:45:29 +0800 (CST) Subject: [petsc-users] Change the matrix row order In-Reply-To: <88747128-D9EE-48A2-9E48-36399373CA3E@anl.gov> References: <8c41787.11671b0.162afcb0256.Coremail.dayedut123@163.com> <88747128-D9EE-48A2-9E48-36399373CA3E@anl.gov> Message-ID: <638bddf5.f4d2.162be654040.Coremail.dayedut123@163.com> Thanks for your reply! And sorry for my late answer. I want to parallel a particle-method code and solve Ax=b in each timestep. In this code, each partition forms a CSR matrix and assembles the final coefficient matrix by MatCreateMPIAIJWithArrays. But, the domain decomposition makes that the particles' order doesn't follow the original order. In other words, Solution Vector x follows the original order but Vector b follows a new order due to domain decomposition. So I want to change the Vector b's order and Matrix A's row order to follow the origin order. I have been considering to change my parallel code in these days, but I didn't find a best way. I'm considering to multiply a renumbering identity matrix at both sides of the equation with function MatMatMult. Do you have any suggestions for this problem? If the function MatMatMult can influence the parallel efficiency obviously? Thanks again! Daye At 2018-04-10 22:07:04, "Smith, Barry F." wrote: > > When and where do you want to change the row order? > > For PETSc and external factorization based solvers this is handled automatically, no need to do anything. > > The ordering of matrix rows corresponds to the ordering of the vectors; almost always one partitions and orders the vectors based on the mesh that underlies the problem and there is no reason to reorder once the layout is initially determined. > > Barry > > > >> On Apr 10, 2018, at 8:42 AM, ? wrote: >> >> Hello, >> I want to change the row order of a sparse matrix. Are there any build-in functions and suggestions? >> Thank you very much! >> Daye >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 13 07:14:56 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 13 Apr 2018 08:14:56 -0400 Subject: [petsc-users] Change the matrix row order In-Reply-To: <638bddf5.f4d2.162be654040.Coremail.dayedut123@163.com> References: <8c41787.11671b0.162afcb0256.Coremail.dayedut123@163.com> <88747128-D9EE-48A2-9E48-36399373CA3E@anl.gov> <638bddf5.f4d2.162be654040.Coremail.dayedut123@163.com> Message-ID: On Fri, Apr 13, 2018 at 5:45 AM, ? wrote: > > Thanks for your reply! And sorry for my late answer. I want to parallel a > particle-method code and solve Ax=b in each timestep. In this code, each > partition forms a CSR matrix and assembles the final coefficient matrix by > MatCreateMPIAIJWithArrays. But, the domain decomposition makes that the > particles' order doesn't follow the original order. In other words, > Solution Vector x follows the original order but Vector b follows a new > order due to domain decomposition. So I want to change the Vector b's order > and Matrix A's row order to follow the origin order. I have been > considering to change my parallel code in these days, but I didn't find a > best way. I'm considering to multiply a renumbering identity matrix at both > sides of the equation with function MatMatMult. Do you have any suggestions > for this problem? If the function MatMatMult can influence the parallel > efficiency obviously? > I would not do this. The amount of communication to assemble those things would most likely swamp the speedup you gained. If anything, reoder x. However, why would you need to be the original order ever? Just use the order after partition. That is how parallel codes work for the most part. Thanks, Matt > Thanks again! > Daye > > > > > At 2018-04-10 22:07:04, "Smith, Barry F." wrote: > > > > When and where do you want to change the row order? > > > > For PETSc and external factorization based solvers this is handled automatically, no need to do anything. > > > > The ordering of matrix rows corresponds to the ordering of the vectors; almost always one partitions and orders the vectors based on the mesh that underlies the problem and there is no reason to reorder once the layout is initially determined. > > > > Barry > > > > > > > >> On Apr 10, 2018, at 8:42 AM, ? wrote: > >> > >> Hello, > >> I want to change the row order of a sparse matrix. Are there any build-in functions and suggestions? > >> Thank you very much! > >> Daye > >> > >> > >> > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From moocow.bovine at gmail.com Fri Apr 13 07:25:13 2018 From: moocow.bovine at gmail.com (Bryan Jurish) Date: Fri, 13 Apr 2018 14:25:13 +0200 Subject: [petsc-users] teething troubles / MatView() fails silently for transposed seqaij matrix Message-ID: morning all, Apologies for what is very probably a boundlessly stupid question. I've just begun exploring PETSc, but find myself stumped by a pretty trivial task. I would like to load a real binary seqaij matrix and save its transposition (also as a binary seqaij matrix) to a different file. I've tried PETSc v3.7.5 on debian stretch and v3.6.2 on ubuntu 16.04, both with no joy. In both cases the program runs to completion and exits normally, but no output file ("INFILE.T") is written. Stepping through MatView() with the debugger, it seems that (mat->ops->view) is not set for the transposition I created with MatCreateTranspose(). I suppose I could always loop over the input indices and assemble a physical transposition manually, but that rather defeats the purpose of the exercise... can anyone here tell me what I'm doing wrong (almost certainly something fundamental and obvious)? Thanks in advance for any help you can provide. marmosets, Bryan ::: BEGIN MINIMUM BROKEN EXAMPLE ::: /*-*- Mode: C -*-*/ static char help[] = "Transposes a matrix in BINARY format\n\ Input parameters are:\n\ -file INFILE : input matrix in binary format\n\ -out OUTFILE : write output to OUTFILE (default=INFILE.T)\n\ "; #include //-- handle v3.7 API change #if PETSC_VERSION_MAJOR>3 || (PETSC_VERSION_MAJOR==3 && PETSC_VERSION_MINOR>=7) #define OPTIONS_NULL NULL,NULL #else #define OPTIONS_NULL NULL #endif #undef __FUNCT__ #define __FUNCT__ "main" int main(int argc,char **args) { Mat A,B; char infile[PETSC_MAX_PATH_LEN], outfile[PETSC_MAX_PATH_LEN]; PetscErrorCode ierr; PetscInt m,n; PetscViewer view; PetscBool flg_file,flg_out; PetscMPIInt size; PetscInitialize(&argc,&args,(char*)0,help); ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD," MPI_Comm_size reports %d process(es) available.\n", size);CHKERRQ(ierr); if (size != 1) SETERRQ(PETSC_COMM_WORLD,PETSC_ERR_SUP,"This is a uniprocessor operation only!"); /*-- Read in source matrix, binary --*/ ierr = PetscOptionsGetString(OPTIONS_NULL,"-file",infile,PETSC_MAX_PATH_LEN,&flg_file);CHKERRQ(ierr); if (!flg_file) SETERRQ(PETSC_COMM_WORLD,1,"Must specify a file name with the -file option"); #if defined(PETSC_USE_COMPLEX) ierr = PetscPrintf(PETSC_COMM_WORLD," Reading COMPLEX matrix from binary file `%s'...\n", infile);CHKERRQ(ierr); #else ierr = PetscPrintf(PETSC_COMM_WORLD," Reading REAL matrix from binary file `%s'...\n", infile);CHKERRQ(ierr); #endif ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,infile,FILE_MODE_READ,&view);CHKERRQ(ierr); ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); ierr = MatLoad(A,view);CHKERRQ(ierr); ierr = PetscViewerDestroy(&view);CHKERRQ(ierr); // ierr = MatGetSize(A,&m,&n); ierr = PetscPrintf(PETSC_COMM_WORLD," Read input matrix of size (m=%D,n=%D)\n", m,n);CHKERRQ(ierr); /*-- guts: transpose --*/ ierr = PetscPrintf(PETSC_COMM_WORLD," Performing transposition (constructive)\n");CHKERRQ(ierr); ierr = MatCreateTranspose(A,&B);CHKERRQ(ierr); /*-- dump transposed matrix --*/ ierr = PetscOptionsGetString(OPTIONS_NULL,"-out",outfile,PETSC_MAX_PATH_LEN,&flg_out);CHKERRQ(ierr); if (!flg_out) { strcpy(outfile,infile); strcat(outfile,".T"); } ierr = PetscPrintf(PETSC_COMM_SELF," Writing transposed matrix in binary format to '%s' ...\n", outfile);CHKERRQ(ierr); ierr = PetscViewerBinaryOpen(PETSC_COMM_SELF,outfile,FILE_MODE_WRITE,&view);CHKERRQ(ierr); ierr = MatView(B,view);CHKERRQ(ierr); /* !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! * output file is not written: much wailing and gnashing of teeth * !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! */ /*-- cleanup --*/ ierr = MatDestroy(&A);CHKERRQ(ierr); ierr = MatDestroy(&B);CHKERRQ(ierr); ierr = PetscViewerDestroy(&view);CHKERRQ(ierr); ierr = PetscFinalize(); return 0; } ::: END MINIMUM BROKEN EXAMPLE ::: -- Bryan Jurish "There is *always* one more bug." moocow.bovine at gmail.com -Lubarsky's Law of Cybernetic Entomology -------------- next part -------------- An HTML attachment was scrubbed... URL: From jczhang at mcs.anl.gov Fri Apr 13 08:46:06 2018 From: jczhang at mcs.anl.gov (Junchao Zhang) Date: Fri, 13 Apr 2018 08:46:06 -0500 Subject: [petsc-users] teething troubles / MatView() fails silently for transposed seqaij matrix In-Reply-To: References: Message-ID: Looks like MatCreateTranspose does not form the transpose, See the notes here . You may want to use MatTranspose . --Junchao Zhang On Fri, Apr 13, 2018 at 7:25 AM, Bryan Jurish wrote: > morning all, > > Apologies for what is very probably a boundlessly stupid question. I've > just begun exploring PETSc, but find myself stumped by a pretty trivial > task. I would like to load a real binary seqaij matrix and save its > transposition (also as a binary seqaij matrix) to a different file. > I've tried PETSc v3.7.5 on debian stretch and v3.6.2 on ubuntu 16.04, both > with no joy. In both cases the program runs to completion and exits > normally, but no output file ("INFILE.T") is written. Stepping through > MatView() with the debugger, it seems that (mat->ops->view) is not set for > the transposition I created with MatCreateTranspose(). I suppose I could > always loop over the input indices and assemble a physical transposition > manually, but that rather defeats the purpose of the exercise... can anyone > here tell me what I'm doing wrong (almost certainly something fundamental > and obvious)? Thanks in advance for any help you can provide. > > marmosets, > Bryan > > ::: BEGIN MINIMUM BROKEN EXAMPLE ::: > /*-*- Mode: C -*-*/ > > static char help[] = "Transposes a matrix in BINARY format\n\ > Input parameters are:\n\ > -file INFILE : input matrix in binary format\n\ > -out OUTFILE : write output to OUTFILE (default=INFILE.T)\n\ > "; > > > #include > > //-- handle v3.7 API change > #if PETSC_VERSION_MAJOR>3 || (PETSC_VERSION_MAJOR==3 && > PETSC_VERSION_MINOR>=7) > #define OPTIONS_NULL NULL,NULL > #else > #define OPTIONS_NULL NULL > #endif > > > #undef __FUNCT__ > #define __FUNCT__ "main" > int main(int argc,char **args) > { > Mat A,B; > char infile[PETSC_MAX_PATH_LEN], outfile[PETSC_MAX_PATH_LEN]; > PetscErrorCode ierr; > PetscInt m,n; > PetscViewer view; > PetscBool flg_file,flg_out; > PetscMPIInt size; > > PetscInitialize(&argc,&args,(char*)0,help); > ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD," MPI_Comm_size reports %d > process(es) available.\n", size);CHKERRQ(ierr); > if (size != 1) SETERRQ(PETSC_COMM_WORLD,PETSC_ERR_SUP,"This is a > uniprocessor operation only!"); > > /*-- Read in source matrix, binary --*/ > ierr = PetscOptionsGetString(OPTIONS_NULL,"-file",infile,PETSC_MAX_ > PATH_LEN,&flg_file);CHKERRQ(ierr); > if (!flg_file) SETERRQ(PETSC_COMM_WORLD,1,"Must specify a file name > with the -file option"); > > #if defined(PETSC_USE_COMPLEX) > ierr = PetscPrintf(PETSC_COMM_WORLD," Reading COMPLEX matrix from binary > file `%s'...\n", infile);CHKERRQ(ierr); > #else > ierr = PetscPrintf(PETSC_COMM_WORLD," Reading REAL matrix from binary > file `%s'...\n", infile);CHKERRQ(ierr); > #endif > ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,infile,FILE_MODE_ > READ,&view);CHKERRQ(ierr); > ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); > ierr = MatSetFromOptions(A);CHKERRQ(ierr); > ierr = MatLoad(A,view);CHKERRQ(ierr); > ierr = PetscViewerDestroy(&view);CHKERRQ(ierr); > // > ierr = MatGetSize(A,&m,&n); > ierr = PetscPrintf(PETSC_COMM_WORLD," Read input matrix of size > (m=%D,n=%D)\n", m,n);CHKERRQ(ierr); > > /*-- guts: transpose --*/ > ierr = PetscPrintf(PETSC_COMM_WORLD," Performing transposition > (constructive)\n");CHKERRQ(ierr); > ierr = MatCreateTranspose(A,&B);CHKERRQ(ierr); > > /*-- dump transposed matrix --*/ > ierr = PetscOptionsGetString(OPTIONS_NULL,"-out",outfile,PETSC_MAX_ > PATH_LEN,&flg_out);CHKERRQ(ierr); > if (!flg_out) { > strcpy(outfile,infile); > strcat(outfile,".T"); > } > ierr = PetscPrintf(PETSC_COMM_SELF," Writing transposed matrix in binary > format to '%s' ...\n", outfile);CHKERRQ(ierr); > ierr = PetscViewerBinaryOpen(PETSC_COMM_SELF,outfile,FILE_MODE_ > WRITE,&view);CHKERRQ(ierr); > ierr = MatView(B,view);CHKERRQ(ierr); > > /* !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > * output file is not written: much wailing and gnashing of teeth > * !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > */ > > /*-- cleanup --*/ > ierr = MatDestroy(&A);CHKERRQ(ierr); > ierr = MatDestroy(&B);CHKERRQ(ierr); > ierr = PetscViewerDestroy(&view);CHKERRQ(ierr); > ierr = PetscFinalize(); > return 0; > } > ::: END MINIMUM BROKEN EXAMPLE ::: > > -- > Bryan Jurish "There is *always* one more bug." > moocow.bovine at gmail.com -Lubarsky's Law of Cybernetic Entomology > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srossi at email.unc.edu Fri Apr 13 09:50:05 2018 From: srossi at email.unc.edu (Rossi, Simone) Date: Fri, 13 Apr 2018 14:50:05 +0000 Subject: [petsc-users] FIELDSPLIT preconditioner in parallel Message-ID: Hello, I'm trying to use PCFIELDSPLIT to solve a two by two block system and it is working pretty nicely (thanks a lot for that!). I'm still having some issues in parallel though: the ksp solver throws an error when one of the fields does not have DOFs on a processor. Is there a way around it? Thanks a lot for the help, Simone -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 13 09:52:13 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 13 Apr 2018 14:52:13 +0000 Subject: [petsc-users] FIELDSPLIT preconditioner in parallel In-Reply-To: References: Message-ID: 1) Is this with the latest PETSc? some of these issues may have been fixed. 2) Please send use something that reproduces the problem and we will fix it. We aren't aware of this problem. Barry > On Apr 13, 2018, at 9:50 AM, Rossi, Simone wrote: > > Hello, > I'm trying to use PCFIELDSPLIT to solve a two by two block system and it is working pretty nicely (thanks a lot for that!). > I'm still having some issues in parallel though: the ksp solver throws an error when one of the fields does not have DOFs on a processor. Is there a way around it? > Thanks a lot for the help, > Simone From srossi at email.unc.edu Fri Apr 13 09:56:56 2018 From: srossi at email.unc.edu (Rossi, Simone) Date: Fri, 13 Apr 2018 14:56:56 +0000 Subject: [petsc-users] FIELDSPLIT preconditioner in parallel In-Reply-To: References: , Message-ID: I have not tried with the latest version yet. I found this problem in v3.7.7. I'll try that and let you know. If the problem is still there, I'll send you an example. Thanks for the help, Simone ________________________________ From: Smith, Barry F. Sent: Friday, April 13, 2018 10:52:13 AM To: Rossi, Simone Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] FIELDSPLIT preconditioner in parallel 1) Is this with the latest PETSc? some of these issues may have been fixed. 2) Please send use something that reproduces the problem and we will fix it. We aren't aware of this problem. Barry > On Apr 13, 2018, at 9:50 AM, Rossi, Simone wrote: > > Hello, > I'm trying to use PCFIELDSPLIT to solve a two by two block system and it is working pretty nicely (thanks a lot for that!). > I'm still having some issues in parallel though: the ksp solver throws an error when one of the fields does not have DOFs on a processor. Is there a way around it? > Thanks a lot for the help, > Simone -------------- next part -------------- An HTML attachment was scrubbed... URL: From srossi at email.unc.edu Fri Apr 13 10:45:18 2018 From: srossi at email.unc.edu (Rossi, Simone) Date: Fri, 13 Apr 2018 15:45:18 +0000 Subject: [petsc-users] FIELDSPLIT preconditioner in parallel In-Reply-To: References: , Message-ID: It looks like the problem was solved in one of the updates. I cannot reproduce that behavior with the latest PETSc. Thanks! I'll let you know if it happens again. Simone ________________________________ From: Smith, Barry F. Sent: Friday, April 13, 2018 10:52:13 AM To: Rossi, Simone Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] FIELDSPLIT preconditioner in parallel 1) Is this with the latest PETSc? some of these issues may have been fixed. 2) Please send use something that reproduces the problem and we will fix it. We aren't aware of this problem. Barry > On Apr 13, 2018, at 9:50 AM, Rossi, Simone wrote: > > Hello, > I'm trying to use PCFIELDSPLIT to solve a two by two block system and it is working pretty nicely (thanks a lot for that!). > I'm still having some issues in parallel though: the ksp solver throws an error when one of the fields does not have DOFs on a processor. Is there a way around it? > Thanks a lot for the help, > Simone -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlwalker at usc.edu Fri Apr 13 12:40:05 2018 From: rlwalker at usc.edu (Robert Walker) Date: Fri, 13 Apr 2018 10:40:05 -0700 Subject: [petsc-users] PETScFE Point Source Message-ID: Hello, Is there an example of application of a point source in one of the tutorial PETScFE style examples? Ideally this would be some point on the interior, and not necessarily on a boundary. Thanks, and apologies in advance if this is a stupid question, Robert Robert L. Walker MS Petroleum Engineering Mork Family Department of Chemicals and Materials Sciences University of Southern California ---------------------------------------------- Mobile US: +1 (213) - 290 -7101 Mobile EU: +34 62 274 66 40 rlwalker at usc.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 13 12:46:11 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 13 Apr 2018 13:46:11 -0400 Subject: [petsc-users] PETScFE Point Source In-Reply-To: References: Message-ID: On Fri, Apr 13, 2018 at 1:40 PM, Robert Walker wrote: > Hello, > > Is there an example of application of a point source in one of the > tutorial PETScFE style examples? Ideally this would be some point on the > interior, and not necessarily on a boundary. > > Thanks, and apologies in advance if this is a stupid question, > No, good question. You would have to decide what that meant in FEM terms. All the internal integrals are done with quadrature. So putting the source at a quadrature point would work, although its a little difficult to know where they will be. You could smooth it out a little, and check coordinates in the forcing function. Last, you could check for a "close enough" quadrature point and put it there. I think making a slightly extended source is usually best. Thanks, Matt > Robert > > > Robert L. Walker > MS Petroleum Engineering > Mork Family Department of Chemicals and Materials Sciences > University of Southern California > ---------------------------------------------- > Mobile US: +1 (213) - 290 -7101 > Mobile EU: +34 62 274 66 40 > rlwalker at usc.edu > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlwalker at usc.edu Fri Apr 13 12:51:13 2018 From: rlwalker at usc.edu (Robert Walker) Date: Fri, 13 Apr 2018 10:51:13 -0700 Subject: [petsc-users] PETScFE Point Source In-Reply-To: References: Message-ID: Ok, let me rephrase that...something like way FEniCS (if i remember correctly) handles this ... between two values with a very small difference between them on each axis? Is there an example of something like that to use as a security blanket from which to expand? Robert Robert L. Walker MS Petroleum Engineering Mork Family Department of Chemicals and Materials Sciences University of Southern California ---------------------------------------------- Mobile US: +1 (213) - 290 -7101 Mobile EU: +34 62 274 66 40 rlwalker at usc.edu On Fri, Apr 13, 2018 at 10:46 AM, Matthew Knepley wrote: > On Fri, Apr 13, 2018 at 1:40 PM, Robert Walker wrote: > >> Hello, >> >> Is there an example of application of a point source in one of the >> tutorial PETScFE style examples? Ideally this would be some point on the >> interior, and not necessarily on a boundary. >> >> Thanks, and apologies in advance if this is a stupid question, >> > > No, good question. You would have to decide what that meant in FEM terms. > All the internal integrals are done > with quadrature. So putting the source at a quadrature point would work, > although its a little difficult to know where > they will be. You could smooth it out a little, and check coordinates in > the forcing function. Last, you could check > for a "close enough" quadrature point and put it there. I think making a > slightly extended source is usually best. > > Thanks, > > Matt > > >> Robert >> >> >> Robert L. Walker >> MS Petroleum Engineering >> Mork Family Department of Chemicals and Materials Sciences >> University of Southern California >> ---------------------------------------------- >> Mobile US: +1 (213) - 290 -7101 >> Mobile EU: +34 62 274 66 40 >> rlwalker at usc.edu >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 13 13:01:20 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 13 Apr 2018 14:01:20 -0400 Subject: [petsc-users] PETScFE Point Source In-Reply-To: References: Message-ID: On Fri, Apr 13, 2018 at 1:51 PM, Robert Walker wrote: > Ok, let me rephrase that...something like way FEniCS (if i remember > correctly) handles this ... between two values with a very small difference > between them on each axis? Is there an example of something like that to > use as a security blanket from which to expand? > I would start with a Gaussian source at some point, and then push it toward the limit. That way you know all the correct normalization, and its smooth. Matt > Robert > > Robert L. Walker > MS Petroleum Engineering > Mork Family Department of Chemicals and Materials Sciences > University of Southern California > ---------------------------------------------- > Mobile US: +1 (213) - 290 -7101 > Mobile EU: +34 62 274 66 40 > rlwalker at usc.edu > > On Fri, Apr 13, 2018 at 10:46 AM, Matthew Knepley > wrote: > >> On Fri, Apr 13, 2018 at 1:40 PM, Robert Walker wrote: >> >>> Hello, >>> >>> Is there an example of application of a point source in one of the >>> tutorial PETScFE style examples? Ideally this would be some point on the >>> interior, and not necessarily on a boundary. >>> >>> Thanks, and apologies in advance if this is a stupid question, >>> >> >> No, good question. You would have to decide what that meant in FEM terms. >> All the internal integrals are done >> with quadrature. So putting the source at a quadrature point would work, >> although its a little difficult to know where >> they will be. You could smooth it out a little, and check coordinates in >> the forcing function. Last, you could check >> for a "close enough" quadrature point and put it there. I think making a >> slightly extended source is usually best. >> >> Thanks, >> >> Matt >> >> >>> Robert >>> >>> >>> Robert L. Walker >>> MS Petroleum Engineering >>> Mork Family Department of Chemicals and Materials Sciences >>> University of Southern California >>> ---------------------------------------------- >>> Mobile US: +1 (213) - 290 -7101 >>> Mobile EU: +34 62 274 66 40 >>> rlwalker at usc.edu >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Fri Apr 13 13:07:35 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Fri, 13 Apr 2018 18:07:35 +0000 Subject: [petsc-users] PETScFE Point Source In-Reply-To: References: Message-ID: What do you mean by point sources? Dirac's delta terms ? If so, these translates into basis function evaluations , and should not be hard to support it, but it will require a different API. Matt? Il Ven 13 Apr 2018, 20:51 Robert Walker ha scritto: > Ok, let me rephrase that...something like way FEniCS (if i remember > correctly) handles this ... between two values with a very small difference > between them on each axis? Is there an example of something like that to > use as a security blanket from which to expand? > > Robert > > Robert L. Walker > MS Petroleum Engineering > Mork Family Department of Chemicals and Materials Sciences > University of Southern California > ---------------------------------------------- > Mobile US: +1 (213) - 290 -7101 > Mobile EU: +34 62 274 66 40 > rlwalker at usc.edu > > On Fri, Apr 13, 2018 at 10:46 AM, Matthew Knepley > wrote: > >> On Fri, Apr 13, 2018 at 1:40 PM, Robert Walker wrote: >> >>> Hello, >>> >>> Is there an example of application of a point source in one of the >>> tutorial PETScFE style examples? Ideally this would be some point on the >>> interior, and not necessarily on a boundary. >>> >>> Thanks, and apologies in advance if this is a stupid question, >>> >> >> No, good question. You would have to decide what that meant in FEM terms. >> All the internal integrals are done >> with quadrature. So putting the source at a quadrature point would work, >> although its a little difficult to know where >> they will be. You could smooth it out a little, and check coordinates in >> the forcing function. Last, you could check >> for a "close enough" quadrature point and put it there. I think making a >> slightly extended source is usually best. >> >> Thanks, >> >> Matt >> >> >>> Robert >>> >>> >>> Robert L. Walker >>> MS Petroleum Engineering >>> Mork Family Department of Chemicals and Materials Sciences >>> University of Southern California >>> ---------------------------------------------- >>> Mobile US: +1 (213) - 290 -7101 >>> Mobile EU: +34 62 274 66 40 >>> rlwalker at usc.edu >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 13 13:13:25 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 13 Apr 2018 14:13:25 -0400 Subject: [petsc-users] PETScFE Point Source In-Reply-To: References: Message-ID: On Fri, Apr 13, 2018 at 2:07 PM, Stefano Zampini wrote: > What do you mean by point sources? Dirac's delta terms ? If so, these > translates into basis function evaluations , and should not be hard to > support it, but it will require a different API. Matt? > That is another thing you could do. We will do that shortly. Matt > Il Ven 13 Apr 2018, 20:51 Robert Walker ha scritto: > >> Ok, let me rephrase that...something like way FEniCS (if i remember >> correctly) handles this ... between two values with a very small difference >> between them on each axis? Is there an example of something like that to >> use as a security blanket from which to expand? >> >> Robert >> >> Robert L. Walker >> MS Petroleum Engineering >> Mork Family Department of Chemicals and Materials Sciences >> University of Southern California >> ---------------------------------------------- >> Mobile US: +1 (213) - 290 -7101 >> Mobile EU: +34 62 274 66 40 >> rlwalker at usc.edu >> >> On Fri, Apr 13, 2018 at 10:46 AM, Matthew Knepley >> wrote: >> >>> On Fri, Apr 13, 2018 at 1:40 PM, Robert Walker wrote: >>> >>>> Hello, >>>> >>>> Is there an example of application of a point source in one of the >>>> tutorial PETScFE style examples? Ideally this would be some point on the >>>> interior, and not necessarily on a boundary. >>>> >>>> Thanks, and apologies in advance if this is a stupid question, >>>> >>> >>> No, good question. You would have to decide what that meant in FEM >>> terms. All the internal integrals are done >>> with quadrature. So putting the source at a quadrature point would work, >>> although its a little difficult to know where >>> they will be. You could smooth it out a little, and check coordinates in >>> the forcing function. Last, you could check >>> for a "close enough" quadrature point and put it there. I think making a >>> slightly extended source is usually best. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Robert >>>> >>>> >>>> Robert L. Walker >>>> MS Petroleum Engineering >>>> Mork Family Department of Chemicals and Materials Sciences >>>> University of Southern California >>>> ---------------------------------------------- >>>> Mobile US: +1 (213) - 290 -7101 >>>> Mobile EU: +34 62 274 66 40 >>>> rlwalker at usc.edu >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Fri Apr 13 13:27:32 2018 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Fri, 13 Apr 2018 20:27:32 +0200 Subject: [petsc-users] PETScFE Point Source In-Reply-To: References: Message-ID: Wouldn't it make the most sense to do it in a variationally consistent manner rhs_a = int{ N_a delta(x - x_o) } where x_o is the location of the point source? -sanjay On 4/13/18 7:46 PM, Matthew Knepley wrote: > On Fri, Apr 13, 2018 at 1:40 PM, Robert Walker > wrote: > > Hello, > > Is there an example of application of a point source in one of the > tutorial? PETScFE style examples? Ideally this would be some point > on the interior, and not necessarily on a boundary. > > Thanks, and apologies in advance if this is a stupid question, > > > No, good question. You would have to decide what that meant in FEM > terms. All the internal integrals are done > with quadrature. So putting the source at a quadrature point would > work, although its a little difficult to know where > they will be. You could smooth it out a little, and check coordinates > in the forcing function. Last, you could check > for a "close enough" quadrature point and put it there. I think making > a slightly extended source is usually best. > > ? Thanks, > > ? ? ?Matt > > Robert > > > Robert L. Walker > MS Petroleum Engineering > Mork Family Department of Chemicals and Materials Sciences > University of Southern California > ---------------------------------------------- > Mobile US: +1 (213) - 290 -7101 > Mobile EU: +34 62 274 66 40 > rlwalker at usc.edu > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -- ------------------------------------------------------------------- Sanjay Govindjee, PhD, PE Horace, Dorothy, and Katherine Johnson Professor in Engineering 779 Davis Hall University of California Berkeley, CA 94720-1710 Voice: +1 510 642 6060 FAX: +1 510 643 5264 s_g at berkeley.edu http://faculty.ce.berkeley.edu/sanjay ------------------------------------------------------------------- Books: Engineering Mechanics of Deformable Solids http://amzn.com/0199651647 Engineering Mechanics 3 (Dynamics) 2nd Edition http://amzn.com/3642537111 Engineering Mechanics 3, Supplementary Problems: Dynamics http://www.amzn.com/B00SOXN8JU NSF NHERI SimCenter https://simcenter.designsafe-ci.org/ ------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Fri Apr 13 13:37:05 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Fri, 13 Apr 2018 18:37:05 +0000 Subject: [petsc-users] PETScFE Point Source In-Reply-To: References: Message-ID: Il Ven 13 Apr 2018, 21:27 Sanjay Govindjee ha scritto: > Wouldn't it make the most sense to do it in a variationally consistent > manner > > rhs_a = int{ N_a delta(x - x_o) } > > where x_o is the location of the point source? > This is what I meant. However, the current callbacks for residual evaluation in PetscDS do not allow to do it. > -sanjay > > On 4/13/18 7:46 PM, Matthew Knepley wrote: > > On Fri, Apr 13, 2018 at 1:40 PM, Robert Walker wrote: > >> Hello, >> >> Is there an example of application of a point source in one of the >> tutorial PETScFE style examples? Ideally this would be some point on the >> interior, and not necessarily on a boundary. >> >> Thanks, and apologies in advance if this is a stupid question, >> > > No, good question. You would have to decide what that meant in FEM terms. > All the internal integrals are done > with quadrature. So putting the source at a quadrature point would work, > although its a little difficult to know where > they will be. You could smooth it out a little, and check coordinates in > the forcing function. Last, you could check > for a "close enough" quadrature point and put it there. I think making a > slightly extended source is usually best. > > Thanks, > > Matt > > >> Robert >> >> >> Robert L. Walker >> MS Petroleum Engineering >> Mork Family Department of Chemicals and Materials Sciences >> University of Southern California >> ---------------------------------------------- >> Mobile US: +1 (213) - 290 -7101 >> Mobile EU: +34 62 274 66 40 >> rlwalker at usc.edu >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > ------------------------------------------------------------------- > Sanjay Govindjee, PhD, PE > Horace, Dorothy, and Katherine Johnson Professor in Engineering > > 779 Davis Hall > University of California > Berkeley, CA 94720-1710 > > Voice: +1 510 642 6060 > FAX: +1 510 643 5264s_g at berkeley.eduhttp://faculty.ce.berkeley.edu/sanjay > ------------------------------------------------------------------- > > Books: > > Engineering Mechanics of Deformable Solidshttp://amzn.com/0199651647 > > Engineering Mechanics 3 (Dynamics) 2nd Editionhttp://amzn.com/3642537111 > > Engineering Mechanics 3, Supplementary Problems: Dynamics http://www.amzn.com/B00SOXN8JU > > NSF NHERI SimCenterhttps://simcenter.designsafe-ci.org/ > ------------------------------------------------------------------- > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 13 13:52:38 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 13 Apr 2018 14:52:38 -0400 Subject: [petsc-users] PETScFE Point Source In-Reply-To: References: Message-ID: On Fri, Apr 13, 2018 at 2:27 PM, Sanjay Govindjee wrote: > Wouldn't it make the most sense to do it in a variationally consistent > manner > > rhs_a = int{ N_a delta(x - x_o) } > > where x_o is the location of the point source? > That is what Stefano is saying. You can do that, it just means an API change. We are doing that for DMSwarm Matt > -sanjay > > On 4/13/18 7:46 PM, Matthew Knepley wrote: > > On Fri, Apr 13, 2018 at 1:40 PM, Robert Walker wrote: > >> Hello, >> >> Is there an example of application of a point source in one of the >> tutorial PETScFE style examples? Ideally this would be some point on the >> interior, and not necessarily on a boundary. >> >> Thanks, and apologies in advance if this is a stupid question, >> > > No, good question. You would have to decide what that meant in FEM terms. > All the internal integrals are done > with quadrature. So putting the source at a quadrature point would work, > although its a little difficult to know where > they will be. You could smooth it out a little, and check coordinates in > the forcing function. Last, you could check > for a "close enough" quadrature point and put it there. I think making a > slightly extended source is usually best. > > Thanks, > > Matt > > >> Robert >> >> >> Robert L. Walker >> MS Petroleum Engineering >> Mork Family Department of Chemicals and Materials Sciences >> University of Southern California >> ---------------------------------------------- >> Mobile US: +1 (213) - 290 -7101 >> Mobile EU: +34 62 274 66 40 >> rlwalker at usc.edu >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > ------------------------------------------------------------------- > Sanjay Govindjee, PhD, PE > Horace, Dorothy, and Katherine Johnson Professor in Engineering > > 779 Davis Hall > University of California > Berkeley, CA 94720-1710 > > Voice: +1 510 642 6060 > FAX: +1 510 643 5264s_g at berkeley.eduhttp://faculty.ce.berkeley.edu/sanjay > ------------------------------------------------------------------- > > Books: > > Engineering Mechanics of Deformable Solidshttp://amzn.com/0199651647 > > Engineering Mechanics 3 (Dynamics) 2nd Editionhttp://amzn.com/3642537111 > > Engineering Mechanics 3, Supplementary Problems: Dynamics http://www.amzn.com/B00SOXN8JU > > NSF NHERI SimCenterhttps://simcenter.designsafe-ci.org/ > ------------------------------------------------------------------- > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 13 13:53:39 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 13 Apr 2018 14:53:39 -0400 Subject: [petsc-users] PETScFE Point Source In-Reply-To: References: Message-ID: On Fri, Apr 13, 2018 at 2:37 PM, Stefano Zampini wrote: > Il Ven 13 Apr 2018, 21:27 Sanjay Govindjee ha scritto: > >> Wouldn't it make the most sense to do it in a variationally consistent >> manner >> >> rhs_a = int{ N_a delta(x - x_o) } >> >> where x_o is the location of the point source? >> > > > This is what I meant. However, the current callbacks for residual > evaluation in PetscDS do not allow to do it. > And I don't think they should since that is not something that can be done by quadrature, so the interface is still unclear. Anyone can code something up ;) Matt > -sanjay >> >> On 4/13/18 7:46 PM, Matthew Knepley wrote: >> >> On Fri, Apr 13, 2018 at 1:40 PM, Robert Walker wrote: >> >>> Hello, >>> >>> Is there an example of application of a point source in one of the >>> tutorial PETScFE style examples? Ideally this would be some point on the >>> interior, and not necessarily on a boundary. >>> >>> Thanks, and apologies in advance if this is a stupid question, >>> >> >> No, good question. You would have to decide what that meant in FEM terms. >> All the internal integrals are done >> with quadrature. So putting the source at a quadrature point would work, >> although its a little difficult to know where >> they will be. You could smooth it out a little, and check coordinates in >> the forcing function. Last, you could check >> for a "close enough" quadrature point and put it there. I think making a >> slightly extended source is usually best. >> >> Thanks, >> >> Matt >> >> >>> Robert >>> >>> >>> Robert L. Walker >>> MS Petroleum Engineering >>> Mork Family Department of Chemicals and Materials Sciences >>> University of Southern California >>> ---------------------------------------------- >>> Mobile US: +1 (213) - 290 -7101 >>> Mobile EU: +34 62 274 66 40 >>> rlwalker at usc.edu >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> -- >> ------------------------------------------------------------------- >> Sanjay Govindjee, PhD, PE >> Horace, Dorothy, and Katherine Johnson Professor in Engineering >> >> 779 Davis Hall >> University of California >> Berkeley, CA 94720-1710 >> >> Voice: +1 510 642 6060 >> FAX: +1 510 643 5264s_g at berkeley.eduhttp://faculty.ce.berkeley.edu/sanjay >> ------------------------------------------------------------------- >> >> Books: >> >> Engineering Mechanics of Deformable Solidshttp://amzn.com/0199651647 >> >> Engineering Mechanics 3 (Dynamics) 2nd Editionhttp://amzn.com/3642537111 >> >> Engineering Mechanics 3, Supplementary Problems: Dynamics http://www.amzn.com/B00SOXN8JU >> >> NSF NHERI SimCenterhttps://simcenter.designsafe-ci.org/ >> ------------------------------------------------------------------- >> >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From moocow.bovine at gmail.com Sat Apr 14 03:27:14 2018 From: moocow.bovine at gmail.com (Bryan Jurish) Date: Sat, 14 Apr 2018 08:27:14 +0000 Subject: [petsc-users] teething troubles / MatView() fails silently for transposed seqaij matrix In-Reply-To: References: Message-ID: hi Junchao, thanks for the hint... I'll give it try when I'm back on my work machine on Monday. marmosets, Bryan -- typed with my opposable thumbs! On Fri, Apr 13, 2018, 15:46 Junchao Zhang wrote: > Looks like MatCreateTranspose does not form the transpose, See the notes > here > . You > may want to use MatTranspose > > . > > --Junchao Zhang > > On Fri, Apr 13, 2018 at 7:25 AM, Bryan Jurish > wrote: > >> morning all, >> >> Apologies for what is very probably a boundlessly stupid question. I've >> just begun exploring PETSc, but find myself stumped by a pretty trivial >> task. I would like to load a real binary seqaij matrix and save its >> transposition (also as a binary seqaij matrix) to a different file. >> I've tried PETSc v3.7.5 on debian stretch and v3.6.2 on ubuntu 16.04, both >> with no joy. In both cases the program runs to completion and exits >> normally, but no output file ("INFILE.T") is written. Stepping through >> MatView() with the debugger, it seems that (mat->ops->view) is not set for >> the transposition I created with MatCreateTranspose(). I suppose I could >> always loop over the input indices and assemble a physical transposition >> manually, but that rather defeats the purpose of the exercise... can anyone >> here tell me what I'm doing wrong (almost certainly something fundamental >> and obvious)? Thanks in advance for any help you can provide. >> >> marmosets, >> Bryan >> >> ::: BEGIN MINIMUM BROKEN EXAMPLE ::: >> /*-*- Mode: C -*-*/ >> >> static char help[] = "Transposes a matrix in BINARY format\n\ >> Input parameters are:\n\ >> -file INFILE : input matrix in binary format\n\ >> -out OUTFILE : write output to OUTFILE (default=INFILE.T)\n\ >> "; >> >> >> #include >> >> //-- handle v3.7 API change >> #if PETSC_VERSION_MAJOR>3 || (PETSC_VERSION_MAJOR==3 && >> PETSC_VERSION_MINOR>=7) >> #define OPTIONS_NULL NULL,NULL >> #else >> #define OPTIONS_NULL NULL >> #endif >> >> >> #undef __FUNCT__ >> #define __FUNCT__ "main" >> int main(int argc,char **args) >> { >> Mat A,B; >> char infile[PETSC_MAX_PATH_LEN], outfile[PETSC_MAX_PATH_LEN]; >> PetscErrorCode ierr; >> PetscInt m,n; >> PetscViewer view; >> PetscBool flg_file,flg_out; >> PetscMPIInt size; >> >> PetscInitialize(&argc,&args,(char*)0,help); >> ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr); >> ierr = PetscPrintf(PETSC_COMM_WORLD," MPI_Comm_size reports %d >> process(es) available.\n", size);CHKERRQ(ierr); >> if (size != 1) SETERRQ(PETSC_COMM_WORLD,PETSC_ERR_SUP,"This is a >> uniprocessor operation only!"); >> >> /*-- Read in source matrix, binary --*/ >> ierr = >> PetscOptionsGetString(OPTIONS_NULL,"-file",infile,PETSC_MAX_PATH_LEN,&flg_file);CHKERRQ(ierr); >> if (!flg_file) SETERRQ(PETSC_COMM_WORLD,1,"Must specify a file name >> with the -file option"); >> >> #if defined(PETSC_USE_COMPLEX) >> ierr = PetscPrintf(PETSC_COMM_WORLD," Reading COMPLEX matrix from >> binary file `%s'...\n", infile);CHKERRQ(ierr); >> #else >> ierr = PetscPrintf(PETSC_COMM_WORLD," Reading REAL matrix from binary >> file `%s'...\n", infile);CHKERRQ(ierr); >> #endif >> ierr = >> PetscViewerBinaryOpen(PETSC_COMM_WORLD,infile,FILE_MODE_READ,&view);CHKERRQ(ierr); >> ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); >> ierr = MatSetFromOptions(A);CHKERRQ(ierr); >> ierr = MatLoad(A,view);CHKERRQ(ierr); >> ierr = PetscViewerDestroy(&view);CHKERRQ(ierr); >> // >> ierr = MatGetSize(A,&m,&n); >> ierr = PetscPrintf(PETSC_COMM_WORLD," Read input matrix of size >> (m=%D,n=%D)\n", m,n);CHKERRQ(ierr); >> >> /*-- guts: transpose --*/ >> ierr = PetscPrintf(PETSC_COMM_WORLD," Performing transposition >> (constructive)\n");CHKERRQ(ierr); >> ierr = MatCreateTranspose(A,&B);CHKERRQ(ierr); >> >> /*-- dump transposed matrix --*/ >> ierr = >> PetscOptionsGetString(OPTIONS_NULL,"-out",outfile,PETSC_MAX_PATH_LEN,&flg_out);CHKERRQ(ierr); >> if (!flg_out) { >> strcpy(outfile,infile); >> strcat(outfile,".T"); >> } >> ierr = PetscPrintf(PETSC_COMM_SELF," Writing transposed matrix in >> binary format to '%s' ...\n", outfile);CHKERRQ(ierr); >> ierr = >> PetscViewerBinaryOpen(PETSC_COMM_SELF,outfile,FILE_MODE_WRITE,&view);CHKERRQ(ierr); >> ierr = MatView(B,view);CHKERRQ(ierr); >> >> /* !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! >> * output file is not written: much wailing and gnashing of teeth >> * !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! >> */ >> >> /*-- cleanup --*/ >> ierr = MatDestroy(&A);CHKERRQ(ierr); >> ierr = MatDestroy(&B);CHKERRQ(ierr); >> ierr = PetscViewerDestroy(&view);CHKERRQ(ierr); >> ierr = PetscFinalize(); >> return 0; >> } >> ::: END MINIMUM BROKEN EXAMPLE ::: >> >> -- >> Bryan Jurish "There is *always* one more bug." >> moocow.bovine at gmail.com -Lubarsky's Law of Cybernetic Entomology >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k_burkart at yahoo.com Mon Apr 16 05:51:26 2018 From: k_burkart at yahoo.com (Klaus Burkart) Date: Mon, 16 Apr 2018 10:51:26 +0000 (UTC) Subject: [petsc-users] preallocate space for matrices if only the global number of nonzeros is known References: <16888007.824157.1523875886094.ref@mail.yahoo.com> Message-ID: <16888007.824157.1523875886094@mail.yahoo.com> Hi, How can I preallocate space for matrices (some symmetric, others asymmetric) if I have the global number of nonzeros (NNZ) but not the number of nonzeros per row? I could compute the NNZ for the upper or lower part separately if this would be useful for symmetric matrices. I create the matrix this way: ??? MatCreate(PETSC_COMM_WORLD, &M); ??? MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); ??? MatSetFromOptions(M); Klaus -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Apr 16 07:22:52 2018 From: jed at jedbrown.org (Jed Brown) Date: Mon, 16 Apr 2018 06:22:52 -0600 Subject: [petsc-users] preallocate space for matrices if only the global number of nonzeros is known In-Reply-To: <16888007.824157.1523875886094@mail.yahoo.com> References: <16888007.824157.1523875886094.ref@mail.yahoo.com> <16888007.824157.1523875886094@mail.yahoo.com> Message-ID: <87fu3vs5lv.fsf@jedbrown.org> Klaus Burkart writes: > Hi, > How can I preallocate space for matrices (some symmetric, others > asymmetric) if I have the global number of nonzeros (NNZ) but not the > number of nonzeros per row? Unfortunately, this is not enough for efficient assembly. It is more efficient to over-estimate. > I could compute the NNZ for the upper or lower part separately if this > would be useful for symmetric matrices. > > I create the matrix this way: > ??? MatCreate(PETSC_COMM_WORLD, &M); > ??? MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > ??? MatSetFromOptions(M); > Klaus From yongxiang27 at gmail.com Mon Apr 16 10:42:03 2018 From: yongxiang27 at gmail.com (Yongxiang Wu) Date: Mon, 16 Apr 2018 17:42:03 +0200 Subject: [petsc-users] memory access out of range with MUMPS Message-ID: Hello everyone, I am using slepc4py solving a generalized eigenvalue problem, with a large spase matrix. The following is my python code. The matrix is first created using PETSc.Mat().createAIJ() and saved to file matrix-A.dat . opts = PETSc.Options() opts.setValue('-st_pc_factor_mat_solver_package','mumps') viewer = PETSc.Viewer().createBinary('matrix-A.dat', 'r',comm=PETSc.COMM_SELF) A = PETSc.Mat().create(comm=PETSc.COMM_SELF) A.load(viewer) A.setFromOptions() A.setUp() Print('load petsc binary finished') t0 = time() E = SLEPc.EPS().create(comm=PETSc.COMM_SELF) E.setOperators(A,None) E.setDimensions(noEigs2solv,PETSc.DECIDE) # set number of eigenvalues to compute E.setWhichEigenpairs(E.Which.TARGET_MAGNITUDE) E.setProblemType(SLEPc.EPS.ProblemType.GNHEP) # generalized non-Hermitian eigenvalue problem E.setTolerances(arnoldiTol,1000) # set tolerance if SIGMA is not None: E.setTarget(SIGMA) # set the desired eigenvalue if options_v0 is not None: E.setInitialSpace(options_v0) # set the initial vector st = E.getST() st.setType('sinvert') E.setFromOptions() Print('Start solving ...') E.solve() My problem here is, when I run this code on cluster, I would end up with the following error. Strange thing is that I used the save library on my local PC, and it works without any error. If any one of you could give me hint what is wrong? Thanks ** ERROR RETURN ** FROM ZMUMPS INFO(1)= -1 ** INFO(2)= 0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscInitializeMUMPS line 1188 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/petsc/src/mat/impls/aij/mpi/mumps/mumps.c [0]PETSC ERROR: [0] MatGetFactor_aij_mumps line 2125 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/petsc/src/mat/impls/aij/mpi/mumps/mumps.c [0]PETSC ERROR: [0] MatGetFactor line 4341 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: [0] PCSetUp_LU line 59 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/petsc/src/ksp/pc/impls/factor/lu/lu.c [0]PETSC ERROR: [0] PCSetUp line 886 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSPSetUp line 294 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] STSetUp_Sinvert line 96 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/slepc-3.8.2/src/sys/classes/st/impls/sinvert/sinvert.c [0]PETSC ERROR: [0] STSetUp line 233 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/slepc-3.8.2/src/sys/classes/st/interface/stsolve.c [0]PETSC ERROR: [0] EPSSetUp line 104 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/slepc-3.8.2/src/eps/interface/epssetup.c [0]PETSC ERROR: [0] EPSSolve line 129 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/slepc-3.8.2/src/eps/interface/epssolve.c application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59 with regards Yong -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Apr 16 10:49:18 2018 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 16 Apr 2018 17:49:18 +0200 Subject: [petsc-users] memory access out of range with MUMPS In-Reply-To: References: Message-ID: <5278FCC6-35C6-4135-B2AC-19A6604C60BD@dsic.upv.es> If you are going to run in parallel then you should not provide the communicator comm=PETSc.COMM_SELF in any object. Jose > El 16 abr 2018, a las 17:42, Yongxiang Wu escribi?: > > Hello everyone, > I am using slepc4py solving a generalized eigenvalue problem, with a large spase matrix. The following is my python code. The matrix is first created using PETSc.Mat().createAIJ() and saved to file matrix-A.dat . > > > opts = PETSc.Options() > opts.setValue('-st_pc_factor_mat_solver_package','mumps') > > viewer = PETSc.Viewer().createBinary('matrix-A.dat', 'r',comm=PETSc.COMM_SELF) > A = PETSc.Mat().create(comm=PETSc.COMM_SELF) > A.load(viewer) > A.setFromOptions() > A.setUp() > > Print('load petsc binary finished') > > t0 = time() > > E = SLEPc.EPS().create(comm=PETSc.COMM_SELF) > E.setOperators(A,None) > E.setDimensions(noEigs2solv,PETSc.DECIDE) # set number of eigenvalues to compute > E.setWhichEigenpairs(E.Which.TARGET_MAGNITUDE) > E.setProblemType(SLEPc.EPS.ProblemType.GNHEP) # generalized non-Hermitian eigenvalue problem > > E.setTolerances(arnoldiTol,1000) # set tolerance > if SIGMA is not None: > E.setTarget(SIGMA) # set the desired eigenvalue > > if options_v0 is not None: > E.setInitialSpace(options_v0) # set the initial vector > > st = E.getST() > st.setType('sinvert') > > E.setFromOptions() > Print('Start solving ...') > E.solve() > > My problem here is, when I run this code on cluster, I would end up with the following error. Strange thing is that I used the save library on my local PC, and it works without any error. If any one of you could give me hint what is wrong? Thanks > > > ** ERROR RETURN ** FROM ZMUMPS INFO(1)= -1 > ** INFO(2)= 0 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscInitializeMUMPS line 1188 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/petsc/src/mat/impls/aij/mpi/mumps/mumps.c > [0]PETSC ERROR: [0] MatGetFactor_aij_mumps line 2125 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/petsc/src/mat/impls/aij/mpi/mumps/mumps.c > [0]PETSC ERROR: [0] MatGetFactor line 4341 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: [0] PCSetUp_LU line 59 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/petsc/src/ksp/pc/impls/factor/lu/lu.c > [0]PETSC ERROR: [0] PCSetUp line 886 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPSetUp line 294 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] STSetUp_Sinvert line 96 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/slepc-3.8.2/src/sys/classes/st/impls/sinvert/sinvert.c > [0]PETSC ERROR: [0] STSetUp line 233 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/slepc-3.8.2/src/sys/classes/st/interface/stsolve.c > [0]PETSC ERROR: [0] EPSSetUp line 104 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/slepc-3.8.2/src/eps/interface/epssetup.c > [0]PETSC ERROR: [0] EPSSolve line 129 /zhome/academic/HLRS/iag/iagyonwu/.local/lib/slepc-3.8.2/src/eps/interface/epssolve.c > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59 > > with regards > Yong From k_burkart at yahoo.com Mon Apr 16 13:55:32 2018 From: k_burkart at yahoo.com (Klaus Burkart) Date: Mon, 16 Apr 2018 18:55:32 +0000 (UTC) Subject: [petsc-users] preallocate space for matrices if only the global number of nonzeros is known In-Reply-To: <87fu3vs5lv.fsf@jedbrown.org> References: <16888007.824157.1523875886094.ref@mail.yahoo.com> <16888007.824157.1523875886094@mail.yahoo.com> <87fu3vs5lv.fsf@jedbrown.org> Message-ID: <1018049008.1070783.1523904932376@mail.yahoo.com> For a symmetric matrix: I assume, I should compute the number of nonzeros per row just for the upper triangle? Am Montag, 16. April 2018, 14:23:19 MESZ hat Jed Brown Folgendes geschrieben: Klaus Burkart writes: > Hi, > How can I preallocate space for matrices (some symmetric, others > asymmetric) if I have the global number of nonzeros (NNZ) but not the > number of nonzeros per row? Unfortunately, this is not enough for efficient assembly.? It is more efficient to over-estimate. > I could compute the NNZ for the upper or lower part separately if this > would be useful for symmetric matrices. > > I create the matrix this way: > ??? MatCreate(PETSC_COMM_WORLD, &M); > ??? MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > ??? MatSetFromOptions(M); > Klaus -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Apr 16 13:58:29 2018 From: jed at jedbrown.org (Jed Brown) Date: Mon, 16 Apr 2018 12:58:29 -0600 Subject: [petsc-users] preallocate space for matrices if only the global number of nonzeros is known In-Reply-To: <1018049008.1070783.1523904932376@mail.yahoo.com> References: <16888007.824157.1523875886094.ref@mail.yahoo.com> <16888007.824157.1523875886094@mail.yahoo.com> <87fu3vs5lv.fsf@jedbrown.org> <1018049008.1070783.1523904932376@mail.yahoo.com> Message-ID: <87sh7vou5m.fsf@jedbrown.org> Klaus Burkart writes: > For a symmetric matrix: > I assume, I should compute the number of nonzeros per row just for the upper triangle? Yes. > > Am Montag, 16. April 2018, 14:23:19 MESZ hat Jed Brown Folgendes geschrieben: > > Klaus Burkart writes: > >> Hi, >> How can I preallocate space for matrices (some symmetric, others >> asymmetric) if I have the global number of nonzeros (NNZ) but not the >> number of nonzeros per row? > > Unfortunately, this is not enough for efficient assembly.? It is more > efficient to over-estimate. > >> I could compute the NNZ for the upper or lower part separately if this >> would be useful for symmetric matrices. >> >> I create the matrix this way: >> ??? MatCreate(PETSC_COMM_WORLD, &M); >> ??? MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); >> ??? MatSetFromOptions(M); >> Klaus From stefano.zampini at gmail.com Mon Apr 16 14:03:36 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Mon, 16 Apr 2018 19:03:36 +0000 Subject: [petsc-users] preallocate space for matrices if only the global number of nonzeros is known In-Reply-To: <1018049008.1070783.1523904932376@mail.yahoo.com> References: <16888007.824157.1523875886094.ref@mail.yahoo.com> <16888007.824157.1523875886094@mail.yahoo.com> <87fu3vs5lv.fsf@jedbrown.org> <1018049008.1070783.1523904932376@mail.yahoo.com> Message-ID: Yes if you use the SBAIJ format. If you use the AIJ format, you should provide the nonzeros for the entire row Il Lun 16 Apr 2018, 21:55 Klaus Burkart ha scritto: > For a symmetric matrix: > > I assume, I should compute the number of nonzeros per row just for the > upper triangle? > > > Am Montag, 16. April 2018, 14:23:19 MESZ hat Jed Brown > Folgendes geschrieben: > > > Klaus Burkart writes: > > > Hi, > > How can I preallocate space for matrices (some symmetric, others > > asymmetric) if I have the global number of nonzeros (NNZ) but not the > > number of nonzeros per row? > > Unfortunately, this is not enough for efficient assembly. It is more > efficient to over-estimate. > > > > I could compute the NNZ for the upper or lower part separately if this > > would be useful for symmetric matrices. > > > > I create the matrix this way: > > MatCreate(PETSC_COMM_WORLD, &M); > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > > MatSetFromOptions(M); > > Klaus > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Apr 16 16:53:56 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 16 Apr 2018 21:53:56 +0000 Subject: [petsc-users] PetscPrintf In-Reply-To: <3955a17e-4dd9-ac23-0e89-0e9a8a3145ac@gmail.com> References: <3955a17e-4dd9-ac23-0e89-0e9a8a3145ac@gmail.com> Message-ID: <00F566F5-62F1-46A9-972C-70447F091B88@anl.gov> Rongliang, I have prepared a fix for this in the branch barry/bugfix-petscformatconvert which comes from the master branch of PETSc. Unfortunately it requires many changes so cannot be back ported to previous releases. Thanks for reporting the problem, Barry > On Apr 11, 2018, at 6:17 PM, Rongliang Chen wrote: > > Dear All, > > > When I tried to print a long string using PetscPrintf() I found that it truncated the string. Attached is a simple example for this (run with single processor). I used PetscPrintf() and printf() to print the same string and the printf() seems OK. I am using petsc-3.8.4. > > > Best, > > Rongliang > > From skavou1 at lsu.edu Mon Apr 16 20:20:26 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Tue, 17 Apr 2018 01:20:26 +0000 Subject: [petsc-users] error running parallel on cluster Message-ID: Hello, I am solving a PDE where I have the spacial derivtive of the time derivative of a variable. In the DMDATSSetIFunctionLocal function I defined d(dY/dt)/dx as: (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) on my workstation, it is working but on the cluster it gives an error. I can not run with debugger on the cluster but I have checked and the error is related to this part and I do not have any problem on cluster when running in series. Can you please help me about it. the error is: 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c [0]PETSC ERROR: [0] TS user implicit function line 829 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSComputeIFunction line 815 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNES user function line 2202 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESComputeFunction line 2187 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 20:11:44 2018 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ==29057== ==29057== HEAP SUMMARY: ==29057== in use at exit: 560,151 bytes in 383 blocks ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes allocated ==29057== ==29057== LEAK SUMMARY: ==29057== definitely lost: 0 bytes in 0 blocks ==29057== indirectly lost: 0 bytes in 0 blocks ==29057== possibly lost: 0 bytes in 0 blocks ==29057== still reachable: 560,151 bytes in 383 blocks ==29057== suppressed: 0 bytes in 0 blocks ==29057== Rerun with --leak-check=full to see details of leaked memory ==29057== ==29057== For counts of detected and suppressed errors, rerun with: -v ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) Thanks, Sepideh -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 16 20:36:37 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 16 Apr 2018 21:36:37 -0400 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: Message-ID: On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi wrote: > Hello, > I am solving a PDE where I have the spacial derivtive of the time > derivative of a variable. In the DMDATSSetIFunctionLocal function I > defined d(dY/dt)/dx as: > (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) > I do not think that you get ghosted Ydot in that function. Matt > on my workstation, it is working but on the cluster it gives an error. I > can not run with debugger on the cluster but I have checked and the error > is related to this part and I do not have any problem on cluster when > running in series. Can you please help me about it. > > the error is: > 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 > /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c > [0]PETSC ERROR: [0] TS user implicit function line 829 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] TSComputeIFunction line 815 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNES user function line 2202 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESComputeFunction line 2187 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 > /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c > [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/ > snes/interface/snes.c > [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/ > ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/ > ts/interface/ts.c > [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/ > ts/interface/ts.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 > 20:11:44 2018 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > ==29057== > ==29057== HEAP SUMMARY: > ==29057== in use at exit: 560,151 bytes in 383 blocks > ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes > allocated > ==29057== > ==29057== LEAK SUMMARY: > ==29057== definitely lost: 0 bytes in 0 blocks > ==29057== indirectly lost: 0 bytes in 0 blocks > ==29057== possibly lost: 0 bytes in 0 blocks > ==29057== still reachable: 560,151 bytes in 383 blocks > ==29057== suppressed: 0 bytes in 0 blocks > ==29057== Rerun with --leak-check=full to see details of leaked memory > ==29057== > ==29057== For counts of detected and suppressed errors, rerun with: -v > ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) > > > > > Thanks, > Sepideh > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Mon Apr 16 20:39:17 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Tue, 17 Apr 2018 01:39:17 +0000 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: , Message-ID: this is not used forghost points . my code is : for (j=info2->ys;jys+info2->ym;j++){ for (i=info2->xs;ixs+info2->xm;i++){ if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} else { //derivatives of U and p Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); Pxy=((aY[j+1][i+1].p+aY[j-1][i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); if (((Px*Px)+(Py*Py))>0.0 ) { // theta=PetscAtanReal(Py/Px); thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); } else { theta=PETSC_PI*0.5; thetax=0.0; thetay=0.0; } e=(1.0+user->epsilon*cos(4.0*theta));; ep=-(4.0*user->epsilon*sin(4.0*theta)); epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); U1=user->D*user->tau_0/(user->W*user->W); U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : 0.0 ; //U2=0.0; P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; aF[j][i].U=(U1*(-0.5*(Ux*Px+Uy*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k)*aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)*aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax*Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i].p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; } } Sepideh ________________________________ From: Matthew Knepley Sent: Monday, April 16, 2018 8:36:37 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi > wrote: Hello, I am solving a PDE where I have the spacial derivtive of the time derivative of a variable. In the DMDATSSetIFunctionLocal function I defined d(dY/dt)/dx as: (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) I do not think that you get ghosted Ydot in that function. Matt on my workstation, it is working but on the cluster it gives an error. I can not run with debugger on the cluster but I have checked and the error is related to this part and I do not have any problem on cluster when running in series. Can you please help me about it. the error is: 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c [0]PETSC ERROR: [0] TS user implicit function line 829 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSComputeIFunction line 815 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNES user function line 2202 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESComputeFunction line 2187 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 20:11:44 2018 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ==29057== ==29057== HEAP SUMMARY: ==29057== in use at exit: 560,151 bytes in 383 blocks ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes allocated ==29057== ==29057== LEAK SUMMARY: ==29057== definitely lost: 0 bytes in 0 blocks ==29057== indirectly lost: 0 bytes in 0 blocks ==29057== possibly lost: 0 bytes in 0 blocks ==29057== still reachable: 560,151 bytes in 383 blocks ==29057== suppressed: 0 bytes in 0 blocks ==29057== Rerun with --leak-check=full to see details of leaked memory ==29057== ==29057== For counts of detected and suppressed errors, rerun with: -v ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) Thanks, Sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 16 20:44:43 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 16 Apr 2018 21:44:43 -0400 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: Message-ID: On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi wrote: > this is not used forghost points . my code is : > 1) You are using ghost points, since you use i-1 as an index 2) I was wrong. X_t should also be ghosted. Run under valgrind. Thanks, Matt > for (j=info2->ys;jys+info2->ym;j++){ > for (i=info2->xs;ixs+info2->xm;i++){ > if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; > aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} > else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} > else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; > aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} > else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} > else { > //derivatives of U and p > Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); > Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); > Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); > Pxy=((aY[j+1][i+1].p+aY[j-1][ > i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); > Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); > Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); > Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); > > > if (((Px*Px)+(Py*Py))>0.0 ) { // > theta=PetscAtanReal(Py/Px); > thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); > thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); > } > else { > theta=PETSC_PI*0.5; > thetax=0.0; > thetay=0.0; > } > e=(1.0+user->epsilon*cos(4.0*theta));; > ep=-(4.0*user->epsilon*sin(4.0*theta)); > epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); > > > > Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); > Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); > Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); > Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); > > U1=user->D*user->tau_0/(user->W*user->W); > U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : > 0.0 ; > //U2=0.0; > P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; > > aF[j][i].U=(U1*(-0.5*(Ux*Px+Uy*Py)+0.5*(1-aY[j][i].p)*( > Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][i].U)*(Pxt*Px+Pyt*Py+ > aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k)*aYdot[j][i].p*(Ux* > Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)*aYdot[j][i].p-0.5* > (1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; > > aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*( > thetax*Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+( > aY[j][i].p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*( > aY[j][i].p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][ > i].U+P1)-(1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0* > theta))*(1.0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; > > } > } > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:36:37 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi wrote: > > Hello, > I am solving a PDE where I have the spacial derivtive of the time > derivative of a variable. In the DMDATSSetIFunctionLocal function I > defined d(dY/dt)/dx as: > (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) > > > I do not think that you get ghosted Ydot in that function. > > Matt > > > on my workstation, it is working but on the cluster it gives an error. I > can not run with debugger on the cluster but I have checked and the error > is related to this part and I do not have any problem on cluster when > running in series. Can you please help me about it. > > the error is: > 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d > ocumentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 > /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c > [0]PETSC ERROR: [0] TS user implicit function line 829 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] TSComputeIFunction line 815 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNES user function line 2202 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESComputeFunction line 2187 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 > /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c > [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/s > nes/interface/snes.c > [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/t > s/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 > 20:11:44 2018 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > ==29057== > ==29057== HEAP SUMMARY: > ==29057== in use at exit: 560,151 bytes in 383 blocks > ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes > allocated > ==29057== > ==29057== LEAK SUMMARY: > ==29057== definitely lost: 0 bytes in 0 blocks > ==29057== indirectly lost: 0 bytes in 0 blocks > ==29057== possibly lost: 0 bytes in 0 blocks > ==29057== still reachable: 560,151 bytes in 383 blocks > ==29057== suppressed: 0 bytes in 0 blocks > ==29057== Rerun with --leak-check=full to see details of leaked memory > ==29057== > ==29057== For counts of detected and suppressed errors, rerun with: -v > ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) > > > > > Thanks, > Sepideh > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Mon Apr 16 21:22:31 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Tue, 17 Apr 2018 02:22:31 +0000 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: , Message-ID: I run with valgrind with the following command: /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason -ksp_monitor the valgringoutput is as following: ==39396== Invalid read of size 8 ==39396== at 0x402679: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 alloc'd ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== ==39396== Invalid read of size 8 ==39396== at 0x402689: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd ==39396== Thanks, Sepideh ________________________________ From: Matthew Knepley Sent: Monday, April 16, 2018 8:44:43 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi > wrote: this is not used forghost points . my code is : 1) You are using ghost points, since you use i-1 as an index 2) I was wrong. X_t should also be ghosted. Run under valgrind. Thanks, Matt for (j=info2->ys;jys+info2->ym;j++){ for (i=info2->xs;ixs+info2->xm;i++){ if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} else { //derivatives of U and p Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); Pxy=((aY[j+1][i+1].p+aY[j-1][i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); if (((Px*Px)+(Py*Py))>0.0 ) { // theta=PetscAtanReal(Py/Px); thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); } else { theta=PETSC_PI*0.5; thetax=0.0; thetay=0.0; } e=(1.0+user->epsilon*cos(4.0*theta));; ep=-(4.0*user->epsilon*sin(4.0*theta)); epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); U1=user->D*user->tau_0/(user->W*user->W); U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : 0.0 ; //U2=0.0; P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; aF[j][i].U=(U1*(-0.5*(Ux*Px+Uy*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k)*aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)*aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax*Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i].p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; } } Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:36:37 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi > wrote: Hello, I am solving a PDE where I have the spacial derivtive of the time derivative of a variable. In the DMDATSSetIFunctionLocal function I defined d(dY/dt)/dx as: (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) I do not think that you get ghosted Ydot in that function. Matt on my workstation, it is working but on the cluster it gives an error. I can not run with debugger on the cluster but I have checked and the error is related to this part and I do not have any problem on cluster when running in series. Can you please help me about it. the error is: 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c [0]PETSC ERROR: [0] TS user implicit function line 829 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSComputeIFunction line 815 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNES user function line 2202 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESComputeFunction line 2187 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 20:11:44 2018 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ==29057== ==29057== HEAP SUMMARY: ==29057== in use at exit: 560,151 bytes in 383 blocks ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes allocated ==29057== ==29057== LEAK SUMMARY: ==29057== definitely lost: 0 bytes in 0 blocks ==29057== indirectly lost: 0 bytes in 0 blocks ==29057== possibly lost: 0 bytes in 0 blocks ==29057== still reachable: 560,151 bytes in 383 blocks ==29057== suppressed: 0 bytes in 0 blocks ==29057== Rerun with --leak-check=full to see details of leaked memory ==29057== ==29057== For counts of detected and suppressed errors, rerun with: -v ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) Thanks, Sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Mon Apr 16 21:33:00 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Tue, 17 Apr 2018 02:33:00 +0000 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: , , Message-ID: I think I find the problem but I do not know how to solve it. I believe the problem is in line 112 of code which is the derivative of aYdot[i][j] in y-direction: Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); in here I used 2 processors and the error is exactly in the middle of the domain in y-direction where I believe aYdot[j-1][i].p belongs to the other processor. It is interesting that I have this problem because always when discretizing derivatives, points can belong to different processors and it does not normally cause any problem (at least I did not have such problem when there were only spacial derivatives in FormFunction calculations of DMDATSSetIFunctionLocal. Can you please help me what I should do? Sepideh ________________________________ From: Sepideh Kavousi Sent: Monday, April 16, 2018 9:22:31 PM To: Matthew Knepley Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster I run with valgrind with the following command: /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason -ksp_monitor the valgringoutput is as following: ==39396== Invalid read of size 8 ==39396== at 0x402679: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 alloc'd ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== ==39396== Invalid read of size 8 ==39396== at 0x402689: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd ==39396== Thanks, Sepideh ________________________________ From: Matthew Knepley Sent: Monday, April 16, 2018 8:44:43 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi > wrote: this is not used forghost points . my code is : 1) You are using ghost points, since you use i-1 as an index 2) I was wrong. X_t should also be ghosted. Run under valgrind. Thanks, Matt for (j=info2->ys;jys+info2->ym;j++){ for (i=info2->xs;ixs+info2->xm;i++){ if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} else { //derivatives of U and p Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); Pxy=((aY[j+1][i+1].p+aY[j-1][i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); if (((Px*Px)+(Py*Py))>0.0 ) { // theta=PetscAtanReal(Py/Px); thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); } else { theta=PETSC_PI*0.5; thetax=0.0; thetay=0.0; } e=(1.0+user->epsilon*cos(4.0*theta));; ep=-(4.0*user->epsilon*sin(4.0*theta)); epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); U1=user->D*user->tau_0/(user->W*user->W); U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : 0.0 ; //U2=0.0; P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; aF[j][i].U=(U1*(-0.5*(Ux*Px+Uy*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k)*aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)*aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax*Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i].p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; } } Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:36:37 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi > wrote: Hello, I am solving a PDE where I have the spacial derivtive of the time derivative of a variable. In the DMDATSSetIFunctionLocal function I defined d(dY/dt)/dx as: (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) I do not think that you get ghosted Ydot in that function. Matt on my workstation, it is working but on the cluster it gives an error. I can not run with debugger on the cluster but I have checked and the error is related to this part and I do not have any problem on cluster when running in series. Can you please help me about it. the error is: 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c [0]PETSC ERROR: [0] TS user implicit function line 829 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSComputeIFunction line 815 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNES user function line 2202 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESComputeFunction line 2187 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 20:11:44 2018 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ==29057== ==29057== HEAP SUMMARY: ==29057== in use at exit: 560,151 bytes in 383 blocks ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes allocated ==29057== ==29057== LEAK SUMMARY: ==29057== definitely lost: 0 bytes in 0 blocks ==29057== indirectly lost: 0 bytes in 0 blocks ==29057== possibly lost: 0 bytes in 0 blocks ==29057== still reachable: 560,151 bytes in 383 blocks ==29057== suppressed: 0 bytes in 0 blocks ==29057== Rerun with --leak-check=full to see details of leaked memory ==29057== ==29057== For counts of detected and suppressed errors, rerun with: -v ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) Thanks, Sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 16 21:34:01 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 16 Apr 2018 22:34:01 -0400 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: Message-ID: On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi wrote: > I run with valgrind with the following command: > > /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q > --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off > -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler > -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr > -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason > -ksp_monitor > Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only chooses to ghost the input vector X and not X_t https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Unfortunately, this was done by Peter in 2012 and he is gone, so we can't blame him. It is true that very few people use spatial derivatives of X_t, but it does make sense. Right there, you want to add ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); and it should work. Matt > the valgringoutput is as following: > > ==39396== Invalid read of size 8 > ==39396== at 0x402679: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 > alloc'd > ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) > ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) > ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) > ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) > ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) > ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== > ==39396== Invalid read of size 8 > ==39396== at 0x402689: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd > ==39396== > > Thanks, > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:44:43 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi wrote: > > this is not used forghost points . my code is : > > > 1) You are using ghost points, since you use i-1 as an index > > 2) I was wrong. X_t should also be ghosted. > > Run under valgrind. > > Thanks, > > Matt > > > for (j=info2->ys;jys+info2->ym;j++){ > for (i=info2->xs;ixs+info2->xm;i++){ > if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; > aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} > else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} > else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; > aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} > else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} > else { > //derivatives of U and p > Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); > Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); > Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); > Pxy=((aY[j+1][i+1].p+aY[j-1][ > i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); > Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); > Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); > Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); > > > if (((Px*Px)+(Py*Py))>0.0 ) { // > theta=PetscAtanReal(Py/Px); > thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); > thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); > } > else { > theta=PETSC_PI*0.5; > thetax=0.0; > thetay=0.0; > } > e=(1.0+user->epsilon*cos(4.0*theta));; > ep=-(4.0*user->epsilon*sin(4.0*theta)); > epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); > > > > Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); > Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); > Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); > Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); > > U1=user->D*user->tau_0/(user->W*user->W); > U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : > 0.0 ; > //U2=0.0; > P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; > > aF[j][i].U=(U1*(-0.5*(Ux*Px+U > y*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][ > i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k) > *aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U) > *aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; > > aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax > *Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i] > .p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i]. > p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)- > (1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))* > (1.0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; > > } > } > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:36:37 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi wrote: > > Hello, > I am solving a PDE where I have the spacial derivtive of the time > derivative of a variable. In the DMDATSSetIFunctionLocal function I > defined d(dY/dt)/dx as: > (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) > > > I do not think that you get ghosted Ydot in that function. > > Matt > > > on my workstation, it is working but on the cluster it gives an error. I > can not run with debugger on the cluster but I have checked and the error > is related to this part and I do not have any problem on cluster when > running in series. Can you please help me about it. > > the error is: > 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d > ocumentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 > /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c > [0]PETSC ERROR: [0] TS user implicit function line 829 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] TSComputeIFunction line 815 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNES user function line 2202 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESComputeFunction line 2187 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 > /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c > [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/s > nes/interface/snes.c > [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/t > s/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 > 20:11:44 2018 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > ==29057== > ==29057== HEAP SUMMARY: > ==29057== in use at exit: 560,151 bytes in 383 blocks > ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes > allocated > ==29057== > ==29057== LEAK SUMMARY: > ==29057== definitely lost: 0 bytes in 0 blocks > ==29057== indirectly lost: 0 bytes in 0 blocks > ==29057== possibly lost: 0 bytes in 0 blocks > ==29057== still reachable: 560,151 bytes in 383 blocks > ==29057== suppressed: 0 bytes in 0 blocks > ==29057== Rerun with --leak-check=full to see details of leaked memory > ==29057== > ==29057== For counts of detected and suppressed errors, rerun with: -v > ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) > > > > > Thanks, > Sepideh > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rongliang.chan at gmail.com Mon Apr 16 23:12:13 2018 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Tue, 17 Apr 2018 12:12:13 +0800 Subject: [petsc-users] PetscPrintf In-Reply-To: <00F566F5-62F1-46A9-972C-70447F091B88@anl.gov> References: <3955a17e-4dd9-ac23-0e89-0e9a8a3145ac@gmail.com> <00F566F5-62F1-46A9-972C-70447F091B88@anl.gov> Message-ID: Thanks Barry. I will update my petsc. By the way, I simply replaced the functions PetscFormatConvert() and PetscVFPrintfDefault() with those in petsc-3.6 and it works for me. Best, Rongliang On 04/17/2018 05:53 AM, Smith, Barry F. wrote: > Rongliang, > > I have prepared a fix for this in the branch barry/bugfix-petscformatconvert which comes from the master branch of PETSc. > > Unfortunately it requires many changes so cannot be back ported to previous releases. > > Thanks for reporting the problem, > > Barry > > >> On Apr 11, 2018, at 6:17 PM, Rongliang Chen wrote: >> >> Dear All, >> >> >> When I tried to print a long string using PetscPrintf() I found that it truncated the string. Attached is a simple example for this (run with single processor). I used PetscPrintf() and printf() to print the same string and the printf() seems OK. I am using petsc-3.8.4. >> >> >> Best, >> >> Rongliang >> >> From may at bu.edu Tue Apr 17 07:33:21 2018 From: may at bu.edu (Young, Matthew, Adam) Date: Tue, 17 Apr 2018 12:33:21 +0000 Subject: [petsc-users] Reuse KSP Message-ID: My code (C/C++) uses PETSc to solve a linear system at each iteration in a time-stepping scheme. The PETSc routines are separated out because the scheme that uses them is optional in the larger code. At each time step, the PETSc scheme calls a routine that includes: { Mat A; Vec x,b; KSP ksp; [Set up A,x,b] KSPCreate(PETSC_COMM_WORLD,&ksp); [Build A and b from simulation data] [Set nullspace of A with MatSetNullSpace] KSPSetOperators(ksp,A,A); KSPSetFromOptions(ksp); KSPSolve(ksp,b,x); [Write x into an instance of our array class, perform checks, etc.] KSPDestroy(&ksp); return(0); } My question is: If I want to use superLU_DIST with the option -mat_superlu_dist_fact SamePattern (or SamePattern_SameRowPerm), should I move KSPCreate/Destroy outside of the time-step loop in the larger code? Thanks. ---------------------------- Matthew Young PhD Candidate Astronomy Department Boston University -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 17 07:38:40 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 17 Apr 2018 08:38:40 -0400 Subject: [petsc-users] Reuse KSP In-Reply-To: References: Message-ID: On Tue, Apr 17, 2018 at 8:33 AM, Young, Matthew, Adam wrote: > My code (C/C++) uses PETSc to solve a linear system at each iteration in a > time-stepping scheme. The PETSc routines are separated out because the > scheme that uses them is optional in the larger code. > > > At each time step, the PETSc scheme calls a routine that includes: > > > { > > Mat A; > > Vec x,b; > > KSP ksp; > > > [Set up A,x,b] > > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > > [Build A and b from simulation data] > > > [Set nullspace of A with MatSetNullSpace] > > > KSPSetOperators(ksp,A,A); > > KSPSetFromOptions(ksp); > > KSPSolve(ksp,b,x); > > > [Write x into an instance of our array class, perform checks, etc.] > > > KSPDestroy(&ksp); > > > return(0); > > } > > > My question is: If I want to use superLU_DIST with the option > -mat_superlu_dist_fact SamePattern (or SamePattern_SameRowPerm), should I > move KSPCreate/Destroy outside of the time-step loop in the larger code? > 1) Yes 2) How are you using LU when A has a nullspace? 3) You can also move out SetFromOptions(). Matt > Thanks. > ---------------------------- > Matthew Young > PhD Candidate > Astronomy Department > Boston University > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From may at bu.edu Tue Apr 17 07:48:15 2018 From: may at bu.edu (Young, Matthew, Adam) Date: Tue, 17 Apr 2018 12:48:15 +0000 Subject: [petsc-users] Reuse KSP In-Reply-To: References: , Message-ID: ________________________________ From: Matthew Knepley Sent: Tuesday, April 17, 2018 8:38 AM To: Young, Matthew, Adam Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Reuse KSP On Tue, Apr 17, 2018 at 8:33 AM, Young, Matthew, Adam > wrote: My code (C/C++) uses PETSc to solve a linear system at each iteration in a time-stepping scheme. The PETSc routines are separated out because the scheme that uses them is optional in the larger code. At each time step, the PETSc scheme calls a routine that includes: { Mat A; Vec x,b; KSP ksp; [Set up A,x,b] KSPCreate(PETSC_COMM_WORLD,&ksp); [Build A and b from simulation data] [Set nullspace of A with MatSetNullSpace] KSPSetOperators(ksp,A,A); KSPSetFromOptions(ksp); KSPSolve(ksp,b,x); [Write x into an instance of our array class, perform checks, etc.] KSPDestroy(&ksp); return(0); } My question is: If I want to use superLU_DIST with the option -mat_superlu_dist_fact SamePattern (or SamePattern_SameRowPerm), should I move KSPCreate/Destroy outside of the time-step loop in the larger code? 1) Yes 2) How are you using LU when A has a nullspace? My bad - that code is in there from when I was trying iterative solvers. Should I remove it if I use LU or will PETSc just ignore it? 3) You can also move out SetFromOptions(). Thanks for the advice. So, that means I'll build A and b from simulation data, SetOperators(), Solve(), and carry on? Matt Thanks. ---------------------------- Matthew Young PhD Candidate Astronomy Department Boston University -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Tue Apr 17 10:33:51 2018 From: epscodes at gmail.com (Xiangdong) Date: Tue, 17 Apr 2018 11:33:51 -0400 Subject: [petsc-users] question on VecPointwiseMax Message-ID: Hello everyone, When I call VecPointwiseMax(x,x,y) , I got an error message "object is in wrong state. Vec is locked read only, argument #1." However, In the online manual of output parameters of VecPointwiseMax, it says: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPointwiseMax.html Output Parameter*w *-the result Notes: any subset of the x, y, and w may be the same vector. For complex numbers compares only the real part However, in the implementation of VecPointwiseMax_Seq, http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/impls/seq/bvec2.c.html#VecPointwiseMax_Seq , it seems that xin is read locked, and win and xin cannot be the same vectors. *PetscErrorCode VecPointwiseMax_Seq(Vec win,Vec xin,Vec yin)* 17: { 19: PetscInt n = win->map->n,i; 20: PetscScalar *ww,*xx,*yy; /* cannot make xx or yy const since might be ww */ 23: VecGetArrayRead (xin,(const PetscScalar **)&xx); 24: VecGetArrayRead (yin,(const PetscScalar **)&yy); 25: VecGetArray (win,&ww); 27: for (i=0; i(PetscRealPart(xx[i]),PetscRealPart(yy[i])); 29: VecRestoreArrayRead (xin,(const PetscScalar **)&xx); 30: VecRestoreArrayRead (yin,(const PetscScalar **)&yy); 31: VecRestoreArray (win,&ww); 32: PetscLogFlops (n); 33: return(0); 34: } Can w and x really be same vector in the VecPointwiseMax? Thanks. Best, Xiangdong -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Tue Apr 17 12:04:04 2018 From: epscodes at gmail.com (Xiangdong) Date: Tue, 17 Apr 2018 13:04:04 -0400 Subject: [petsc-users] question on VecPointwiseMax In-Reply-To: References: Message-ID: Hello everyone, Here are a little more information. Here is the function calls I have: SNESGetSolution(snes,&x); VecPointwiseMax(x,x,xl); Fail VecPointwiseMax(x,y,xl); Fail The error messages are"Object is in wrong state, Vec is locked read only, argument #1." It seems that the vector x returned from SNESGetSolution is locked read only, Is it true? If I want do some post processing by modifying the x after each nonlinear iteration, how can I get the solution and modify it? Thank you. Best, Xiangdong On Tue, Apr 17, 2018 at 11:33 AM, Xiangdong wrote: > Hello everyone, > > When I call VecPointwiseMax(x,x,y) , I got an error message "object is in > wrong state. Vec is locked read only, argument #1." > > However, In the online manual of output parameters of VecPointwiseMax, it > says: > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/ > VecPointwiseMax.html > Output Parameter*w *-the result > > Notes: any subset of the x, y, and w may be the same vector. For complex > numbers compares only the real part > However, in the implementation of VecPointwiseMax_Seq, > http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/ > impls/seq/bvec2.c.html#VecPointwiseMax_Seq, > > it seems that xin is read locked, and win and xin cannot be the same > vectors. > > *PetscErrorCode VecPointwiseMax_Seq(Vec win,Vec xin,Vec yin)* 17: { 19: PetscInt n = win->map->n,i; 20: PetscScalar *ww,*xx,*yy; /* cannot make xx or yy const since might be ww */ > 23: VecGetArrayRead (xin,(const PetscScalar **)&xx); 24: VecGetArrayRead (yin,(const PetscScalar **)&yy); 25: VecGetArray (win,&ww); > 27: for (i=0; i(PetscRealPart(xx[i]),PetscRealPart(yy[i])); > 29: VecRestoreArrayRead (xin,(const PetscScalar **)&xx); 30: VecRestoreArrayRead (yin,(const PetscScalar **)&yy); 31: VecRestoreArray (win,&ww); 32: PetscLogFlops (n); 33: return(0); 34: } > > Can w and x really be same vector in the VecPointwiseMax? Thanks. > > Best, > Xiangdong > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Tue Apr 17 12:50:11 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Tue, 17 Apr 2018 17:50:11 +0000 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: , Message-ID: Mathew, I previously use DMDATSSetIFunctionLocal(user.da,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. If I want to use your solution I can not use it because in the FormFunction definition I must use arrays, not vectors.So to solve this issue I followed two methods where none were able to solve it. 1- in first method I decided to use TSSetIFunction instead of DMDATSSetIFunctionLocal for this means first in the main function, I use TSSetDM and my form function variables were as: PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, struct VAR_STRUCT *user) { . . . . ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); . . . } But still, it does not consider vectors y,ydot,f related to dmda (problem executing DMDAVecGetArray) 2- in second method I decided to use DMTSSetIFunction but still, FormFunction is in form of TSIFunction where we do not define dm object and I think it does not understand dm and da are connected, although I have used TSSetDM in the main function. Can you please help me what should I do? Regards, Sepideh ________________________________ From: Matthew Knepley Sent: Monday, April 16, 2018 9:34:01 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi > wrote: I run with valgrind with the following command: /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason -ksp_monitor Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only chooses to ghost the input vector X and not X_t https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Unfortunately, this was done by Peter in 2012 and he is gone, so we can't blame him. It is true that very few people use spatial derivatives of X_t, but it does make sense. Right there, you want to add ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); and it should work. Matt the valgringoutput is as following: ==39396== Invalid read of size 8 ==39396== at 0x402679: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 alloc'd ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== ==39396== Invalid read of size 8 ==39396== at 0x402689: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd ==39396== Thanks, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:44:43 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi > wrote: this is not used forghost points . my code is : 1) You are using ghost points, since you use i-1 as an index 2) I was wrong. X_t should also be ghosted. Run under valgrind. Thanks, Matt for (j=info2->ys;jys+info2->ym;j++){ for (i=info2->xs;ixs+info2->xm;i++){ if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} else { //derivatives of U and p Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); Pxy=((aY[j+1][i+1].p+aY[j-1][i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); if (((Px*Px)+(Py*Py))>0.0 ) { // theta=PetscAtanReal(Py/Px); thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); } else { theta=PETSC_PI*0.5; thetax=0.0; thetay=0.0; } e=(1.0+user->epsilon*cos(4.0*theta));; ep=-(4.0*user->epsilon*sin(4.0*theta)); epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); U1=user->D*user->tau_0/(user->W*user->W); U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : 0.0 ; //U2=0.0; P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; aF[j][i].U=(U1*(-0.5*(Ux*Px+Uy*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k)*aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)*aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax*Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i].p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; } } Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:36:37 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi > wrote: Hello, I am solving a PDE where I have the spacial derivtive of the time derivative of a variable. In the DMDATSSetIFunctionLocal function I defined d(dY/dt)/dx as: (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) I do not think that you get ghosted Ydot in that function. Matt on my workstation, it is working but on the cluster it gives an error. I can not run with debugger on the cluster but I have checked and the error is related to this part and I do not have any problem on cluster when running in series. Can you please help me about it. the error is: 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c [0]PETSC ERROR: [0] TS user implicit function line 829 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSComputeIFunction line 815 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNES user function line 2202 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESComputeFunction line 2187 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 20:11:44 2018 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ==29057== ==29057== HEAP SUMMARY: ==29057== in use at exit: 560,151 bytes in 383 blocks ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes allocated ==29057== ==29057== LEAK SUMMARY: ==29057== definitely lost: 0 bytes in 0 blocks ==29057== indirectly lost: 0 bytes in 0 blocks ==29057== possibly lost: 0 bytes in 0 blocks ==29057== still reachable: 560,151 bytes in 383 blocks ==29057== suppressed: 0 bytes in 0 blocks ==29057== Rerun with --leak-check=full to see details of leaked memory ==29057== ==29057== For counts of detected and suppressed errors, rerun with: -v ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) Thanks, Sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From olk548 at mail.usask.ca Tue Apr 17 13:06:33 2018 From: olk548 at mail.usask.ca (Oleksandr Koshkarov) Date: Tue, 17 Apr 2018 12:06:33 -0600 Subject: [petsc-users] get a DOF component of DMDA and save it to file Message-ID: <48ee1435-f50b-127f-e8bd-ebcb10b3e4fa@mail.usask.ca> Hello All, I have a 3d DMDA with big DOF number. I want to save one component of DOF to the file, however I see only a solution where I create 3D DMDA with DOF=1, then copy a slice and then save the resulting vector with VecView. There should be a simpler way. I hope you can help me with this :) Thank you, Oleksandr Koshkarov. From knepley at gmail.com Tue Apr 17 13:22:53 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 17 Apr 2018 14:22:53 -0400 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: Message-ID: On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi wrote: > Mathew, > I previously use DMDATSSetIFunctionLocal(user.da,INSERT_VALUES,(DMDATSIFunctionLocal) > FormFunction,&user) in my code. If I want to use your solution I can not > use it because in the FormFunction definition I must use arrays, not > vectors.So to solve this issue I followed two methods where none were able > to solve it. > 1- in first method I decided to use TSSetIFunction instead of > DMDATSSetIFunctionLocal > > for this means first in the main function, I use TSSetDM and my form > function variables were as: > PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, > struct VAR_STRUCT *user) { > . > . > . > . > ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); > ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); > > ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local) > ;CHKERRQ(ierr); > ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local); > CHKERRQ(ierr); > . > . > . > > } > But still, it does not consider vectors y,ydot,f related to dmda (problem > executing DMDAVecGetArray) > We cannot help you if you do not show full error messages. Why not fix the code with SetIFunctionLocal(), as I said in my last email. I will fix PETSc proper in branch at the end of the week. I have a proposal due tomorrow, so I cannot do it right now. Thanks, Matt > 2- in second method I decided to use DMTSSetIFunction > but still, FormFunction is in form of TSIFunction where we do not define > dm object and I think it does not understand dm and da are connected, > although I have used TSSetDM in the main function. > > Can you please help me what should I do? > Regards, > Sepideh > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 9:34:01 PM > > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi wrote: > > I run with valgrind with the following command: > > /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q > --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off > -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler > -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr > -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason > -ksp_monitor > > > Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only > chooses to ghost the input vector X and not X_t > > https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c5 > 82694ff011/src/ts/utils/dmdats.c?at=master&fileviewer= > file-view-default#dmdats.c-68 > > Unfortunately, this was done by Peter in 2012 and he is gone, so we can't > blame him. > It is true that very few people use spatial derivatives of X_t, but it > does make sense. > Right there, you want to add > > ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > > ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); > > > and it should work. > > Matt > > > the valgringoutput is as following: > > ==39396== Invalid read of size 8 > ==39396== at 0x402679: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 > alloc'd > ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) > ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) > ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) > ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) > ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) > ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== > ==39396== Invalid read of size 8 > ==39396== at 0x402689: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd > ==39396== > > Thanks, > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:44:43 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi wrote: > > this is not used forghost points . my code is : > > > 1) You are using ghost points, since you use i-1 as an index > > 2) I was wrong. X_t should also be ghosted. > > Run under valgrind. > > Thanks, > > Matt > > > for (j=info2->ys;jys+info2->ym;j++){ > for (i=info2->xs;ixs+info2->xm;i++){ > if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; > aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} > else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} > else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; > aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} > else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} > else { > //derivatives of U and p > Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); > Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); > Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); > Pxy=((aY[j+1][i+1].p+aY[j-1][ > i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); > Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); > Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); > Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); > > > if (((Px*Px)+(Py*Py))>0.0 ) { // > theta=PetscAtanReal(Py/Px); > thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); > thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); > } > else { > theta=PETSC_PI*0.5; > thetax=0.0; > thetay=0.0; > } > e=(1.0+user->epsilon*cos(4.0*theta));; > ep=-(4.0*user->epsilon*sin(4.0*theta)); > epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); > > > > Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); > Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); > Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); > Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); > > U1=user->D*user->tau_0/(user->W*user->W); > U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : > 0.0 ; > //U2=0.0; > P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; > > aF[j][i].U=(U1*(-0.5*(Ux*Px+U > y*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][ > i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k) > *aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)* > aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; > > aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax > *Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i] > .p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i]. > p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-( > 1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1. > 0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; > > } > } > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:36:37 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi wrote: > > Hello, > I am solving a PDE where I have the spacial derivtive of the time > derivative of a variable. In the DMDATSSetIFunctionLocal function I > defined d(dY/dt)/dx as: > (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) > > > I do not think that you get ghosted Ydot in that function. > > Matt > > > on my workstation, it is working but on the cluster it gives an error. I > can not run with debugger on the cluster but I have checked and the error > is related to this part and I do not have any problem on cluster when > running in series. Can you please help me about it. > > the error is: > 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d > ocumentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 > /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c > [0]PETSC ERROR: [0] TS user implicit function line 829 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] TSComputeIFunction line 815 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNES user function line 2202 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESComputeFunction line 2187 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 > /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c > [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/s > nes/interface/snes.c > [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/t > s/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 > 20:11:44 2018 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > ==29057== > ==29057== HEAP SUMMARY: > ==29057== in use at exit: 560,151 bytes in 383 blocks > ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes > allocated > ==29057== > ==29057== LEAK SUMMARY: > ==29057== definitely lost: 0 bytes in 0 blocks > ==29057== indirectly lost: 0 bytes in 0 blocks > ==29057== possibly lost: 0 bytes in 0 blocks > ==29057== still reachable: 560,151 bytes in 383 blocks > ==29057== suppressed: 0 bytes in 0 blocks > ==29057== Rerun with --leak-check=full to see details of leaked memory > ==29057== > ==29057== For counts of detected and suppressed errors, rerun with: -v > ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) > > > > > Thanks, > Sepideh > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 17 13:42:07 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 17 Apr 2018 14:42:07 -0400 Subject: [petsc-users] get a DOF component of DMDA and save it to file In-Reply-To: <48ee1435-f50b-127f-e8bd-ebcb10b3e4fa@mail.usask.ca> References: <48ee1435-f50b-127f-e8bd-ebcb10b3e4fa@mail.usask.ca> Message-ID: On Tue, Apr 17, 2018 at 2:06 PM, Oleksandr Koshkarov wrote: > Hello All, > > I have a 3d DMDA with big DOF number. I want to save one component of DOF > to the file, however I see only a solution where I create 3D DMDA with > DOF=1, then copy a slice and then save the resulting vector with VecView. > There should be a simpler way. I hope you can help me with this :) > I would create the same DMDA with just 1 component (one line), get a vector from it (one line), call this to map your big vector to the smaller one (one line), http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecStrideGather.html and VecView() it, so 4 lines instead of 1-2. Thanks, Matt Thank you, > > Oleksandr Koshkarov. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Tue Apr 17 14:07:37 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Tue, 17 Apr 2018 19:07:37 +0000 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: , Message-ID: The reason I can not use your method is that the input arguments of SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your method which was: ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); I need to have the vector of Xdot, not the array. So I think I should use SetIFunction instead of SetIFunctionLocal. Sepideh ________________________________ From: Matthew Knepley Sent: Tuesday, April 17, 2018 1:22:53 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi > wrote: Mathew, I previously use DMDATSSetIFunctionLocal(user.da,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. If I want to use your solution I can not use it because in the FormFunction definition I must use arrays, not vectors.So to solve this issue I followed two methods where none were able to solve it. 1- in first method I decided to use TSSetIFunction instead of DMDATSSetIFunctionLocal for this means first in the main function, I use TSSetDM and my form function variables were as: PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, struct VAR_STRUCT *user) { . . . . ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); . . . } But still, it does not consider vectors y,ydot,f related to dmda (problem executing DMDAVecGetArray) We cannot help you if you do not show full error messages. Why not fix the code with SetIFunctionLocal(), as I said in my last email. I will fix PETSc proper in branch at the end of the week. I have a proposal due tomorrow, so I cannot do it right now. Thanks, Matt 2- in second method I decided to use DMTSSetIFunction but still, FormFunction is in form of TSIFunction where we do not define dm object and I think it does not understand dm and da are connected, although I have used TSSetDM in the main function. Can you please help me what should I do? Regards, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 9:34:01 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi > wrote: I run with valgrind with the following command: /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason -ksp_monitor Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only chooses to ghost the input vector X and not X_t https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Unfortunately, this was done by Peter in 2012 and he is gone, so we can't blame him. It is true that very few people use spatial derivatives of X_t, but it does make sense. Right there, you want to add ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); and it should work. Matt the valgringoutput is as following: ==39396== Invalid read of size 8 ==39396== at 0x402679: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 alloc'd ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== ==39396== Invalid read of size 8 ==39396== at 0x402689: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd ==39396== Thanks, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:44:43 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi > wrote: this is not used forghost points . my code is : 1) You are using ghost points, since you use i-1 as an index 2) I was wrong. X_t should also be ghosted. Run under valgrind. Thanks, Matt for (j=info2->ys;jys+info2->ym;j++){ for (i=info2->xs;ixs+info2->xm;i++){ if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} else { //derivatives of U and p Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); Pxy=((aY[j+1][i+1].p+aY[j-1][i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); if (((Px*Px)+(Py*Py))>0.0 ) { // theta=PetscAtanReal(Py/Px); thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); } else { theta=PETSC_PI*0.5; thetax=0.0; thetay=0.0; } e=(1.0+user->epsilon*cos(4.0*theta));; ep=-(4.0*user->epsilon*sin(4.0*theta)); epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); U1=user->D*user->tau_0/(user->W*user->W); U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : 0.0 ; //U2=0.0; P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; aF[j][i].U=(U1*(-0.5*(Ux*Px+Uy*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k)*aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)*aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax*Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i].p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; } } Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:36:37 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi > wrote: Hello, I am solving a PDE where I have the spacial derivtive of the time derivative of a variable. In the DMDATSSetIFunctionLocal function I defined d(dY/dt)/dx as: (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) I do not think that you get ghosted Ydot in that function. Matt on my workstation, it is working but on the cluster it gives an error. I can not run with debugger on the cluster but I have checked and the error is related to this part and I do not have any problem on cluster when running in series. Can you please help me about it. the error is: 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c [0]PETSC ERROR: [0] TS user implicit function line 829 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSComputeIFunction line 815 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNES user function line 2202 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESComputeFunction line 2187 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 20:11:44 2018 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ==29057== ==29057== HEAP SUMMARY: ==29057== in use at exit: 560,151 bytes in 383 blocks ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes allocated ==29057== ==29057== LEAK SUMMARY: ==29057== definitely lost: 0 bytes in 0 blocks ==29057== indirectly lost: 0 bytes in 0 blocks ==29057== possibly lost: 0 bytes in 0 blocks ==29057== still reachable: 560,151 bytes in 383 blocks ==29057== suppressed: 0 bytes in 0 blocks ==29057== Rerun with --leak-check=full to see details of leaked memory ==29057== ==29057== For counts of detected and suppressed errors, rerun with: -v ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) Thanks, Sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 17 16:13:10 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 17 Apr 2018 21:13:10 +0000 Subject: [petsc-users] Reuse KSP In-Reply-To: References: Message-ID: <6970E094-0CCC-4452-9330-405D25C98765@anl.gov> If the matrix has a null space then, in general, direct LU factorizations should fail. If they don't fail they can produce an incorrect answer If your matrix has a null space then you should not remove the [Set nullspace of A with MatSetNullSpace] Barry > On Apr 17, 2018, at 7:48 AM, Young, Matthew, Adam wrote: > > > > > From: Matthew Knepley > Sent: Tuesday, April 17, 2018 8:38 AM > To: Young, Matthew, Adam > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Reuse KSP > > On Tue, Apr 17, 2018 at 8:33 AM, Young, Matthew, Adam wrote: > My code (C/C++) uses PETSc to solve a linear system at each iteration in a time-stepping scheme. The PETSc routines are separated out because the scheme that uses them is optional in the larger code. > > At each time step, the PETSc scheme calls a routine that includes: > > { > Mat A; > Vec x,b; > KSP ksp; > > [Set up A,x,b] > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > [Build A and b from simulation data] > > [Set nullspace of A with MatSetNullSpace] > > KSPSetOperators(ksp,A,A); > KSPSetFromOptions(ksp); > KSPSolve(ksp,b,x); > > [Write x into an instance of our array class, perform checks, etc.] > > KSPDestroy(&ksp); > > return(0); > } > > My question is: If I want to use superLU_DIST with the option -mat_superlu_dist_fact SamePattern (or SamePattern_SameRowPerm), should I move KSPCreate/Destroy outside of the time-step loop in the larger code? > > 1) Yes > > 2) How are you using LU when A has a nullspace? > My bad - that code is in there from when I was trying iterative solvers. Should I remove it if I use LU or will PETSc just ignore it? > > 3) You can also move out SetFromOptions(). > Thanks for the advice. So, that means I'll build A and b from simulation data, SetOperators(), Solve(), and carry on? > > Matt > > Thanks. > ---------------------------- > Matthew Young > PhD Candidate > Astronomy Department > Boston University > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From bsmith at mcs.anl.gov Tue Apr 17 16:24:05 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 17 Apr 2018 21:24:05 +0000 Subject: [petsc-users] question on VecPointwiseMax In-Reply-To: References: Message-ID: Where are you call SNESGetSolution() from? Inside a SNES monitor routine, a SNES convergence test routine, a SNESLineSearchPostCheck() routine? The SNES linear solvers are solving the system, you really shouldn't be changing the values of the solution any time while they are doing their job solving the system. Of course you can change the values after the solver is done. If you are trying to "solve" a linear system with upper and/or lower bounds on the variables you should solve it as a variational inequality and use SNESVISetVariableBounds(), you should not try to manually restrict the values (that won't work). And use -snes_type vinewtonrsls Barry Barry > On Apr 17, 2018, at 12:04 PM, Xiangdong wrote: > > Hello everyone, > > Here are a little more information. > > Here is the function calls I have: > > SNESGetSolution(snes,&x); > VecPointwiseMax(x,x,xl); Fail > VecPointwiseMax(x,y,xl); Fail > > The error messages are"Object is in wrong state, Vec is locked read only, argument #1." > > It seems that the vector x returned from SNESGetSolution is locked read only, Is it true? > > If I want do some post processing by modifying the x after each nonlinear iteration, how can I get the solution and modify it? > > Thank you. > > Best, > Xiangdong > > > > On Tue, Apr 17, 2018 at 11:33 AM, Xiangdong wrote: > Hello everyone, > > When I call VecPointwiseMax(x,x,y) , I got an error message "object is in wrong state. Vec is locked read only, argument #1." > > However, In the online manual of output parameters of VecPointwiseMax, it says: > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPointwiseMax.html > Output Parameter > > w -the result > > Notes: any subset of the x, y, and w may be the same vector. For complex numbers compares only the real part > > However, in the implementation of VecPointwiseMax_Seq, > http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/impls/seq/bvec2.c.html#VecPointwiseMax_Seq, > > it seems that xin is read locked, and win and xin cannot be the same vectors. > PetscErrorCode VecPointwiseMax_Seq(Vec win,Vec xin,Vec yin) > 17: > { > > 19: PetscInt > n = win->map->n,i; > > 20: PetscScalar *ww,*xx,*yy; /* cannot make xx or yy const since might be ww */ > > > > 23: VecGetArrayRead(xin,(const PetscScalar > **)&xx); > > 24: VecGetArrayRead(yin,(const PetscScalar > **)&yy); > > 25: VecGetArray > (win,&ww); > > > 27: for (i=0; i PetscRealPart(yy[i])); > > > 29: VecRestoreArrayRead(xin,(const PetscScalar > **)&xx); > > 30: VecRestoreArrayRead(yin,(const PetscScalar > **)&yy); > > 31: VecRestoreArray > (win,&ww); > > 32: PetscLogFlops > (n); > > 33: return > (0); > > 34: } > Can w and x really be same vector in the VecPointwiseMax? Thanks. > > Best, > Xiangdong > From may at bu.edu Tue Apr 17 16:25:22 2018 From: may at bu.edu (Young, Matthew, Adam) Date: Tue, 17 Apr 2018 21:25:22 +0000 Subject: [petsc-users] Reuse KSP In-Reply-To: <6970E094-0CCC-4452-9330-405D25C98765@anl.gov> References: , <6970E094-0CCC-4452-9330-405D25C98765@anl.gov> Message-ID: That all makes sense. Thanks. Is it okay to MatCreate() and MatDestroy() in each time step or am I just defeating the purpose of preserving the KSP? Is there a way to tell if the KSP is reusing the factorization (e.g., with -ksp_view)? ________________________________ From: Smith, Barry F. Sent: Tuesday, April 17, 2018 5:13:10 PM To: Young, Matthew, Adam Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Reuse KSP If the matrix has a null space then, in general, direct LU factorizations should fail. If they don't fail they can produce an incorrect answer If your matrix has a null space then you should not remove the [Set nullspace of A with MatSetNullSpace] Barry > On Apr 17, 2018, at 7:48 AM, Young, Matthew, Adam wrote: > > > > > From: Matthew Knepley > Sent: Tuesday, April 17, 2018 8:38 AM > To: Young, Matthew, Adam > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Reuse KSP > > On Tue, Apr 17, 2018 at 8:33 AM, Young, Matthew, Adam wrote: > My code (C/C++) uses PETSc to solve a linear system at each iteration in a time-stepping scheme. The PETSc routines are separated out because the scheme that uses them is optional in the larger code. > > At each time step, the PETSc scheme calls a routine that includes: > > { > Mat A; > Vec x,b; > KSP ksp; > > [Set up A,x,b] > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > [Build A and b from simulation data] > > [Set nullspace of A with MatSetNullSpace] > > KSPSetOperators(ksp,A,A); > KSPSetFromOptions(ksp); > KSPSolve(ksp,b,x); > > [Write x into an instance of our array class, perform checks, etc.] > > KSPDestroy(&ksp); > > return(0); > } > > My question is: If I want to use superLU_DIST with the option -mat_superlu_dist_fact SamePattern (or SamePattern_SameRowPerm), should I move KSPCreate/Destroy outside of the time-step loop in the larger code? > > 1) Yes > > 2) How are you using LU when A has a nullspace? > My bad - that code is in there from when I was trying iterative solvers. Should I remove it if I use LU or will PETSc just ignore it? > > 3) You can also move out SetFromOptions(). > Thanks for the advice. So, that means I'll build A and b from simulation data, SetOperators(), Solve(), and carry on? > > Matt > > Thanks. > ---------------------------- > Matthew Young > PhD Candidate > Astronomy Department > Boston University > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 17 16:39:18 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 17 Apr 2018 21:39:18 +0000 Subject: [petsc-users] Reuse KSP In-Reply-To: References: <6970E094-0CCC-4452-9330-405D25C98765@anl.gov> Message-ID: <6C5417AC-BAD6-43CB-B784-FEA67F5DF09A@mcs.anl.gov> > On Apr 17, 2018, at 4:25 PM, Young, Matthew, Adam wrote: > > That all makes sense. Thanks. > > Is it okay to MatCreate() and MatDestroy() in each time step or am I just defeating the purpose of preserving the KSP? You are defeating the purpose. Barry > > Is there a way to tell if the KSP is reusing the factorization (e.g., with -ksp_view)? > From: Smith, Barry F. > Sent: Tuesday, April 17, 2018 5:13:10 PM > To: Young, Matthew, Adam > Cc: Matthew Knepley; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Reuse KSP > > > If the matrix has a null space then, in general, direct LU factorizations should fail. If they don't fail they can produce an incorrect answer > > If your matrix has a null space then you should not remove the [Set nullspace of A with MatSetNullSpace] > > Barry > > > On Apr 17, 2018, at 7:48 AM, Young, Matthew, Adam wrote: > > > > > > > > > > From: Matthew Knepley > > Sent: Tuesday, April 17, 2018 8:38 AM > > To: Young, Matthew, Adam > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] Reuse KSP > > > > On Tue, Apr 17, 2018 at 8:33 AM, Young, Matthew, Adam wrote: > > My code (C/C++) uses PETSc to solve a linear system at each iteration in a time-stepping scheme. The PETSc routines are separated out because the scheme that uses them is optional in the larger code. > > > > At each time step, the PETSc scheme calls a routine that includes: > > > > { > > Mat A; > > Vec x,b; > > KSP ksp; > > > > [Set up A,x,b] > > > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > > > [Build A and b from simulation data] > > > > [Set nullspace of A with MatSetNullSpace] > > > > KSPSetOperators(ksp,A,A); > > KSPSetFromOptions(ksp); > > KSPSolve(ksp,b,x); > > > > [Write x into an instance of our array class, perform checks, etc.] > > > > KSPDestroy(&ksp); > > > > return(0); > > } > > > > My question is: If I want to use superLU_DIST with the option -mat_superlu_dist_fact SamePattern (or SamePattern_SameRowPerm), should I move KSPCreate/Destroy outside of the time-step loop in the larger code? > > > > 1) Yes > > > > 2) How are you using LU when A has a nullspace? > > My bad - that code is in there from when I was trying iterative solvers. Should I remove it if I use LU or will PETSc just ignore it? > > > > 3) You can also move out SetFromOptions(). > > Thanks for the advice. So, that means I'll build A and b from simulation data, SetOperators(), Solve(), and carry on? > > > > Matt > > > > Thanks. > > ---------------------------- > > Matthew Young > > PhD Candidate > > Astronomy Department > > Boston University > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ From knepley at gmail.com Tue Apr 17 17:59:12 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 17 Apr 2018 18:59:12 -0400 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: Message-ID: On Tue, Apr 17, 2018 at 3:07 PM, Sepideh Kavousi wrote: > The reason I can not use your method is that the input arguments of > SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your > method which was: > > ierr *=* DMGetLocalVector(dm,*&*Xdotloc);CHKERRQ(ierr); ierr *=* DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr *=* DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > ierr *=* DMDAVecGetArray(dm,Xdotloc,*&*xdot);CHKERRQ(ierr); > > You misunderstand my suggestion. I mean stick this code in right here in PETSc https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a 0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master& fileviewer=file-view-default#dmdats.c-68 Then the X_t array you get in your local function will be ghosted. Matt > I need to have the vector of Xdot, not the array. So I think I should use > SetIFunction instead of SetIFunctionLocal. > > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, April 17, 2018 1:22:53 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi wrote: > > Mathew, > I previously use DMDATSSetIFunctionLocal(user.d > a,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. > If I want to use your solution I can not use it because in the FormFunction > definition I must use arrays, not vectors.So to solve this issue I followed > two methods where none were able to solve it. > 1- in first method I decided to use TSSetIFunction instead of > DMDATSSetIFunctionLocal > > for this means first in the main function, I use TSSetDM and my form > function variables were as: > PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, > struct VAR_STRUCT *user) { > . > . > . > . > ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); > ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); > > ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local); > CHKERRQ(ierr); > ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKER > RQ(ierr); > . > . > . > > } > But still, it does not consider vectors y,ydot,f related to dmda (problem > executing DMDAVecGetArray) > > > We cannot help you if you do not show full error messages. > > Why not fix the code with SetIFunctionLocal(), as I said in my last email. > I will fix PETSc proper in branch at the end of the week. I > have a proposal due tomorrow, so I cannot do it right now. > > Thanks, > > Matt > > > 2- in second method I decided to use DMTSSetIFunction > but still, FormFunction is in form of TSIFunction where we do not define > dm object and I think it does not understand dm and da are connected, > although I have used TSSetDM in the main function. > > Can you please help me what should I do? > Regards, > Sepideh > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 9:34:01 PM > > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi wrote: > > I run with valgrind with the following command: > > /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q > --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off > -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler > -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr > -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason > -ksp_monitor > > > Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only > chooses to ghost the input vector X and not X_t > > https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a > 0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master& > fileviewer=file-view-default#dmdats.c-68 > > Unfortunately, this was done by Peter in 2012 and he is gone, so we can't > blame him. > It is true that very few people use spatial derivatives of X_t, but it > does make sense. > Right there, you want to add > > ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > > ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); > > > and it should work. > > Matt > > > the valgringoutput is as following: > > ==39396== Invalid read of size 8 > ==39396== at 0x402679: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 > alloc'd > ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) > ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) > ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) > ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) > ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) > ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== > ==39396== Invalid read of size 8 > ==39396== at 0x402689: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd > ==39396== > > Thanks, > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:44:43 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi wrote: > > this is not used forghost points . my code is : > > > 1) You are using ghost points, since you use i-1 as an index > > 2) I was wrong. X_t should also be ghosted. > > Run under valgrind. > > Thanks, > > Matt > > > for (j=info2->ys;jys+info2->ym;j++){ > for (i=info2->xs;ixs+info2->xm;i++){ > if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; > aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} > else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} > else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; > aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} > else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} > else { > //derivatives of U and p > Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); > Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); > Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); > Pxy=((aY[j+1][i+1].p+aY[j-1][ > i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); > Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); > Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); > Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); > > > if (((Px*Px)+(Py*Py))>0.0 ) { // > theta=PetscAtanReal(Py/Px); > thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); > thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); > } > else { > theta=PETSC_PI*0.5; > thetax=0.0; > thetay=0.0; > } > e=(1.0+user->epsilon*cos(4.0*theta));; > ep=-(4.0*user->epsilon*sin(4.0*theta)); > epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); > > > > Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); > Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); > Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); > Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); > > U1=user->D*user->tau_0/(user->W*user->W); > U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : > 0.0 ; > //U2=0.0; > P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; > > aF[j][i].U=(U1*(-0.5*(Ux*Px+U > y*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][ > i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k) > *aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)* > aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; > > aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax > *Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i] > .p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i]. > p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1 > -(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0 > +4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; > > } > } > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:36:37 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi wrote: > > Hello, > I am solving a PDE where I have the spacial derivtive of the time > derivative of a variable. In the DMDATSSetIFunctionLocal function I > defined d(dY/dt)/dx as: > (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) > > > I do not think that you get ghosted Ydot in that function. > > Matt > > > on my workstation, it is working but on the cluster it gives an error. I > can not run with debugger on the cluster but I have checked and the error > is related to this part and I do not have any problem on cluster when > running in series. Can you please help me about it. > > the error is: > 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d > ocumentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 > /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c > [0]PETSC ERROR: [0] TS user implicit function line 829 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] TSComputeIFunction line 815 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNES user function line 2202 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESComputeFunction line 2187 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 > /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c > [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/s > nes/interface/snes.c > [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/t > s/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 > 20:11:44 2018 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > ==29057== > ==29057== HEAP SUMMARY: > ==29057== in use at exit: 560,151 bytes in 383 blocks > ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes > allocated > ==29057== > ==29057== LEAK SUMMARY: > ==29057== definitely lost: 0 bytes in 0 blocks > ==29057== indirectly lost: 0 bytes in 0 blocks > ==29057== possibly lost: 0 bytes in 0 blocks > ==29057== still reachable: 560,151 bytes in 383 blocks > ==29057== suppressed: 0 bytes in 0 blocks > ==29057== Rerun with --leak-check=full to see details of leaked memory > ==29057== > ==29057== For counts of detected and suppressed errors, rerun with: -v > ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) > > > > > Thanks, > Sepideh > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sonia.pozzi at usi.ch Wed Apr 18 07:01:03 2018 From: sonia.pozzi at usi.ch (Sonia Pozzi) Date: Wed, 18 Apr 2018 14:01:03 +0200 Subject: [petsc-users] transform in a block matrix Message-ID: Dear Petsc users and developers, I have a MPIAIJ matrix (already assembled) and I would like to transform it in a block matrix with of block dimension 3. Is there a easy way to do that? Thank you in advance, Sonia Ph.D. Student Group of Prof. Rolf Krause Institute of Computational Science Universit? della Svizzera italiana Center for Computation Medicine in Cardiology Via Giuseppe Buffi, 13 CH-6900 Lugano Switzerland Email: pozzis at usi.ch Phone: +41 58 666 4972 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 18 07:04:23 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 18 Apr 2018 08:04:23 -0400 Subject: [petsc-users] transform in a block matrix In-Reply-To: References: Message-ID: On Wed, Apr 18, 2018 at 8:01 AM, Sonia Pozzi wrote: > Dear Petsc users and developers, > > I have a MPIAIJ matrix (already assembled) and I would like to transform > it in a block matrix with of block dimension 3. > Is there a easy way to do that? > I believe that http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatConvert.html will work. Thanks, Matt > Thank you in advance, > > Sonia > > > Ph.D. Student > Group of Prof. Rolf Krause > Institute of Computational Science > Universit? della Svizzera italiana > Center for Computation Medicine in Cardiology > Via Giuseppe Buffi, 13 > > CH-6900 Lugano > > Switzerland > > > Email: pozzis at usi.ch > Phone: +41 58 666 4972 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From k_burkart at yahoo.com Wed Apr 18 10:48:52 2018 From: k_burkart at yahoo.com (Klaus Burkart) Date: Wed, 18 Apr 2018 15:48:52 +0000 (UTC) Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> Message-ID: <246102299.2103671.1524066532893@mail.yahoo.com> More questions about matrix and vector type selection for my application: My starting point is a huge sparse matrix which can be symmetric or asymmetric and a rhs vector. There's no defined local or block structure at all, just row and column indices and the values and an array style rhs vector together describing the entire linear system to be solved. With quite some effort, I should be able to create an array nnz[N] containing the number of nonzeros per row in the global matrix for memory allocation which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I assume here, that 0 indicates different numbers of nonzero values in each row, the exact number being stored in the nnz array. Regarding this detail but one example assume a constant number of nz per row so I am not sure whether I should write 0 or NULL for nz? I started with: ??? MatCreate(PETSC_COMM_WORLD, &M); ??? MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); ??? MatSetFromOptions(M); taken from a paper and assume, the latter would set the matrix type to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe decompositioning took place at an earlier stage and the authors of the paper were able to retrieve the local data and structure. What type of matrix and vector should I use for my application e.g. MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, nnz); for efficient memory allocation? In this case, where would the? decompositioning / MPI process allocation take place? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 18 11:37:51 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 18 Apr 2018 16:37:51 +0000 Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: <246102299.2103671.1524066532893@mail.yahoo.com> References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> Message-ID: If you can only generate the nonzero allocation sequentially you can only solve sequentially which means your matrix is MATSEQAIJ and your vector is VECSEQ and your communicator is PETSC_COMM_SELF. If you pass and array for nnz, what you pass for nz is irrelevant, you might as well pass 0. Barry > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: > > More questions about matrix and vector type selection for my application: > > My starting point is a huge sparse matrix which can be symmetric or asymmetric and a rhs vector. There's no defined local or block structure at all, just row and column indices and the values and an array style rhs vector together describing the entire linear system to be solved. With quite some effort, I should be able to create an array nnz[N] containing the number of nonzeros per row in the global matrix for memory allocation which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I assume here, that 0 indicates different numbers of nonzero values in each row, the exact number being stored in the nnz array. Regarding this detail but one example assume a constant number of nz per row so I am not sure whether I should write 0 or NULL for nz? > > I started with: > > MatCreate(PETSC_COMM_WORLD, &M); > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > MatSetFromOptions(M); > > taken from a paper and assume, the latter would set the matrix type to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe decompositioning took place at an earlier stage and the authors of the paper were able to retrieve the local data and structure. > > What type of matrix and vector should I use for my application e.g. MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, nnz); for efficient memory allocation? > > In this case, where would the decompositioning / MPI process allocation take place? From skavou1 at lsu.edu Wed Apr 18 15:06:05 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Wed, 18 Apr 2018 20:06:05 +0000 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: , Message-ID: Mathew I added the lines and I still have the same issue. It may be a silly question but should I configure and install petsc again using this new lines added? or changing the line is enough? the highlighted lines are the lines I modified. PetscErrorCode ierr; DM dm; DMTS_DA *dmdats = (DMTS_DA*)ctx; DMDALocalInfo info; Vec Xloc,Xdotloc; void *x,*f,*xdot; PetscFunctionBegin; PetscValidHeaderSpecific(ts,TS_CLASSID,1); PetscValidHeaderSpecific(X,VEC_CLASSID,2); PetscValidHeaderSpecific(F,VEC_CLASSID,3); if (!dmdats->ifunctionlocal) SETERRQ(PetscObjectComm((PetscObject)ts),PETSC_ERR_PLIB,"Corrupt context"); ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGetLocalVector(dm,&Xloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xloc,&x);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); Thanks, Sepideh ________________________________ From: Matthew Knepley Sent: Tuesday, April 17, 2018 5:59:12 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Tue, Apr 17, 2018 at 3:07 PM, Sepideh Kavousi > wrote: The reason I can not use your method is that the input arguments of SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your method which was: ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); You misunderstand my suggestion. I mean stick this code in right here in PETSc https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Then the X_t array you get in your local function will be ghosted. Matt I need to have the vector of Xdot, not the array. So I think I should use SetIFunction instead of SetIFunctionLocal. Sepideh ________________________________ From: Matthew Knepley > Sent: Tuesday, April 17, 2018 1:22:53 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi > wrote: Mathew, I previously use DMDATSSetIFunctionLocal(user.da,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. If I want to use your solution I can not use it because in the FormFunction definition I must use arrays, not vectors.So to solve this issue I followed two methods where none were able to solve it. 1- in first method I decided to use TSSetIFunction instead of DMDATSSetIFunctionLocal for this means first in the main function, I use TSSetDM and my form function variables were as: PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, struct VAR_STRUCT *user) { . . . . ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); . . . } But still, it does not consider vectors y,ydot,f related to dmda (problem executing DMDAVecGetArray) We cannot help you if you do not show full error messages. Why not fix the code with SetIFunctionLocal(), as I said in my last email. I will fix PETSc proper in branch at the end of the week. I have a proposal due tomorrow, so I cannot do it right now. Thanks, Matt 2- in second method I decided to use DMTSSetIFunction but still, FormFunction is in form of TSIFunction where we do not define dm object and I think it does not understand dm and da are connected, although I have used TSSetDM in the main function. Can you please help me what should I do? Regards, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 9:34:01 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi > wrote: I run with valgrind with the following command: /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason -ksp_monitor Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only chooses to ghost the input vector X and not X_t https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Unfortunately, this was done by Peter in 2012 and he is gone, so we can't blame him. It is true that very few people use spatial derivatives of X_t, but it does make sense. Right there, you want to add ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); and it should work. Matt the valgringoutput is as following: ==39396== Invalid read of size 8 ==39396== at 0x402679: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 alloc'd ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== ==39396== Invalid read of size 8 ==39396== at 0x402689: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd ==39396== Thanks, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:44:43 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi > wrote: this is not used forghost points . my code is : 1) You are using ghost points, since you use i-1 as an index 2) I was wrong. X_t should also be ghosted. Run under valgrind. Thanks, Matt for (j=info2->ys;jys+info2->ym;j++){ for (i=info2->xs;ixs+info2->xm;i++){ if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} else { //derivatives of U and p Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); Pxy=((aY[j+1][i+1].p+aY[j-1][i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); if (((Px*Px)+(Py*Py))>0.0 ) { // theta=PetscAtanReal(Py/Px); thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); } else { theta=PETSC_PI*0.5; thetax=0.0; thetay=0.0; } e=(1.0+user->epsilon*cos(4.0*theta));; ep=-(4.0*user->epsilon*sin(4.0*theta)); epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); U1=user->D*user->tau_0/(user->W*user->W); U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : 0.0 ; //U2=0.0; P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; aF[j][i].U=(U1*(-0.5*(Ux*Px+Uy*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k)*aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)*aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax*Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i].p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; } } Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:36:37 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi > wrote: Hello, I am solving a PDE where I have the spacial derivtive of the time derivative of a variable. In the DMDATSSetIFunctionLocal function I defined d(dY/dt)/dx as: (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) I do not think that you get ghosted Ydot in that function. Matt on my workstation, it is working but on the cluster it gives an error. I can not run with debugger on the cluster but I have checked and the error is related to this part and I do not have any problem on cluster when running in series. Can you please help me about it. the error is: 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c [0]PETSC ERROR: [0] TS user implicit function line 829 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSComputeIFunction line 815 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNES user function line 2202 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESComputeFunction line 2187 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 20:11:44 2018 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ==29057== ==29057== HEAP SUMMARY: ==29057== in use at exit: 560,151 bytes in 383 blocks ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes allocated ==29057== ==29057== LEAK SUMMARY: ==29057== definitely lost: 0 bytes in 0 blocks ==29057== indirectly lost: 0 bytes in 0 blocks ==29057== possibly lost: 0 bytes in 0 blocks ==29057== still reachable: 560,151 bytes in 383 blocks ==29057== suppressed: 0 bytes in 0 blocks ==29057== Rerun with --leak-check=full to see details of leaked memory ==29057== ==29057== For counts of detected and suppressed errors, rerun with: -v ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) Thanks, Sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 18 15:08:06 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 18 Apr 2018 16:08:06 -0400 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: Message-ID: On Wed, Apr 18, 2018 at 4:06 PM, Sepideh Kavousi wrote: > Mathew > > I added the lines and I still have the same issue. It may be a silly > question but should I configure and install petsc again using this new > lines added? > No, but you need to make cd $PETSC_DIR make all Thanks, Matt > or changing the line is enough? the highlighted lines are the lines I > modified. > > > PetscErrorCode ierr; > DM dm; > DMTS_DA *dmdats = (DMTS_DA*)ctx; > DMDALocalInfo info; > Vec Xloc,Xdotloc; > void *x,*f,*xdot; > > PetscFunctionBegin; > PetscValidHeaderSpecific(ts,TS_CLASSID,1); > PetscValidHeaderSpecific(X,VEC_CLASSID,2); > PetscValidHeaderSpecific(F,VEC_CLASSID,3); > if (!dmdats->ifunctionlocal) SETERRQ(PetscObjectComm(( > PetscObject)ts),PETSC_ERR_PLIB,"Corrupt context"); > ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); > ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc); > CHKERRQ(ierr); > ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > ierr = DMGetLocalVector(dm,&Xloc);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); > ierr = DMGlobalToLocalEnd(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); > ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr); > ierr = DMDAVecGetArray(dm,Xloc,&x);CHKERRQ(ierr); > ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); > > Thanks, > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, April 17, 2018 5:59:12 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Tue, Apr 17, 2018 at 3:07 PM, Sepideh Kavousi wrote: > > The reason I can not use your method is that the input arguments of > SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your > method which was: > > ierr *=* DMGetLocalVector(dm,*&*Xdotloc);CHKERRQ(ierr); ierr *=* DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr *=* DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > ierr *=* DMDAVecGetArray(dm,Xdotloc,*&*xdot);CHKERRQ(ierr); > > > You misunderstand my suggestion. I mean stick this code in right here in > PETSc > > https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a > 0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&file > viewer=file-view-default#dmdats.c-68 > > Then the X_t array you get in your local function will be ghosted. > > Matt > > > I need to have the vector of Xdot, not the array. So I think I should use > SetIFunction instead of SetIFunctionLocal. > > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, April 17, 2018 1:22:53 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi wrote: > > Mathew, > I previously use DMDATSSetIFunctionLocal(user.d > a,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. > If I want to use your solution I can not use it because in the FormFunction > definition I must use arrays, not vectors.So to solve this issue I followed > two methods where none were able to solve it. > 1- in first method I decided to use TSSetIFunction instead of > DMDATSSetIFunctionLocal > > for this means first in the main function, I use TSSetDM and my form > function variables were as: > PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, > struct VAR_STRUCT *user) { > . > . > . > . > ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); > ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); > > ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local);CHK > ERRQ(ierr); > ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKER > RQ(ierr); > . > . > . > > } > But still, it does not consider vectors y,ydot,f related to dmda (problem > executing DMDAVecGetArray) > > > We cannot help you if you do not show full error messages. > > Why not fix the code with SetIFunctionLocal(), as I said in my last email. > I will fix PETSc proper in branch at the end of the week. I > have a proposal due tomorrow, so I cannot do it right now. > > Thanks, > > Matt > > > 2- in second method I decided to use DMTSSetIFunction > but still, FormFunction is in form of TSIFunction where we do not define > dm object and I think it does not understand dm and da are connected, > although I have used TSSetDM in the main function. > > Can you please help me what should I do? > Regards, > Sepideh > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 9:34:01 PM > > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi wrote: > > I run with valgrind with the following command: > > /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q > --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off > -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler > -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr > -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason > -ksp_monitor > > > Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only > chooses to ghost the input vector X and not X_t > > https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a > 0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&file > viewer=file-view-default#dmdats.c-68 > > Unfortunately, this was done by Peter in 2012 and he is gone, so we can't > blame him. > It is true that very few people use spatial derivatives of X_t, but it > does make sense. > Right there, you want to add > > ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > > ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); > > > and it should work. > > Matt > > > the valgringoutput is as following: > > ==39396== Invalid read of size 8 > ==39396== at 0x402679: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 > alloc'd > ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) > ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) > ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) > ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) > ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) > ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== > ==39396== Invalid read of size 8 > ==39396== at 0x402689: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd > ==39396== > > Thanks, > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:44:43 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi wrote: > > this is not used forghost points . my code is : > > > 1) You are using ghost points, since you use i-1 as an index > > 2) I was wrong. X_t should also be ghosted. > > Run under valgrind. > > Thanks, > > Matt > > > for (j=info2->ys;jys+info2->ym;j++){ > for (i=info2->xs;ixs+info2->xm;i++){ > if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; > aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} > else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} > else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; > aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} > else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} > else { > //derivatives of U and p > Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); > Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); > Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); > Pxy=((aY[j+1][i+1].p+aY[j-1][ > i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); > Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); > Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); > Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); > > > if (((Px*Px)+(Py*Py))>0.0 ) { // > theta=PetscAtanReal(Py/Px); > thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); > thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); > } > else { > theta=PETSC_PI*0.5; > thetax=0.0; > thetay=0.0; > } > e=(1.0+user->epsilon*cos(4.0*theta));; > ep=-(4.0*user->epsilon*sin(4.0*theta)); > epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); > > > > Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); > Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); > Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); > Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); > > U1=user->D*user->tau_0/(user->W*user->W); > U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : > 0.0 ; > //U2=0.0; > P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; > > aF[j][i].U=(U1*(-0.5*(Ux*Px+U > y*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][ > i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k) > *aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)* > aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; > > aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax > *Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i] > .p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i]. > p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1 > -(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0 > +4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; > > } > } > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:36:37 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi wrote: > > Hello, > I am solving a PDE where I have the spacial derivtive of the time > derivative of a variable. In the DMDATSSetIFunctionLocal function I > defined d(dY/dt)/dx as: > (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) > > > I do not think that you get ghosted Ydot in that function. > > Matt > > > on my workstation, it is working but on the cluster it gives an error. I > can not run with debugger on the cluster but I have checked and the error > is related to this part and I do not have any problem on cluster when > running in series. Can you please help me about it. > > the error is: > 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d > ocumentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 > /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c > [0]PETSC ERROR: [0] TS user implicit function line 829 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] TSComputeIFunction line 815 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNES user function line 2202 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESComputeFunction line 2187 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 > /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c > [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/s > nes/interface/snes.c > [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/t > s/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 > 20:11:44 2018 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > ==29057== > ==29057== HEAP SUMMARY: > ==29057== in use at exit: 560,151 bytes in 383 blocks > ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes > allocated > ==29057== > ==29057== LEAK SUMMARY: > ==29057== definitely lost: 0 bytes in 0 blocks > ==29057== indirectly lost: 0 bytes in 0 blocks > ==29057== possibly lost: 0 bytes in 0 blocks > ==29057== still reachable: 560,151 bytes in 383 blocks > ==29057== suppressed: 0 bytes in 0 blocks > ==29057== Rerun with --leak-check=full to see details of leaked memory > ==29057== > ==29057== For counts of detected and suppressed errors, rerun with: -v > ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) > > > > > Thanks, > Sepideh > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Wed Apr 18 15:13:33 2018 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 18 Apr 2018 21:13:33 +0100 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: Message-ID: On 18 April 2018 at 21:06, Sepideh Kavousi wrote: > Mathew > > I added the lines and I still have the same issue. It may be a silly > question but should I configure and install petsc again using this new > lines added? or changing the line is enough? the highlighted lines are the > lines I modified. > > > PetscErrorCode ierr; > DM dm; > DMTS_DA *dmdats = (DMTS_DA*)ctx; > DMDALocalInfo info; > Vec Xloc,Xdotloc; > void *x,*f,*xdot; > > PetscFunctionBegin; > PetscValidHeaderSpecific(ts,TS_CLASSID,1); > PetscValidHeaderSpecific(X,VEC_CLASSID,2); > PetscValidHeaderSpecific(F,VEC_CLASSID,3); > if (!dmdats->ifunctionlocal) SETERRQ(PetscObjectComm(( > PetscObject)ts),PETSC_ERR_PLIB,"Corrupt context"); > ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); > ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc); > CHKERRQ(ierr); > ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > ierr = DMGetLocalVector(dm,&Xloc);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); > ierr = DMGlobalToLocalEnd(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); > ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr); > ierr = DMDAVecGetArray(dm,Xloc,&x);CHKERRQ(ierr); > ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); > > Don't forget to include these calls (in this order) after you are done with the Xdotloc vector ierr = DMDAVecRestoreArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); ierr = DMRestoreLocalVector(dm,&Xdotloc);CHKERRQ(ierr); Failure to do so will result in a memory leak. > Thanks, > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, April 17, 2018 5:59:12 PM > > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Tue, Apr 17, 2018 at 3:07 PM, Sepideh Kavousi wrote: > > The reason I can not use your method is that the input arguments of > SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your > method which was: > > ierr *=* DMGetLocalVector(dm,*&*Xdotloc);CHKERRQ(ierr); ierr *=* DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr *=* DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > ierr *=* DMDAVecGetArray(dm,Xdotloc,*&*xdot);CHKERRQ(ierr); > > > You misunderstand my suggestion. I mean stick this code in right here in > PETSc > > https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a > 0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&file > viewer=file-view-default#dmdats.c-68 > > Then the X_t array you get in your local function will be ghosted. > > Matt > > > I need to have the vector of Xdot, not the array. So I think I should use > SetIFunction instead of SetIFunctionLocal. > > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, April 17, 2018 1:22:53 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi wrote: > > Mathew, > I previously use DMDATSSetIFunctionLocal(user.d > a,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. > If I want to use your solution I can not use it because in the FormFunction > definition I must use arrays, not vectors.So to solve this issue I followed > two methods where none were able to solve it. > 1- in first method I decided to use TSSetIFunction instead of > DMDATSSetIFunctionLocal > > for this means first in the main function, I use TSSetDM and my form > function variables were as: > PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, > struct VAR_STRUCT *user) { > . > . > . > . > ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); > ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); > > ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local);CHK > ERRQ(ierr); > ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKER > RQ(ierr); > . > . > . > > } > But still, it does not consider vectors y,ydot,f related to dmda (problem > executing DMDAVecGetArray) > > > We cannot help you if you do not show full error messages. > > Why not fix the code with SetIFunctionLocal(), as I said in my last email. > I will fix PETSc proper in branch at the end of the week. I > have a proposal due tomorrow, so I cannot do it right now. > > Thanks, > > Matt > > > 2- in second method I decided to use DMTSSetIFunction > but still, FormFunction is in form of TSIFunction where we do not define > dm object and I think it does not understand dm and da are connected, > although I have used TSSetDM in the main function. > > Can you please help me what should I do? > Regards, > Sepideh > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 9:34:01 PM > > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi wrote: > > I run with valgrind with the following command: > > /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q > --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off > -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler > -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr > -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason > -ksp_monitor > > > Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only > chooses to ghost the input vector X and not X_t > > https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a > 0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&file > viewer=file-view-default#dmdats.c-68 > > Unfortunately, this was done by Peter in 2012 and he is gone, so we can't > blame him. > It is true that very few people use spatial derivatives of X_t, but it > does make sense. > Right there, you want to add > > ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > > ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); > > > and it should work. > > Matt > > > the valgringoutput is as following: > > ==39396== Invalid read of size 8 > ==39396== at 0x402679: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 > alloc'd > ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) > ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) > ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) > ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) > ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) > ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== > ==39396== Invalid read of size 8 > ==39396== at 0x402689: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd > ==39396== > > Thanks, > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:44:43 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi wrote: > > this is not used forghost points . my code is : > > > 1) You are using ghost points, since you use i-1 as an index > > 2) I was wrong. X_t should also be ghosted. > > Run under valgrind. > > Thanks, > > Matt > > > for (j=info2->ys;jys+info2->ym;j++){ > for (i=info2->xs;ixs+info2->xm;i++){ > if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; > aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} > else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} > else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; > aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} > else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} > else { > //derivatives of U and p > Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); > Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); > Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); > Pxy=((aY[j+1][i+1].p+aY[j-1][ > i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); > Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); > Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); > Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); > > > if (((Px*Px)+(Py*Py))>0.0 ) { // > theta=PetscAtanReal(Py/Px); > thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); > thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); > } > else { > theta=PETSC_PI*0.5; > thetax=0.0; > thetay=0.0; > } > e=(1.0+user->epsilon*cos(4.0*theta));; > ep=-(4.0*user->epsilon*sin(4.0*theta)); > epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); > > > > Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); > Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); > Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); > Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); > > U1=user->D*user->tau_0/(user->W*user->W); > U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : > 0.0 ; > //U2=0.0; > P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; > > aF[j][i].U=(U1*(-0.5*(Ux*Px+U > y*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][ > i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k) > *aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)* > aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; > > aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax > *Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i] > .p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i]. > p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1 > -(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0 > +4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; > > } > } > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:36:37 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi wrote: > > Hello, > I am solving a PDE where I have the spacial derivtive of the time > derivative of a variable. In the DMDATSSetIFunctionLocal function I > defined d(dY/dt)/dx as: > (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) > > > I do not think that you get ghosted Ydot in that function. > > Matt > > > on my workstation, it is working but on the cluster it gives an error. I > can not run with debugger on the cluster but I have checked and the error > is related to this part and I do not have any problem on cluster when > running in series. Can you please help me about it. > > the error is: > 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d > ocumentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 > /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c > [0]PETSC ERROR: [0] TS user implicit function line 829 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] TSComputeIFunction line 815 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNES user function line 2202 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESComputeFunction line 2187 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 > /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c > [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/s > nes/interface/snes.c > [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/t > s/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 > 20:11:44 2018 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > ==29057== > ==29057== HEAP SUMMARY: > ==29057== in use at exit: 560,151 bytes in 383 blocks > ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes > allocated > ==29057== > ==29057== LEAK SUMMARY: > ==29057== definitely lost: 0 bytes in 0 blocks > ==29057== indirectly lost: 0 bytes in 0 blocks > ==29057== possibly lost: 0 bytes in 0 blocks > ==29057== still reachable: 560,151 bytes in 383 blocks > ==29057== suppressed: 0 bytes in 0 blocks > ==29057== Rerun with --leak-check=full to see details of leaked memory > ==29057== > ==29057== For counts of detected and suppressed errors, rerun with: -v > ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) > > > > > Thanks, > Sepideh > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k_burkart at yahoo.com Wed Apr 18 16:42:30 2018 From: k_burkart at yahoo.com (k_burkart at yahoo.com) Date: Wed, 18 Apr 2018 23:42:30 +0200 Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> Message-ID: An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Wed Apr 18 16:52:28 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Wed, 18 Apr 2018 21:52:28 +0000 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: , Message-ID: Mathew and Dave, Thank you so much it is working perfectly now. Sepideh ________________________________ From: Dave May Sent: Wednesday, April 18, 2018 3:13:33 PM To: Sepideh Kavousi Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On 18 April 2018 at 21:06, Sepideh Kavousi > wrote: Mathew I added the lines and I still have the same issue. It may be a silly question but should I configure and install petsc again using this new lines added? or changing the line is enough? the highlighted lines are the lines I modified. PetscErrorCode ierr; DM dm; DMTS_DA *dmdats = (DMTS_DA*)ctx; DMDALocalInfo info; Vec Xloc,Xdotloc; void *x,*f,*xdot; PetscFunctionBegin; PetscValidHeaderSpecific(ts,TS_CLASSID,1); PetscValidHeaderSpecific(X,VEC_CLASSID,2); PetscValidHeaderSpecific(F,VEC_CLASSID,3); if (!dmdats->ifunctionlocal) SETERRQ(PetscObjectComm((PetscObject)ts),PETSC_ERR_PLIB,"Corrupt context"); ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGetLocalVector(dm,&Xloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xloc,&x);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); Don't forget to include these calls (in this order) after you are done with the Xdotloc vector ierr = DMDAVecRestoreArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); ierr = DMRestoreLocalVector(dm,&Xdotloc);CHKERRQ(ierr); Failure to do so will result in a memory leak. Thanks, Sepideh ________________________________ From: Matthew Knepley > Sent: Tuesday, April 17, 2018 5:59:12 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Tue, Apr 17, 2018 at 3:07 PM, Sepideh Kavousi > wrote: The reason I can not use your method is that the input arguments of SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your method which was: ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); You misunderstand my suggestion. I mean stick this code in right here in PETSc https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Then the X_t array you get in your local function will be ghosted. Matt I need to have the vector of Xdot, not the array. So I think I should use SetIFunction instead of SetIFunctionLocal. Sepideh ________________________________ From: Matthew Knepley > Sent: Tuesday, April 17, 2018 1:22:53 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi > wrote: Mathew, I previously use DMDATSSetIFunctionLocal(user.da,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. If I want to use your solution I can not use it because in the FormFunction definition I must use arrays, not vectors.So to solve this issue I followed two methods where none were able to solve it. 1- in first method I decided to use TSSetIFunction instead of DMDATSSetIFunctionLocal for this means first in the main function, I use TSSetDM and my form function variables were as: PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, struct VAR_STRUCT *user) { . . . . ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); . . . } But still, it does not consider vectors y,ydot,f related to dmda (problem executing DMDAVecGetArray) We cannot help you if you do not show full error messages. Why not fix the code with SetIFunctionLocal(), as I said in my last email. I will fix PETSc proper in branch at the end of the week. I have a proposal due tomorrow, so I cannot do it right now. Thanks, Matt 2- in second method I decided to use DMTSSetIFunction but still, FormFunction is in form of TSIFunction where we do not define dm object and I think it does not understand dm and da are connected, although I have used TSSetDM in the main function. Can you please help me what should I do? Regards, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 9:34:01 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi > wrote: I run with valgrind with the following command: /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason -ksp_monitor Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only chooses to ghost the input vector X and not X_t https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Unfortunately, this was done by Peter in 2012 and he is gone, so we can't blame him. It is true that very few people use spatial derivatives of X_t, but it does make sense. Right there, you want to add ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); and it should work. Matt the valgringoutput is as following: ==39396== Invalid read of size 8 ==39396== at 0x402679: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 alloc'd ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== ==39396== Invalid read of size 8 ==39396== at 0x402689: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd ==39396== Thanks, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:44:43 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi > wrote: this is not used forghost points . my code is : 1) You are using ghost points, since you use i-1 as an index 2) I was wrong. X_t should also be ghosted. Run under valgrind. Thanks, Matt for (j=info2->ys;jys+info2->ym;j++){ for (i=info2->xs;ixs+info2->xm;i++){ if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} else { //derivatives of U and p Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); Pxy=((aY[j+1][i+1].p+aY[j-1][i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); if (((Px*Px)+(Py*Py))>0.0 ) { // theta=PetscAtanReal(Py/Px); thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); } else { theta=PETSC_PI*0.5; thetax=0.0; thetay=0.0; } e=(1.0+user->epsilon*cos(4.0*theta));; ep=-(4.0*user->epsilon*sin(4.0*theta)); epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); U1=user->D*user->tau_0/(user->W*user->W); U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : 0.0 ; //U2=0.0; P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; aF[j][i].U=(U1*(-0.5*(Ux*Px+Uy*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k)*aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)*aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax*Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i].p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; } } Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:36:37 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi > wrote: Hello, I am solving a PDE where I have the spacial derivtive of the time derivative of a variable. In the DMDATSSetIFunctionLocal function I defined d(dY/dt)/dx as: (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) I do not think that you get ghosted Ydot in that function. Matt on my workstation, it is working but on the cluster it gives an error. I can not run with debugger on the cluster but I have checked and the error is related to this part and I do not have any problem on cluster when running in series. Can you please help me about it. the error is: 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c [0]PETSC ERROR: [0] TS user implicit function line 829 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSComputeIFunction line 815 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNES user function line 2202 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESComputeFunction line 2187 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 20:11:44 2018 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ==29057== ==29057== HEAP SUMMARY: ==29057== in use at exit: 560,151 bytes in 383 blocks ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes allocated ==29057== ==29057== LEAK SUMMARY: ==29057== definitely lost: 0 bytes in 0 blocks ==29057== indirectly lost: 0 bytes in 0 blocks ==29057== possibly lost: 0 bytes in 0 blocks ==29057== still reachable: 560,151 bytes in 383 blocks ==29057== suppressed: 0 bytes in 0 blocks ==29057== Rerun with --leak-check=full to see details of leaked memory ==29057== ==29057== For counts of detected and suppressed errors, rerun with: -v ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) Thanks, Sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 18 17:52:32 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 18 Apr 2018 18:52:32 -0400 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: Message-ID: On Wed, Apr 18, 2018 at 5:52 PM, Sepideh Kavousi wrote: > Mathew and Dave, > > Thank you so much it is working perfectly now. > Excellent. If you want your name to appear on the next PETSc release as a contributor, you can make a PR with this change :) Thanks, Matt > Sepideh > ------------------------------ > *From:* Dave May > *Sent:* Wednesday, April 18, 2018 3:13:33 PM > *To:* Sepideh Kavousi > *Cc:* Matthew Knepley; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > > > On 18 April 2018 at 21:06, Sepideh Kavousi wrote: > > Mathew > > I added the lines and I still have the same issue. It may be a silly > question but should I configure and install petsc again using this new > lines added? or changing the line is enough? the highlighted lines are the > lines I modified. > > > PetscErrorCode ierr; > DM dm; > DMTS_DA *dmdats = (DMTS_DA*)ctx; > DMDALocalInfo info; > Vec Xloc,Xdotloc; > void *x,*f,*xdot; > > PetscFunctionBegin; > PetscValidHeaderSpecific(ts,TS_CLASSID,1); > PetscValidHeaderSpecific(X,VEC_CLASSID,2); > PetscValidHeaderSpecific(F,VEC_CLASSID,3); > if (!dmdats->ifunctionlocal) SETERRQ(PetscObjectComm((Petsc > Object)ts),PETSC_ERR_PLIB,"Corrupt context"); > ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); > ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ( > ierr); > ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > ierr = DMGetLocalVector(dm,&Xloc);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); > ierr = DMGlobalToLocalEnd(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); > ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr); > ierr = DMDAVecGetArray(dm,Xloc,&x);CHKERRQ(ierr); > ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); > > > Don't forget to include these calls (in this order) after you are done > with the Xdotloc vector > > ierr = DMDAVecRestoreArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); > ierr = DMRestoreLocalVector(dm,&Xdotloc);CHKERRQ(ierr); > > Failure to do so will result in a memory leak. > > > > Thanks, > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, April 17, 2018 5:59:12 PM > > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Tue, Apr 17, 2018 at 3:07 PM, Sepideh Kavousi wrote: > > The reason I can not use your method is that the input arguments of > SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your > method which was: > > ierr *=* DMGetLocalVector(dm,*&*Xdotloc);CHKERRQ(ierr); ierr *=* DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr *=* DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > ierr *=* DMDAVecGetArray(dm,Xdotloc,*&*xdot);CHKERRQ(ierr); > > > You misunderstand my suggestion. I mean stick this code in right here in > PETSc > > https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a > 0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&file > viewer=file-view-default#dmdats.c-68 > > Then the X_t array you get in your local function will be ghosted. > > Matt > > > I need to have the vector of Xdot, not the array. So I think I should use > SetIFunction instead of SetIFunctionLocal. > > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, April 17, 2018 1:22:53 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi wrote: > > Mathew, > I previously use DMDATSSetIFunctionLocal(user.d > a,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. > If I want to use your solution I can not use it because in the FormFunction > definition I must use arrays, not vectors.So to solve this issue I followed > two methods where none were able to solve it. > 1- in first method I decided to use TSSetIFunction instead of > DMDATSSetIFunctionLocal > > for this means first in the main function, I use TSSetDM and my form > function variables were as: > PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, > struct VAR_STRUCT *user) { > . > . > . > . > ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); > ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); > > ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local);CHK > ERRQ(ierr); > ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKER > RQ(ierr); > . > . > . > > } > But still, it does not consider vectors y,ydot,f related to dmda (problem > executing DMDAVecGetArray) > > > We cannot help you if you do not show full error messages. > > Why not fix the code with SetIFunctionLocal(), as I said in my last email. > I will fix PETSc proper in branch at the end of the week. I > have a proposal due tomorrow, so I cannot do it right now. > > Thanks, > > Matt > > > 2- in second method I decided to use DMTSSetIFunction > but still, FormFunction is in form of TSIFunction where we do not define > dm object and I think it does not understand dm and da are connected, > although I have used TSSetDM in the main function. > > Can you please help me what should I do? > Regards, > Sepideh > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 9:34:01 PM > > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi wrote: > > I run with valgrind with the following command: > > /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q > --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off > -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler > -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr > -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason > -ksp_monitor > > > Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only > chooses to ghost the input vector X and not X_t > > https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a > 0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&file > viewer=file-view-default#dmdats.c-68 > > Unfortunately, this was done by Peter in 2012 and he is gone, so we can't > blame him. > It is true that very few people use spatial derivatives of X_t, but it > does make sense. > Right there, you want to add > > ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > > ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); > > > and it should work. > > Matt > > > the valgringoutput is as following: > > ==39396== Invalid read of size 8 > ==39396== at 0x402679: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 > alloc'd > ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) > ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) > ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) > ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) > ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) > ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== > ==39396== Invalid read of size 8 > ==39396== at 0x402689: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd > ==39396== > > Thanks, > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:44:43 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi wrote: > > this is not used forghost points . my code is : > > > 1) You are using ghost points, since you use i-1 as an index > > 2) I was wrong. X_t should also be ghosted. > > Run under valgrind. > > Thanks, > > Matt > > > for (j=info2->ys;jys+info2->ym;j++){ > for (i=info2->xs;ixs+info2->xm;i++){ > if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; > aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} > else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} > else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; > aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} > else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} > else { > //derivatives of U and p > Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); > Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); > Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); > Pxy=((aY[j+1][i+1].p+aY[j-1][ > i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); > Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); > Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); > Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); > > > if (((Px*Px)+(Py*Py))>0.0 ) { // > theta=PetscAtanReal(Py/Px); > thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); > thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); > } > else { > theta=PETSC_PI*0.5; > thetax=0.0; > thetay=0.0; > } > e=(1.0+user->epsilon*cos(4.0*theta));; > ep=-(4.0*user->epsilon*sin(4.0*theta)); > epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); > > > > Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); > Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); > Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); > Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); > > U1=user->D*user->tau_0/(user->W*user->W); > U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : > 0.0 ; > //U2=0.0; > P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; > > aF[j][i].U=(U1*(-0.5*(Ux*Px+U > y*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][ > i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k) > *aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)* > aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; > > aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax > *Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i] > .p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i]. > p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1 > -(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0 > +4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; > > } > } > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:36:37 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi wrote: > > Hello, > I am solving a PDE where I have the spacial derivtive of the time > derivative of a variable. In the DMDATSSetIFunctionLocal function I > defined d(dY/dt)/dx as: > (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) > > > I do not think that you get ghosted Ydot in that function. > > Matt > > > on my workstation, it is working but on the cluster it gives an error. I > can not run with debugger on the cluster but I have checked and the error > is related to this part and I do not have any problem on cluster when > running in series. Can you please help me about it. > > the error is: > 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d > ocumentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 > /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c > [0]PETSC ERROR: [0] TS user implicit function line 829 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] TSComputeIFunction line 815 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNES user function line 2202 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESComputeFunction line 2187 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 > /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c > [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/s > nes/interface/snes.c > [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/t > s/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 > 20:11:44 2018 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > ==29057== > ==29057== HEAP SUMMARY: > ==29057== in use at exit: 560,151 bytes in 383 blocks > ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes > allocated > ==29057== > ==29057== LEAK SUMMARY: > ==29057== definitely lost: 0 bytes in 0 blocks > ==29057== indirectly lost: 0 bytes in 0 blocks > ==29057== possibly lost: 0 bytes in 0 blocks > ==29057== still reachable: 560,151 bytes in 383 blocks > ==29057== suppressed: 0 bytes in 0 blocks > ==29057== Rerun with --leak-check=full to see details of leaked memory > ==29057== > ==29057== For counts of detected and suppressed errors, rerun with: -v > ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) > > > > > Thanks, > Sepideh > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 18 18:47:44 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 18 Apr 2018 23:47:44 +0000 Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> Message-ID: > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote: > > So, practically speaking, l should invent routines to decompose the matrix e.g. into a block matrix structure to be able to make real use of PETSc ie. be able to solve a linear system using more than one process/core? To really use PETSc efficiently/effectively you need to generate your matrix in parallel. Barry > > Klaus > > Von meinem Huawei-Mobiltelefon gesendet > > > -------- Originalnachricht -------- > Betreff: Re: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? > Von: "Smith, Barry F." > An: Klaus Burkart > Cc: PETSc Users List > > > > If you can only generate the nonzero allocation sequentially you can only solve sequentially which means your matrix is MATSEQAIJ and your vector is VECSEQ and your communicator is PETSC_COMM_SELF. > > If you pass and array for nnz, what you pass for nz is irrelevant, you might as well pass 0. > > Barry > > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: > > > > More questions about matrix and vector type selection for my application: > > > > My starting point is a huge sparse matrix which can be symmetric or asymmetric and a rhs vector. There's no defined local or block structure at all, just row and column indices and the values and an array style rhs vector together describing the entire linear system to be solved. With quite some effort, I should be able to create an array nnz[N] containing the number of nonzeros per row in the global matrix for memory allocation which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I assume here, that 0 indicates different numbers of nonzero values in each row, the exact number being stored in the nnz array. Regarding this detail but one example assume a constant number of nz per row so I am not sure whether I should write 0 or NULL for nz? > > > > I started with: > > > > MatCreate(PETSC_COMM_WORLD, &M); > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > > MatSetFromOptions(M); > > > > taken from a paper and assume, the latter would set the matrix type to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe decompositioning took place at an earlier stage and the authors of the paper were able to retrieve the local data and structure. > > > > What type of matrix and vector should I use for my application e.g. MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, nnz); for efficient memory allocation? > > > > In this case, where would the decompositioning / MPI process allocation take place? > From dave.mayhem23 at gmail.com Wed Apr 18 23:14:47 2018 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 19 Apr 2018 05:14:47 +0100 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: Message-ID: On 18 April 2018 at 23:52, Matthew Knepley wrote: > On Wed, Apr 18, 2018 at 5:52 PM, Sepideh Kavousi wrote: > >> Mathew and Dave, >> >> Thank you so much it is working perfectly now. >> > > Excellent. > > If you want your name to appear on the next PETSc release as a > contributor, you > can make a PR with this change :) > Here is a URL describing the PR's protocol for PETSc contribs: https://bitbucket.org/petsc/petsc/wiki/pull-request-instructions-git > > Thanks, > > Matt > > >> Sepideh >> ------------------------------ >> *From:* Dave May >> *Sent:* Wednesday, April 18, 2018 3:13:33 PM >> *To:* Sepideh Kavousi >> *Cc:* Matthew Knepley; petsc-users at mcs.anl.gov >> *Subject:* Re: [petsc-users] error running parallel on cluster >> >> >> >> On 18 April 2018 at 21:06, Sepideh Kavousi wrote: >> >> Mathew >> >> I added the lines and I still have the same issue. It may be a silly >> question but should I configure and install petsc again using this new >> lines added? or changing the line is enough? the highlighted lines are the >> lines I modified. >> >> >> PetscErrorCode ierr; >> DM dm; >> DMTS_DA *dmdats = (DMTS_DA*)ctx; >> DMDALocalInfo info; >> Vec Xloc,Xdotloc; >> void *x,*f,*xdot; >> >> PetscFunctionBegin; >> PetscValidHeaderSpecific(ts,TS_CLASSID,1); >> PetscValidHeaderSpecific(X,VEC_CLASSID,2); >> PetscValidHeaderSpecific(F,VEC_CLASSID,3); >> if (!dmdats->ifunctionlocal) SETERRQ(PetscObjectComm((Petsc >> Object)ts),PETSC_ERR_PLIB,"Corrupt context"); >> ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); >> ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); >> ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ( >> ierr); >> ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); >> >> ierr = DMGetLocalVector(dm,&Xloc);CHKERRQ(ierr); >> ierr = DMGlobalToLocalBegin(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); >> ierr = DMGlobalToLocalEnd(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); >> ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr); >> ierr = DMDAVecGetArray(dm,Xloc,&x);CHKERRQ(ierr); >> ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); >> >> >> Don't forget to include these calls (in this order) after you are done >> with the Xdotloc vector >> >> ierr = DMDAVecRestoreArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); >> ierr = DMRestoreLocalVector(dm,&Xdotloc);CHKERRQ(ierr); >> >> Failure to do so will result in a memory leak. >> >> >> >> Thanks, >> >> Sepideh >> ------------------------------ >> *From:* Matthew Knepley >> *Sent:* Tuesday, April 17, 2018 5:59:12 PM >> >> *To:* Sepideh Kavousi >> *Cc:* petsc-users at mcs.anl.gov >> *Subject:* Re: [petsc-users] error running parallel on cluster >> >> On Tue, Apr 17, 2018 at 3:07 PM, Sepideh Kavousi wrote: >> >> The reason I can not use your method is that the input arguments of >> SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your >> method which was: >> >> ierr *=* DMGetLocalVector(dm,*&*Xdotloc);CHKERRQ(ierr); ierr *=* DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr *=* DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); >> ierr *=* DMDAVecGetArray(dm,Xdotloc,*&*xdot);CHKERRQ(ierr); >> >> >> You misunderstand my suggestion. I mean stick this code in right here in >> PETSc >> >> https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a >> 0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&file >> viewer=file-view-default#dmdats.c-68 >> >> Then the X_t array you get in your local function will be ghosted. >> >> Matt >> >> >> I need to have the vector of Xdot, not the array. So I think I should use >> SetIFunction instead of SetIFunctionLocal. >> >> >> Sepideh >> ------------------------------ >> *From:* Matthew Knepley >> *Sent:* Tuesday, April 17, 2018 1:22:53 PM >> *To:* Sepideh Kavousi >> *Cc:* petsc-users at mcs.anl.gov >> *Subject:* Re: [petsc-users] error running parallel on cluster >> >> On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi wrote: >> >> Mathew, >> I previously use DMDATSSetIFunctionLocal(user.d >> a,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. >> If I want to use your solution I can not use it because in the FormFunction >> definition I must use arrays, not vectors.So to solve this issue I followed >> two methods where none were able to solve it. >> 1- in first method I decided to use TSSetIFunction instead of >> DMDATSSetIFunctionLocal >> >> for this means first in the main function, I use TSSetDM and my form >> function variables were as: >> PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, >> struct VAR_STRUCT *user) { >> . >> . >> . >> . >> ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); >> ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); >> >> ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); >> ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local);CHK >> ERRQ(ierr); >> ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKER >> RQ(ierr); >> . >> . >> . >> >> } >> But still, it does not consider vectors y,ydot,f related to dmda (problem >> executing DMDAVecGetArray) >> >> >> We cannot help you if you do not show full error messages. >> >> Why not fix the code with SetIFunctionLocal(), as I said in my last >> email. I will fix PETSc proper in branch at the end of the week. I >> have a proposal due tomorrow, so I cannot do it right now. >> >> Thanks, >> >> Matt >> >> >> 2- in second method I decided to use DMTSSetIFunction >> but still, FormFunction is in form of TSIFunction where we do not define >> dm object and I think it does not understand dm and da are connected, >> although I have used TSSetDM in the main function. >> >> Can you please help me what should I do? >> Regards, >> Sepideh >> >> >> >> >> ------------------------------ >> *From:* Matthew Knepley >> *Sent:* Monday, April 16, 2018 9:34:01 PM >> >> *To:* Sepideh Kavousi >> *Cc:* petsc-users at mcs.anl.gov >> *Subject:* Re: [petsc-users] error running parallel on cluster >> >> On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi >> wrote: >> >> I run with valgrind with the following command: >> >> /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q >> --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off >> -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler >> -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr >> -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason >> -ksp_monitor >> >> >> Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and >> only chooses to ghost the input vector X and not X_t >> >> https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a >> 0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&file >> viewer=file-view-default#dmdats.c-68 >> >> Unfortunately, this was done by Peter in 2012 and he is gone, so we can't >> blame him. >> It is true that very few people use spatial derivatives of X_t, but it >> does make sense. >> Right there, you want to add >> >> ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); >> >> ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); >> >> >> and it should work. >> >> Matt >> >> >> the valgringoutput is as following: >> >> ==39396== Invalid read of size 8 >> ==39396== at 0x402679: FormFunction (one.c:212) >> ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) >> ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) >> ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) >> ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) >> ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) >> ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) >> ==39396== by 0x60246C0: SNESSolve (snes.c:4312) >> ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) >> ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) >> ==39396== by 0x6153380: TSStep (ts.c:3548) >> ==39396== by 0x615559D: TSSolve (ts.c:3731) >> ==39396== by 0x403C95: main (one.c:303) >> ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 >> alloc'd >> ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) >> ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) >> ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) >> ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) >> ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) >> ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) >> ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) >> ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) >> ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) >> ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) >> ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) >> ==39396== by 0x60246C0: SNESSolve (snes.c:4312) >> ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) >> ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) >> ==39396== by 0x6153380: TSStep (ts.c:3548) >> ==39396== by 0x615559D: TSSolve (ts.c:3731) >> ==39396== by 0x403C95: main (one.c:303) >> ==39396== >> ==39396== Invalid read of size 8 >> ==39396== at 0x402689: FormFunction (one.c:212) >> ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) >> ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) >> ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) >> ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) >> ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) >> ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) >> ==39396== by 0x60246C0: SNESSolve (snes.c:4312) >> ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) >> ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) >> ==39396== by 0x6153380: TSStep (ts.c:3548) >> ==39396== by 0x615559D: TSSolve (ts.c:3731) >> ==39396== by 0x403C95: main (one.c:303) >> ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd >> ==39396== >> >> Thanks, >> >> Sepideh >> ------------------------------ >> *From:* Matthew Knepley >> *Sent:* Monday, April 16, 2018 8:44:43 PM >> *To:* Sepideh Kavousi >> *Cc:* petsc-users at mcs.anl.gov >> *Subject:* Re: [petsc-users] error running parallel on cluster >> >> On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi wrote: >> >> this is not used forghost points . my code is : >> >> >> 1) You are using ghost points, since you use i-1 as an index >> >> 2) I was wrong. X_t should also be ghosted. >> >> Run under valgrind. >> >> Thanks, >> >> Matt >> >> >> for (j=info2->ys;jys+info2->ym;j++){ >> for (i=info2->xs;ixs+info2->xm;i++){ >> if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; >> aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} >> else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U >> ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} >> else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; >> aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} >> else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U >> ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} >> else { >> //derivatives of U and p >> Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); >> Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); >> Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); >> Pxy=((aY[j+1][i+1].p+aY[j-1][ >> i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); >> Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); >> Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); >> Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); >> >> >> if (((Px*Px)+(Py*Py))>0.0 ) { // >> theta=PetscAtanReal(Py/Px); >> thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); >> thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); >> } >> else { >> theta=PETSC_PI*0.5; >> thetax=0.0; >> thetay=0.0; >> } >> e=(1.0+user->epsilon*cos(4.0*theta));; >> ep=-(4.0*user->epsilon*sin(4.0*theta)); >> epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); >> >> >> >> Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); >> Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); >> Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); >> Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); >> >> U1=user->D*user->tau_0/(user->W*user->W); >> U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : >> 0.0 ; >> //U2=0.0; >> P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; >> >> aF[j][i].U=(U1*(-0.5*(Ux*Px+U >> y*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][ >> i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k) >> *aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)* >> aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; >> >> aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax >> *Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i] >> .p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i]. >> p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1 >> -(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0 >> +4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; >> >> } >> } >> >> Sepideh >> ------------------------------ >> *From:* Matthew Knepley >> *Sent:* Monday, April 16, 2018 8:36:37 PM >> *To:* Sepideh Kavousi >> *Cc:* petsc-users at mcs.anl.gov >> *Subject:* Re: [petsc-users] error running parallel on cluster >> >> On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi wrote: >> >> Hello, >> I am solving a PDE where I have the spacial derivtive of the time >> derivative of a variable. In the DMDATSSetIFunctionLocal function I >> defined d(dY/dt)/dx as: >> (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) >> >> >> I do not think that you get ghosted Ydot in that function. >> >> Matt >> >> >> on my workstation, it is working but on the cluster it gives an error. I >> can not run with debugger on the cluster but I have checked and the error >> is related to this part and I do not have any problem on cluster when >> running in series. Can you please help me about it. >> >> the error is: >> 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >> ocumentation/faq.html#valgrind >> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS >> X to find memory corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 >> /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c >> [0]PETSC ERROR: [0] TS user implicit function line 829 >> /ddnB/work/skavou1/petsc/src/ts/interface/ts.c >> [0]PETSC ERROR: [0] TSComputeIFunction line 815 >> /ddnB/work/skavou1/petsc/src/ts/interface/ts.c >> [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 >> /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c >> [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 >> /ddnB/work/skavou1/petsc/src/ts/interface/ts.c >> [0]PETSC ERROR: [0] SNES user function line 2202 >> /ddnB/work/skavou1/petsc/src/snes/interface/snes.c >> [0]PETSC ERROR: [0] SNESComputeFunction line 2187 >> /ddnB/work/skavou1/petsc/src/snes/interface/snes.c >> [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 >> /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c >> [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/s >> nes/interface/snes.c >> [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 >> /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c >> [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/t >> s/impls/implicit/theta/theta.c >> [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/t >> s/interface/ts.c >> [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/t >> s/interface/ts.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Signal received >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown >> [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 >> 20:11:44 2018 >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack >> [0]PETSC ERROR: #1 User provided function() line 0 in unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> ==29057== >> ==29057== HEAP SUMMARY: >> ==29057== in use at exit: 560,151 bytes in 383 blocks >> ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes >> allocated >> ==29057== >> ==29057== LEAK SUMMARY: >> ==29057== definitely lost: 0 bytes in 0 blocks >> ==29057== indirectly lost: 0 bytes in 0 blocks >> ==29057== possibly lost: 0 bytes in 0 blocks >> ==29057== still reachable: 560,151 bytes in 383 blocks >> ==29057== suppressed: 0 bytes in 0 blocks >> ==29057== Rerun with --leak-check=full to see details of leaked memory >> ==29057== >> ==29057== For counts of detected and suppressed errors, rerun with: -v >> ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) >> >> >> >> >> Thanks, >> Sepideh >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Wed Apr 18 23:37:17 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Thu, 19 Apr 2018 04:37:17 +0000 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: , Message-ID: It is really a pleasure for me. Thanks, Sepidwh Get Outlook for Android ________________________________ From: Dave May Sent: Wednesday, April 18, 2018 11:14:47 PM To: Matthew Knepley Cc: Sepideh Kavousi; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On 18 April 2018 at 23:52, Matthew Knepley > wrote: On Wed, Apr 18, 2018 at 5:52 PM, Sepideh Kavousi > wrote: Mathew and Dave, Thank you so much it is working perfectly now. Excellent. If you want your name to appear on the next PETSc release as a contributor, you can make a PR with this change :) Here is a URL describing the PR's protocol for PETSc contribs: https://bitbucket.org/petsc/petsc/wiki/pull-request-instructions-git Thanks, Matt Sepideh ________________________________ From: Dave May > Sent: Wednesday, April 18, 2018 3:13:33 PM To: Sepideh Kavousi Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On 18 April 2018 at 21:06, Sepideh Kavousi > wrote: Mathew I added the lines and I still have the same issue. It may be a silly question but should I configure and install petsc again using this new lines added? or changing the line is enough? the highlighted lines are the lines I modified. PetscErrorCode ierr; DM dm; DMTS_DA *dmdats = (DMTS_DA*)ctx; DMDALocalInfo info; Vec Xloc,Xdotloc; void *x,*f,*xdot; PetscFunctionBegin; PetscValidHeaderSpecific(ts,TS_CLASSID,1); PetscValidHeaderSpecific(X,VEC_CLASSID,2); PetscValidHeaderSpecific(F,VEC_CLASSID,3); if (!dmdats->ifunctionlocal) SETERRQ(PetscObjectComm((PetscObject)ts),PETSC_ERR_PLIB,"Corrupt context"); ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGetLocalVector(dm,&Xloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xloc,&x);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); Don't forget to include these calls (in this order) after you are done with the Xdotloc vector ierr = DMDAVecRestoreArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); ierr = DMRestoreLocalVector(dm,&Xdotloc);CHKERRQ(ierr); Failure to do so will result in a memory leak. Thanks, Sepideh ________________________________ From: Matthew Knepley > Sent: Tuesday, April 17, 2018 5:59:12 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Tue, Apr 17, 2018 at 3:07 PM, Sepideh Kavousi > wrote: The reason I can not use your method is that the input arguments of SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your method which was: ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); You misunderstand my suggestion. I mean stick this code in right here in PETSc https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Then the X_t array you get in your local function will be ghosted. Matt I need to have the vector of Xdot, not the array. So I think I should use SetIFunction instead of SetIFunctionLocal. Sepideh ________________________________ From: Matthew Knepley > Sent: Tuesday, April 17, 2018 1:22:53 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi > wrote: Mathew, I previously use DMDATSSetIFunctionLocal(user.da,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. If I want to use your solution I can not use it because in the FormFunction definition I must use arrays, not vectors.So to solve this issue I followed two methods where none were able to solve it. 1- in first method I decided to use TSSetIFunction instead of DMDATSSetIFunctionLocal for this means first in the main function, I use TSSetDM and my form function variables were as: PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, struct VAR_STRUCT *user) { . . . . ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); . . . } But still, it does not consider vectors y,ydot,f related to dmda (problem executing DMDAVecGetArray) We cannot help you if you do not show full error messages. Why not fix the code with SetIFunctionLocal(), as I said in my last email. I will fix PETSc proper in branch at the end of the week. I have a proposal due tomorrow, so I cannot do it right now. Thanks, Matt 2- in second method I decided to use DMTSSetIFunction but still, FormFunction is in form of TSIFunction where we do not define dm object and I think it does not understand dm and da are connected, although I have used TSSetDM in the main function. Can you please help me what should I do? Regards, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 9:34:01 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi > wrote: I run with valgrind with the following command: /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason -ksp_monitor Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only chooses to ghost the input vector X and not X_t https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Unfortunately, this was done by Peter in 2012 and he is gone, so we can't blame him. It is true that very few people use spatial derivatives of X_t, but it does make sense. Right there, you want to add ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); and it should work. Matt the valgringoutput is as following: ==39396== Invalid read of size 8 ==39396== at 0x402679: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 alloc'd ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== ==39396== Invalid read of size 8 ==39396== at 0x402689: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd ==39396== Thanks, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:44:43 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi > wrote: this is not used forghost points . my code is : 1) You are using ghost points, since you use i-1 as an index 2) I was wrong. X_t should also be ghosted. Run under valgrind. Thanks, Matt for (j=info2->ys;jys+info2->ym;j++){ for (i=info2->xs;ixs+info2->xm;i++){ if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} else { //derivatives of U and p Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); Pxy=((aY[j+1][i+1].p+aY[j-1][i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); if (((Px*Px)+(Py*Py))>0.0 ) { // theta=PetscAtanReal(Py/Px); thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); } else { theta=PETSC_PI*0.5; thetax=0.0; thetay=0.0; } e=(1.0+user->epsilon*cos(4.0*theta));; ep=-(4.0*user->epsilon*sin(4.0*theta)); epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); U1=user->D*user->tau_0/(user->W*user->W); U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : 0.0 ; //U2=0.0; P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; aF[j][i].U=(U1*(-0.5*(Ux*Px+Uy*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k)*aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)*aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax*Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i].p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; } } Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:36:37 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi > wrote: Hello, I am solving a PDE where I have the spacial derivtive of the time derivative of a variable. In the DMDATSSetIFunctionLocal function I defined d(dY/dt)/dx as: (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) I do not think that you get ghosted Ydot in that function. Matt on my workstation, it is working but on the cluster it gives an error. I can not run with debugger on the cluster but I have checked and the error is related to this part and I do not have any problem on cluster when running in series. Can you please help me about it. the error is: 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c [0]PETSC ERROR: [0] TS user implicit function line 829 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSComputeIFunction line 815 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNES user function line 2202 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESComputeFunction line 2187 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 20:11:44 2018 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ==29057== ==29057== HEAP SUMMARY: ==29057== in use at exit: 560,151 bytes in 383 blocks ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes allocated ==29057== ==29057== LEAK SUMMARY: ==29057== definitely lost: 0 bytes in 0 blocks ==29057== indirectly lost: 0 bytes in 0 blocks ==29057== possibly lost: 0 bytes in 0 blocks ==29057== still reachable: 560,151 bytes in 383 blocks ==29057== suppressed: 0 bytes in 0 blocks ==29057== Rerun with --leak-check=full to see details of leaked memory ==29057== ==29057== For counts of detected and suppressed errors, rerun with: -v ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) Thanks, Sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Thu Apr 19 07:43:47 2018 From: epscodes at gmail.com (Xiangdong) Date: Thu, 19 Apr 2018 08:43:47 -0400 Subject: [petsc-users] question on VecPointwiseMax In-Reply-To: References: Message-ID: Hi Barry, I called the SNESGetSolution() through SNESMonitorSet(). I understand that it may not be effective, but I was able to change the values of iteration solution in old releases (two years ago). Was the solution vector set to read status in recent version? About the -snes_type vinewtonrsls, we discussed this a few years ago. It does not work well for my problem, so I want to manually perform some post processing after each iteration. For example, setting a bound for the solution or set the maximum changes for each component of the solution in each iteration. https://lists.mcs.anl.gov/pipermail/petsc-users/2014-May/021520.html Best, Xiangdong On Tue, Apr 17, 2018 at 5:24 PM, Smith, Barry F. wrote: > > Where are you call SNESGetSolution() from? Inside a SNES monitor > routine, a SNES convergence test routine, a SNESLineSearchPostCheck() > routine? > > The SNES linear solvers are solving the system, you really shouldn't > be changing the values of the solution any time while they are doing their > job solving the system. Of course you can change the values after the > solver is done. > > If you are trying to "solve" a linear system with upper and/or lower > bounds on the variables you should solve it as a variational inequality and > use SNESVISetVariableBounds(), you should not try to manually restrict the > values (that won't work). And use -snes_type vinewtonrsls > > Barry > > > > Barry > > > > On Apr 17, 2018, at 12:04 PM, Xiangdong wrote: > > > > Hello everyone, > > > > Here are a little more information. > > > > Here is the function calls I have: > > > > SNESGetSolution(snes,&x); > > VecPointwiseMax(x,x,xl); Fail > > VecPointwiseMax(x,y,xl); Fail > > > > The error messages are"Object is in wrong state, Vec is locked read > only, argument #1." > > > > It seems that the vector x returned from SNESGetSolution is locked read > only, Is it true? > > > > If I want do some post processing by modifying the x after each > nonlinear iteration, how can I get the solution and modify it? > > > > Thank you. > > > > Best, > > Xiangdong > > > > > > > > On Tue, Apr 17, 2018 at 11:33 AM, Xiangdong wrote: > > Hello everyone, > > > > When I call VecPointwiseMax(x,x,y) , I got an error message "object is > in wrong state. Vec is locked read only, argument #1." > > > > However, In the online manual of output parameters of VecPointwiseMax, > it says: > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/ > Vec/VecPointwiseMax.html > > Output Parameter > > > > w -the result > > > > Notes: any subset of the x, y, and w may be the same vector. For complex > numbers compares only the real part > > > > However, in the implementation of VecPointwiseMax_Seq, > > http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/impls > /seq/bvec2.c.html#VecPointwiseMax_Seq, > > > > it seems that xin is read locked, and win and xin cannot be the same > vectors. > > PetscErrorCode VecPointwiseMax_Seq(Vec win,Vec xin,Vec yin) > > 17: > > { > > > > 19: PetscInt > > n = win->map->n,i; > > > > 20: PetscScalar *ww,*xx,*yy; /* cannot make xx or yy const since > might be ww */ > > > > > > > > 23: VecGetArrayRead(xin,(const PetscScalar > > **)&xx); > > > > 24: VecGetArrayRead(yin,(const PetscScalar > > **)&yy); > > > > 25: VecGetArray > > (win,&ww); > > > > > > 27: for (i=0; i > PetscRealPart(yy[i])); > > > > > > 29: VecRestoreArrayRead(xin,(const PetscScalar > > **)&xx); > > > > 30: VecRestoreArrayRead(yin,(const PetscScalar > > **)&yy); > > > > 31: VecRestoreArray > > (win,&ww); > > > > 32: PetscLogFlops > > (n); > > > > 33: return > > (0); > > > > 34: } > > Can w and x really be same vector in the VecPointwiseMax? Thanks. > > > > Best, > > Xiangdong > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Apr 19 08:11:51 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 19 Apr 2018 13:11:51 +0000 Subject: [petsc-users] question on VecPointwiseMax In-Reply-To: References: Message-ID: <354F71CE-006C-496A-B54A-E03E6B21FBA9@mcs.anl.gov> Ok, yes we started locking the solution so that people won't "accidentally" change values in the solution while monitoring The "proper" way to change it is to use SNESLineSearchSetPostCheck() and change the value of the third vector (the current solution) in your callback function. Could you please try that? Barry > On Apr 19, 2018, at 7:43 AM, Xiangdong wrote: > > Hi Barry, > > I called the SNESGetSolution() through SNESMonitorSet(). I understand that it may not be effective, but I was able to change the values of iteration solution in old releases (two years ago). Was the solution vector set to read status in recent version? > > About the -snes_type vinewtonrsls, we discussed this a few years ago. It does not work well for my problem, so I want to manually perform some post processing after each iteration. For example, setting a bound for the solution or set the maximum changes for each component of the solution in each iteration. > https://lists.mcs.anl.gov/pipermail/petsc-users/2014-May/021520.html > > > Best, > Xiangdong > > On Tue, Apr 17, 2018 at 5:24 PM, Smith, Barry F. wrote: > > Where are you call SNESGetSolution() from? Inside a SNES monitor routine, a SNES convergence test routine, a SNESLineSearchPostCheck() routine? > > The SNES linear solvers are solving the system, you really shouldn't be changing the values of the solution any time while they are doing their job solving the system. Of course you can change the values after the solver is done. > > If you are trying to "solve" a linear system with upper and/or lower bounds on the variables you should solve it as a variational inequality and use SNESVISetVariableBounds(), you should not try to manually restrict the values (that won't work). And use -snes_type vinewtonrsls > > Barry > > > > Barry > > > > On Apr 17, 2018, at 12:04 PM, Xiangdong wrote: > > > > Hello everyone, > > > > Here are a little more information. > > > > Here is the function calls I have: > > > > SNESGetSolution(snes,&x); > > VecPointwiseMax(x,x,xl); Fail > > VecPointwiseMax(x,y,xl); Fail > > > > The error messages are"Object is in wrong state, Vec is locked read only, argument #1." > > > > It seems that the vector x returned from SNESGetSolution is locked read only, Is it true? > > > > If I want do some post processing by modifying the x after each nonlinear iteration, how can I get the solution and modify it? > > > > Thank you. > > > > Best, > > Xiangdong > > > > > > > > On Tue, Apr 17, 2018 at 11:33 AM, Xiangdong wrote: > > Hello everyone, > > > > When I call VecPointwiseMax(x,x,y) , I got an error message "object is in wrong state. Vec is locked read only, argument #1." > > > > However, In the online manual of output parameters of VecPointwiseMax, it says: > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPointwiseMax.html > > Output Parameter > > > > w -the result > > > > Notes: any subset of the x, y, and w may be the same vector. For complex numbers compares only the real part > > > > However, in the implementation of VecPointwiseMax_Seq, > > http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/impls/seq/bvec2.c.html#VecPointwiseMax_Seq, > > > > it seems that xin is read locked, and win and xin cannot be the same vectors. > > PetscErrorCode VecPointwiseMax_Seq(Vec win,Vec xin,Vec yin) > > 17: > > { > > > > 19: PetscInt > > n = win->map->n,i; > > > > 20: PetscScalar *ww,*xx,*yy; /* cannot make xx or yy const since might be ww */ > > > > > > > > 23: VecGetArrayRead(xin,(const PetscScalar > > **)&xx); > > > > 24: VecGetArrayRead(yin,(const PetscScalar > > **)&yy); > > > > 25: VecGetArray > > (win,&ww); > > > > > > 27: for (i=0; i > PetscRealPart(yy[i])); > > > > > > 29: VecRestoreArrayRead(xin,(const PetscScalar > > **)&xx); > > > > 30: VecRestoreArrayRead(yin,(const PetscScalar > > **)&yy); > > > > 31: VecRestoreArray > > (win,&ww); > > > > 32: PetscLogFlops > > (n); > > > > 33: return > > (0); > > > > 34: } > > Can w and x really be same vector in the VecPointwiseMax? Thanks. > > > > Best, > > Xiangdong > > > > From skavou1 at lsu.edu Thu Apr 19 13:43:55 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Thu, 19 Apr 2018 18:43:55 +0000 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: , Message-ID: I submitted the pull request this morning. Regards, Sepideh ________________________________ From: Dave May Sent: Wednesday, April 18, 2018 11:14:47 PM To: Matthew Knepley Cc: Sepideh Kavousi; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On 18 April 2018 at 23:52, Matthew Knepley > wrote: On Wed, Apr 18, 2018 at 5:52 PM, Sepideh Kavousi > wrote: Mathew and Dave, Thank you so much it is working perfectly now. Excellent. If you want your name to appear on the next PETSc release as a contributor, you can make a PR with this change :) Here is a URL describing the PR's protocol for PETSc contribs: https://bitbucket.org/petsc/petsc/wiki/pull-request-instructions-git Thanks, Matt Sepideh ________________________________ From: Dave May > Sent: Wednesday, April 18, 2018 3:13:33 PM To: Sepideh Kavousi Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On 18 April 2018 at 21:06, Sepideh Kavousi > wrote: Mathew I added the lines and I still have the same issue. It may be a silly question but should I configure and install petsc again using this new lines added? or changing the line is enough? the highlighted lines are the lines I modified. PetscErrorCode ierr; DM dm; DMTS_DA *dmdats = (DMTS_DA*)ctx; DMDALocalInfo info; Vec Xloc,Xdotloc; void *x,*f,*xdot; PetscFunctionBegin; PetscValidHeaderSpecific(ts,TS_CLASSID,1); PetscValidHeaderSpecific(X,VEC_CLASSID,2); PetscValidHeaderSpecific(F,VEC_CLASSID,3); if (!dmdats->ifunctionlocal) SETERRQ(PetscObjectComm((PetscObject)ts),PETSC_ERR_PLIB,"Corrupt context"); ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGetLocalVector(dm,&Xloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xloc,&x);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); Don't forget to include these calls (in this order) after you are done with the Xdotloc vector ierr = DMDAVecRestoreArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); ierr = DMRestoreLocalVector(dm,&Xdotloc);CHKERRQ(ierr); Failure to do so will result in a memory leak. Thanks, Sepideh ________________________________ From: Matthew Knepley > Sent: Tuesday, April 17, 2018 5:59:12 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Tue, Apr 17, 2018 at 3:07 PM, Sepideh Kavousi > wrote: The reason I can not use your method is that the input arguments of SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your method which was: ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); You misunderstand my suggestion. I mean stick this code in right here in PETSc https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Then the X_t array you get in your local function will be ghosted. Matt I need to have the vector of Xdot, not the array. So I think I should use SetIFunction instead of SetIFunctionLocal. Sepideh ________________________________ From: Matthew Knepley > Sent: Tuesday, April 17, 2018 1:22:53 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi > wrote: Mathew, I previously use DMDATSSetIFunctionLocal(user.da,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. If I want to use your solution I can not use it because in the FormFunction definition I must use arrays, not vectors.So to solve this issue I followed two methods where none were able to solve it. 1- in first method I decided to use TSSetIFunction instead of DMDATSSetIFunctionLocal for this means first in the main function, I use TSSetDM and my form function variables were as: PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, struct VAR_STRUCT *user) { . . . . ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); . . . } But still, it does not consider vectors y,ydot,f related to dmda (problem executing DMDAVecGetArray) We cannot help you if you do not show full error messages. Why not fix the code with SetIFunctionLocal(), as I said in my last email. I will fix PETSc proper in branch at the end of the week. I have a proposal due tomorrow, so I cannot do it right now. Thanks, Matt 2- in second method I decided to use DMTSSetIFunction but still, FormFunction is in form of TSIFunction where we do not define dm object and I think it does not understand dm and da are connected, although I have used TSSetDM in the main function. Can you please help me what should I do? Regards, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 9:34:01 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi > wrote: I run with valgrind with the following command: /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason -ksp_monitor Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only chooses to ghost the input vector X and not X_t https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Unfortunately, this was done by Peter in 2012 and he is gone, so we can't blame him. It is true that very few people use spatial derivatives of X_t, but it does make sense. Right there, you want to add ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); and it should work. Matt the valgringoutput is as following: ==39396== Invalid read of size 8 ==39396== at 0x402679: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 alloc'd ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== ==39396== Invalid read of size 8 ==39396== at 0x402689: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd ==39396== Thanks, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:44:43 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi > wrote: this is not used forghost points . my code is : 1) You are using ghost points, since you use i-1 as an index 2) I was wrong. X_t should also be ghosted. Run under valgrind. Thanks, Matt for (j=info2->ys;jys+info2->ym;j++){ for (i=info2->xs;ixs+info2->xm;i++){ if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} else { //derivatives of U and p Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); Pxy=((aY[j+1][i+1].p+aY[j-1][i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); if (((Px*Px)+(Py*Py))>0.0 ) { // theta=PetscAtanReal(Py/Px); thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); } else { theta=PETSC_PI*0.5; thetax=0.0; thetay=0.0; } e=(1.0+user->epsilon*cos(4.0*theta));; ep=-(4.0*user->epsilon*sin(4.0*theta)); epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); U1=user->D*user->tau_0/(user->W*user->W); U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : 0.0 ; //U2=0.0; P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; aF[j][i].U=(U1*(-0.5*(Ux*Px+Uy*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k)*aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)*aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax*Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i].p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; } } Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:36:37 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi > wrote: Hello, I am solving a PDE where I have the spacial derivtive of the time derivative of a variable. In the DMDATSSetIFunctionLocal function I defined d(dY/dt)/dx as: (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) I do not think that you get ghosted Ydot in that function. Matt on my workstation, it is working but on the cluster it gives an error. I can not run with debugger on the cluster but I have checked and the error is related to this part and I do not have any problem on cluster when running in series. Can you please help me about it. the error is: 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c [0]PETSC ERROR: [0] TS user implicit function line 829 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSComputeIFunction line 815 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNES user function line 2202 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESComputeFunction line 2187 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 20:11:44 2018 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ==29057== ==29057== HEAP SUMMARY: ==29057== in use at exit: 560,151 bytes in 383 blocks ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes allocated ==29057== ==29057== LEAK SUMMARY: ==29057== definitely lost: 0 bytes in 0 blocks ==29057== indirectly lost: 0 bytes in 0 blocks ==29057== possibly lost: 0 bytes in 0 blocks ==29057== still reachable: 560,151 bytes in 383 blocks ==29057== suppressed: 0 bytes in 0 blocks ==29057== Rerun with --leak-check=full to see details of leaked memory ==29057== ==29057== For counts of detected and suppressed errors, rerun with: -v ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) Thanks, Sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Apr 19 13:48:57 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 19 Apr 2018 14:48:57 -0400 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: Message-ID: Great! I have approved it. Now it goes to master. Matt On Thu, Apr 19, 2018 at 2:43 PM, Sepideh Kavousi wrote: > I submitted the pull request this morning. > > Regards, > > Sepideh > ------------------------------ > *From:* Dave May > *Sent:* Wednesday, April 18, 2018 11:14:47 PM > *To:* Matthew Knepley > *Cc:* Sepideh Kavousi; petsc-users at mcs.anl.gov > > *Subject:* Re: [petsc-users] error running parallel on cluster > > > > On 18 April 2018 at 23:52, Matthew Knepley wrote: > > On Wed, Apr 18, 2018 at 5:52 PM, Sepideh Kavousi wrote: > > Mathew and Dave, > > Thank you so much it is working perfectly now. > > > Excellent. > > If you want your name to appear on the next PETSc release as a > contributor, you > can make a PR with this change :) > > > Here is a URL describing the PR's protocol for PETSc contribs: > > https://bitbucket.org/petsc/petsc/wiki/pull-request-instructions-git > > > > > Thanks, > > Matt > > > Sepideh > ------------------------------ > *From:* Dave May > *Sent:* Wednesday, April 18, 2018 3:13:33 PM > *To:* Sepideh Kavousi > *Cc:* Matthew Knepley; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > > > On 18 April 2018 at 21:06, Sepideh Kavousi wrote: > > Mathew > > I added the lines and I still have the same issue. It may be a silly > question but should I configure and install petsc again using this new > lines added? or changing the line is enough? the highlighted lines are the > lines I modified. > > > PetscErrorCode ierr; > DM dm; > DMTS_DA *dmdats = (DMTS_DA*)ctx; > DMDALocalInfo info; > Vec Xloc,Xdotloc; > void *x,*f,*xdot; > > PetscFunctionBegin; > PetscValidHeaderSpecific(ts,TS_CLASSID,1); > PetscValidHeaderSpecific(X,VEC_CLASSID,2); > PetscValidHeaderSpecific(F,VEC_CLASSID,3); > if (!dmdats->ifunctionlocal) SETERRQ(PetscObjectComm((Petsc > Object)ts),PETSC_ERR_PLIB,"Corrupt context"); > ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); > ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ( > ierr); > ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > ierr = DMGetLocalVector(dm,&Xloc);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); > ierr = DMGlobalToLocalEnd(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); > ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr); > ierr = DMDAVecGetArray(dm,Xloc,&x);CHKERRQ(ierr); > ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); > > > Don't forget to include these calls (in this order) after you are done > with the Xdotloc vector > > ierr = DMDAVecRestoreArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); > ierr = DMRestoreLocalVector(dm,&Xdotloc);CHKERRQ(ierr); > > Failure to do so will result in a memory leak. > > > > Thanks, > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, April 17, 2018 5:59:12 PM > > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Tue, Apr 17, 2018 at 3:07 PM, Sepideh Kavousi wrote: > > The reason I can not use your method is that the input arguments of > SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your > method which was: > > ierr *=* DMGetLocalVector(dm,*&*Xdotloc);CHKERRQ(ierr); ierr *=* DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr *=* DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > ierr *=* DMDAVecGetArray(dm,Xdotloc,*&*xdot);CHKERRQ(ierr); > > > You misunderstand my suggestion. I mean stick this code in right here in > PETSc > > https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a > 0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&file > viewer=file-view-default#dmdats.c-68 > > Then the X_t array you get in your local function will be ghosted. > > Matt > > > I need to have the vector of Xdot, not the array. So I think I should use > SetIFunction instead of SetIFunctionLocal. > > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Tuesday, April 17, 2018 1:22:53 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi wrote: > > Mathew, > I previously use DMDATSSetIFunctionLocal(user.d > a,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. > If I want to use your solution I can not use it because in the FormFunction > definition I must use arrays, not vectors.So to solve this issue I followed > two methods where none were able to solve it. > 1- in first method I decided to use TSSetIFunction instead of > DMDATSSetIFunctionLocal > > for this means first in the main function, I use TSSetDM and my form > function variables were as: > PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, > struct VAR_STRUCT *user) { > . > . > . > . > ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); > ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); > > ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local);CHK > ERRQ(ierr); > ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKER > RQ(ierr); > . > . > . > > } > But still, it does not consider vectors y,ydot,f related to dmda (problem > executing DMDAVecGetArray) > > > We cannot help you if you do not show full error messages. > > Why not fix the code with SetIFunctionLocal(), as I said in my last email. > I will fix PETSc proper in branch at the end of the week. I > have a proposal due tomorrow, so I cannot do it right now. > > Thanks, > > Matt > > > 2- in second method I decided to use DMTSSetIFunction > but still, FormFunction is in form of TSIFunction where we do not define > dm object and I think it does not understand dm and da are connected, > although I have used TSSetDM in the main function. > > Can you please help me what should I do? > Regards, > Sepideh > > > > > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 9:34:01 PM > > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi wrote: > > I run with valgrind with the following command: > > /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q > --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off > -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler > -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr > -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason > -ksp_monitor > > > Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only > chooses to ghost the input vector X and not X_t > > https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a > 0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&file > viewer=file-view-default#dmdats.c-68 > > Unfortunately, this was done by Peter in 2012 and he is gone, so we can't > blame him. > It is true that very few people use spatial derivatives of X_t, but it > does make sense. > Right there, you want to add > > ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); > > ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); > > > and it should work. > > Matt > > > the valgringoutput is as following: > > ==39396== Invalid read of size 8 > ==39396== at 0x402679: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 > alloc'd > ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) > ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) > ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) > ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) > ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) > ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== > ==39396== Invalid read of size 8 > ==39396== at 0x402689: FormFunction (one.c:212) > ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) > ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) > ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) > ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) > ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) > ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) > ==39396== by 0x60246C0: SNESSolve (snes.c:4312) > ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) > ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) > ==39396== by 0x6153380: TSStep (ts.c:3548) > ==39396== by 0x615559D: TSSolve (ts.c:3731) > ==39396== by 0x403C95: main (one.c:303) > ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd > ==39396== > > Thanks, > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:44:43 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi wrote: > > this is not used forghost points . my code is : > > > 1) You are using ghost points, since you use i-1 as an index > > 2) I was wrong. X_t should also be ghosted. > > Run under valgrind. > > Thanks, > > Matt > > > for (j=info2->ys;jys+info2->ym;j++){ > for (i=info2->xs;ixs+info2->xm;i++){ > if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; > aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} > else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} > else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; > aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} > else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U > ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} > else { > //derivatives of U and p > Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); > Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); > Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); > Pxy=((aY[j+1][i+1].p+aY[j-1][ > i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); > Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); > Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); > Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); > > > if (((Px*Px)+(Py*Py))>0.0 ) { // > theta=PetscAtanReal(Py/Px); > thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); > thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); > } > else { > theta=PETSC_PI*0.5; > thetax=0.0; > thetay=0.0; > } > e=(1.0+user->epsilon*cos(4.0*theta));; > ep=-(4.0*user->epsilon*sin(4.0*theta)); > epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); > > > > Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); > Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); > Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); > Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); > > U1=user->D*user->tau_0/(user->W*user->W); > U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : > 0.0 ; > //U2=0.0; > P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; > > aF[j][i].U=(U1*(-0.5*(Ux*Px+U > y*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][ > i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k) > *aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)* > aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; > > aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax > *Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i] > .p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i]. > p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1 > -(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0 > +4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; > > } > } > > Sepideh > ------------------------------ > *From:* Matthew Knepley > *Sent:* Monday, April 16, 2018 8:36:37 PM > *To:* Sepideh Kavousi > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] error running parallel on cluster > > On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi wrote: > > Hello, > I am solving a PDE where I have the spacial derivtive of the time > derivative of a variable. In the DMDATSSetIFunctionLocal function I > defined d(dY/dt)/dx as: > (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) > > > I do not think that you get ghosted Ydot in that function. > > Matt > > > on my workstation, it is working but on the cluster it gives an error. I > can not run with debugger on the cluster but I have checked and the error > is related to this part and I do not have any problem on cluster when > running in series. Can you please help me about it. > > the error is: > 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d > ocumentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 > /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c > [0]PETSC ERROR: [0] TS user implicit function line 829 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] TSComputeIFunction line 815 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 > /ddnB/work/skavou1/petsc/src/ts/interface/ts.c > [0]PETSC ERROR: [0] SNES user function line 2202 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESComputeFunction line 2187 > /ddnB/work/skavou1/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 > /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c > [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/s > nes/interface/snes.c > [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 > /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/t > s/impls/implicit/theta/theta.c > [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/t > s/interface/ts.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 > 20:11:44 2018 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > ==29057== > ==29057== HEAP SUMMARY: > ==29057== in use at exit: 560,151 bytes in 383 blocks > ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes > allocated > ==29057== > ==29057== LEAK SUMMARY: > ==29057== definitely lost: 0 bytes in 0 blocks > ==29057== indirectly lost: 0 bytes in 0 blocks > ==29057== possibly lost: 0 bytes in 0 blocks > ==29057== still reachable: 560,151 bytes in 383 blocks > ==29057== suppressed: 0 bytes in 0 blocks > ==29057== Rerun with --leak-check=full to see details of leaked memory > ==29057== > ==29057== For counts of detected and suppressed errors, rerun with: -v > ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) > > > > > Thanks, > Sepideh > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Thu Apr 19 13:52:12 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Thu, 19 Apr 2018 18:52:12 +0000 Subject: [petsc-users] error running parallel on cluster In-Reply-To: References: , Message-ID: Thanks. Sepideh ________________________________ From: Matthew Knepley Sent: Thursday, April 19, 2018 1:48:57 PM To: Sepideh Kavousi Cc: Dave May; petsc-users at mcs.anl.gov; Karl Rupp Subject: Re: [petsc-users] error running parallel on cluster Great! I have approved it. Now it goes to master. Matt On Thu, Apr 19, 2018 at 2:43 PM, Sepideh Kavousi > wrote: I submitted the pull request this morning. Regards, Sepideh ________________________________ From: Dave May > Sent: Wednesday, April 18, 2018 11:14:47 PM To: Matthew Knepley Cc: Sepideh Kavousi; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On 18 April 2018 at 23:52, Matthew Knepley > wrote: On Wed, Apr 18, 2018 at 5:52 PM, Sepideh Kavousi > wrote: Mathew and Dave, Thank you so much it is working perfectly now. Excellent. If you want your name to appear on the next PETSc release as a contributor, you can make a PR with this change :) Here is a URL describing the PR's protocol for PETSc contribs: https://bitbucket.org/petsc/petsc/wiki/pull-request-instructions-git Thanks, Matt Sepideh ________________________________ From: Dave May > Sent: Wednesday, April 18, 2018 3:13:33 PM To: Sepideh Kavousi Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On 18 April 2018 at 21:06, Sepideh Kavousi > wrote: Mathew I added the lines and I still have the same issue. It may be a silly question but should I configure and install petsc again using this new lines added? or changing the line is enough? the highlighted lines are the lines I modified. PetscErrorCode ierr; DM dm; DMTS_DA *dmdats = (DMTS_DA*)ctx; DMDALocalInfo info; Vec Xloc,Xdotloc; void *x,*f,*xdot; PetscFunctionBegin; PetscValidHeaderSpecific(ts,TS_CLASSID,1); PetscValidHeaderSpecific(X,VEC_CLASSID,2); PetscValidHeaderSpecific(F,VEC_CLASSID,3); if (!dmdats->ifunctionlocal) SETERRQ(PetscObjectComm((PetscObject)ts),PETSC_ERR_PLIB,"Corrupt context"); ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGetLocalVector(dm,&Xloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,X,INSERT_VALUES,Xloc);CHKERRQ(ierr); ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xloc,&x);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); Don't forget to include these calls (in this order) after you are done with the Xdotloc vector ierr = DMDAVecRestoreArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); ierr = DMRestoreLocalVector(dm,&Xdotloc);CHKERRQ(ierr); Failure to do so will result in a memory leak. Thanks, Sepideh ________________________________ From: Matthew Knepley > Sent: Tuesday, April 17, 2018 5:59:12 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Tue, Apr 17, 2018 at 3:07 PM, Sepideh Kavousi > wrote: The reason I can not use your method is that the input arguments of SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your method which was: ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); You misunderstand my suggestion. I mean stick this code in right here in PETSc https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Then the X_t array you get in your local function will be ghosted. Matt I need to have the vector of Xdot, not the array. So I think I should use SetIFunction instead of SetIFunctionLocal. Sepideh ________________________________ From: Matthew Knepley > Sent: Tuesday, April 17, 2018 1:22:53 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi > wrote: Mathew, I previously use DMDATSSetIFunctionLocal(user.da,INSERT_VALUES,(DMDATSIFunctionLocal) FormFunction,&user) in my code. If I want to use your solution I can not use it because in the FormFunction definition I must use arrays, not vectors.So to solve this issue I followed two methods where none were able to solve it. 1- in first method I decided to use TSSetIFunction instead of DMDATSSetIFunctionLocal for this means first in the main function, I use TSSetDM and my form function variables were as: PetscErrorCode FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, struct VAR_STRUCT *user) { . . . . ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr); ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr); ierr = DMGetLocalVector(dmda,&Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dmda,Ydot,INSERT_VALUES,Ydot_local);CHKERRQ(ierr); . . . } But still, it does not consider vectors y,ydot,f related to dmda (problem executing DMDAVecGetArray) We cannot help you if you do not show full error messages. Why not fix the code with SetIFunctionLocal(), as I said in my last email. I will fix PETSc proper in branch at the end of the week. I have a proposal due tomorrow, so I cannot do it right now. Thanks, Matt 2- in second method I decided to use DMTSSetIFunction but still, FormFunction is in form of TSIFunction where we do not define dm object and I think it does not understand dm and da are connected, although I have used TSSetDM in the main function. Can you please help me what should I do? Regards, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 9:34:01 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi > wrote: I run with valgrind with the following command: /work/skavou1/petsc/debug/bin/mpiexec -np 2 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler -pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason -ksp_monitor Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only chooses to ghost the input vector X and not X_t https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68 Unfortunately, this was done by Peter in 2012 and he is gone, so we can't blame him. It is true that very few people use spatial derivatives of X_t, but it does make sense. Right there, you want to add ierr = DMGetLocalVector(dm,&Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,Xdot,INSERT_VALUES,Xdotloc);CHKERRQ(ierr); ierr = DMDAVecGetArray(dm,Xdotloc,&xdot);CHKERRQ(ierr); and it should work. Matt the valgringoutput is as following: ==39396== Invalid read of size 8 ==39396== at 0x402679: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 alloc'd ==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727) ==39396== by 0x4D87516: PetscMallocAlign (mal.c:42) ==39396== by 0x4D88DE0: PetscMallocA (mal.c:397) ==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167) ==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73) ==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== ==39396== Invalid read of size 8 ==39396== at 0x402689: FormFunction (one.c:212) ==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79) ==39396== by 0x61349BD: TSComputeIFunction (ts.c:830) ==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649) ==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569) ==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203) ==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105) ==39396== by 0x60246C0: SNESSolve (snes.c:4312) ==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176) ==39396== by 0x6237CA8: TSStep_Theta (theta.c:216) ==39396== by 0x6153380: TSStep (ts.c:3548) ==39396== by 0x615559D: TSSolve (ts.c:3731) ==39396== by 0x403C95: main (one.c:303) ==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd ==39396== Thanks, Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:44:43 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi > wrote: this is not used forghost points . my code is : 1) You are using ghost points, since you use i-1 as an index 2) I was wrong. X_t should also be ghosted. Run under valgrind. Thanks, Matt for (j=info2->ys;jys+info2->ym;j++){ for (i=info2->xs;ixs+info2->xm;i++){ if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j][i].U ; aF[j][i].p=aY[j][i+1].p-aY[j][i].p ;} else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j][i-1].p ;} else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j][i].U; aF[j][i].p=aY[j+1][i].p-aY[j][i].p ;} else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j][i].U ; aF[j][i].p=aY[j][i].p-aY[j-1][i].p ;} else { //derivatives of U and p Px=(aY[j][i+1].p-aY[j][i-1].p)/(2.0*user->hx); Py=(aY[j+1][i].p-aY[j-1][i].p)/(2.0*user->hy); Pxx=((aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2)); Pxy=((aY[j+1][i+1].p+aY[j-1][i-1].p-aY[j+1][i-1].p-aY[j-1][i+1].p)/(4.0*hxy)); Pyy=((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2); Pxt=(aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx); Pyt=(aYdot[j+1][i].p-aYdot[j-1][i].p)/(2.0*user->hy); if (((Px*Px)+(Py*Py))>0.0 ) { // theta=PetscAtanReal(Py/Px); thetax=(Px*Pxy-Py*Pxx)/((Px*Px)+(Py*Py)); thetay=(Px*Pyy-Py*Pxy)/((Px*Px)+(Py*Py)); } else { theta=PETSC_PI*0.5; thetax=0.0; thetay=0.0; } e=(1.0+user->epsilon*cos(4.0*theta));; ep=-(4.0*user->epsilon*sin(4.0*theta)); epp=-(4.0*user->epsilon*4.0*cos(4.0*theta)); Ux=(aY[j][i+1].U-aY[j][i-1].U)/(2.0*user->hx); Uy=(aY[j+1][i].U-aY[j-1][i].U)/(2.0*user->hy); Uxx=((aY[j][i+1].U+aY[j][i-1].U-2.0*aY[j][i].U)/(hx2)); Uyy=((aY[j+1][i].U+aY[j-1][i].U-2.0*aY[j][i].U)/hy2); U1=user->D*user->tau_0/(user->W*user->W); U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : 0.0 ; //U2=0.0; P1= (user->W*user->hy*j-user->Vp*user->tau_0*t)/user->lT; aF[j][i].U=(U1*(-0.5*(Ux*Px+Uy*Py)+0.5*(1-aY[j][i].p)*(Uxx+Uyy))+U2*(1+(1-user->k)*aY[j][i].U)*(Pxt*Px+Pyt*Py+aYdot[j][i].p*(Pxx+Pyy))+U2*(1-user->k)*aYdot[j][i].p*(Ux*Px+Uy*Py)+0.5*(1+(1-user->k)*aY[j][i].U)*aYdot[j][i].p-0.5*(1+user->k-(1-user->k)*aY[j][i].p)*aYdot[j][i].U)*hx2; aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax*Px+thetay*Py)+(e*epp+ep*ep)*(thetay*Px-thetax*Py)+(aY[j][i].p-aY[j][i].p*aY[j][i].p*aY[j][i].p)-user->lambda*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].p*aY[j][i].p-1)*(aY[j][i].U+P1)-(1-(1-user->k)*P1)*(1.0+4.0*user->epsilon*cos(4.0*theta))*(1.0+4.0*user->epsilon*cos(4.0*theta))*aYdot[j][i].p )*hx2 ; } } Sepideh ________________________________ From: Matthew Knepley > Sent: Monday, April 16, 2018 8:36:37 PM To: Sepideh Kavousi Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] error running parallel on cluster On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi > wrote: Hello, I am solving a PDE where I have the spacial derivtive of the time derivative of a variable. In the DMDATSSetIFunctionLocal function I defined d(dY/dt)/dx as: (aYdot[j][i+1].p-aYdot[j][i-1].p)/(2.0*user->hx) I do not think that you get ghosted Ydot in that function. Matt on my workstation, it is working but on the cluster it gives an error. I can not run with debugger on the cluster but I have checked and the error is related to this part and I do not have any problem on cluster when running in series. Can you please help me about it. the error is: 0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 /ddnB/work/skavou1/petsc/src/ts/utils/dmdats.c [0]PETSC ERROR: [0] TS user implicit function line 829 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSComputeIFunction line 815 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] SNESTSFormFunction line 4564 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] SNES user function line 2202 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESComputeFunction line 2187 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 /ddnB/work/skavou1/petsc/src/snes/impls/tr/tr.c [0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/ts/interface/ts.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 20:11:44 2018 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ==29057== ==29057== HEAP SUMMARY: ==29057== in use at exit: 560,151 bytes in 383 blocks ==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes allocated ==29057== ==29057== LEAK SUMMARY: ==29057== definitely lost: 0 bytes in 0 blocks ==29057== indirectly lost: 0 bytes in 0 blocks ==29057== possibly lost: 0 bytes in 0 blocks ==29057== still reachable: 560,151 bytes in 383 blocks ==29057== suppressed: 0 bytes in 0 blocks ==29057== Rerun with --leak-check=full to see details of leaked memory ==29057== ==29057== For counts of detected and suppressed errors, rerun with: -v ==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6) Thanks, Sepideh -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbuesing at eonerc.rwth-aachen.de Fri Apr 20 04:31:13 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Fri, 20 Apr 2018 09:31:13 +0000 Subject: [petsc-users] Matrix-Free Newton Krylov with Automatic Differentiation and User Provided Preconditioner Message-ID: Dear all, I would like to use the matrix-free feature from PETSc, but I do not want to use finite differences to calculate J*u but automatic differentiation (AD). Since I do not want to live totally without preconditioning, I want to build important parts of the Jacobian and pass it for preconditioning. As I understand it, I would create a Shell-Matrix A and associate my UserMult function with it. This uses my AD Jacobian J and computes J*u without actually forming J. Then I will create a different matrix P, which will hold a partly assembled Jacobian for preconditioning. My FormJacobian function then takes this Shell-Matrix as Amat and my preconditioning matrix P as Pmat. Within FormJacobian I build the matrix Pmat, which holds parts of my Jacobian. Does this make sense? Is this the way how I am supposed to do it? Do UserMult and FormJacobian function do the right things? Thank you! Henrik -- Dipl.-Math. Henrik B?sing Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ----------------------------------------------- Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ----------------------------------------------- http://www.eonerc.rwth-aachen.de/GGE hbuesing at eonerc.rwth-aachen.de ----------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 20 05:45:21 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 20 Apr 2018 06:45:21 -0400 Subject: [petsc-users] Matrix-Free Newton Krylov with Automatic Differentiation and User Provided Preconditioner In-Reply-To: References: Message-ID: On Fri, Apr 20, 2018 at 5:31 AM, Buesing, Henrik < hbuesing at eonerc.rwth-aachen.de> wrote: > Dear all, > > I would like to use the matrix-free feature from PETSc, but I do not want > to use finite differences to calculate J*u but automatic differentiation > (AD). Since I do not want to live totally without preconditioning, I want > to build important parts of the Jacobian and pass it for preconditioning. > > As I understand it, I would create a Shell-Matrix A and associate my > UserMult function with it. This uses my AD Jacobian J and computes J*u > without actually forming J. Then I will create a different matrix P, which > will hold a partly assembled Jacobian for preconditioning. > > My FormJacobian function then takes this Shell-Matrix as Amat and my > preconditioning matrix P as Pmat. Within FormJacobian I build the matrix > Pmat, which holds parts of my Jacobian. > > Does this make sense? Is this the way how I am supposed to do it? Do > UserMult and FormJacobian function do the right things? > Yes, this is the intended use. Thanks, Matt > Thank you! > Henrik > > > > > -- > Dipl.-Math. Henrik B?sing > Applied Geophysics and Geothermal Energy > E.ON Energy Research Center > RWTH Aachen University > ----------------------------------------------- > Mathieustr. 10 > | > Tel +49 (0)241 80 49907 > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > ----------------------------------------------- > http://www.eonerc.rwth-aachen.de/GGE > hbuesing at eonerc.rwth-aachen.de > ----------------------------------------------- > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From k_burkart at yahoo.com Fri Apr 20 10:05:44 2018 From: k_burkart at yahoo.com (Klaus Burkart) Date: Fri, 20 Apr 2018 15:05:44 +0000 (UTC) Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> Message-ID: <1215145994.3164869.1524236744544@mail.yahoo.com> I think I understood the matrix structure for parallel computation with the rows, diagonal (d) and off-diagonal (o) structure, where I have problems is how to do the setup including memory allocation in PETSc: Lets assume, I use a 16 core workstation (=16 processes) and the number of nonzeros varies in each row for both d and o and the number of rows assigned to each process differs too - at least for the nth process. Looking at the manual and http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateAIJ.html#MatCreateAIJ, I don't understand how to enter a global matrix when n is NOT the same for each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not arrays? ??? MatCreate(...,&A); ??? MatSetType(A,MATMPIAIJ); ??? MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same for each process which isn't even the case in the example on the page mentioned above?! ??? MatMPIAIJSetPreallocation(A,...); How can I enter the parallel global-local matrix structure? How can the memory preallocation be done? Klaus Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. Folgendes geschrieben: > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote: > > So, practically speaking, l should invent routines to decompose the matrix e.g. into a block matrix structure to be able to make real use of PETSc ie. be able to solve a linear system using more than one process/core? ? To really use PETSc efficiently/effectively you need to generate your matrix in parallel. ? Barry > > Klaus > > Von meinem Huawei-Mobiltelefon gesendet > > > -------- Originalnachricht -------- > Betreff: Re: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? > Von: "Smith, Barry F." > An: Klaus Burkart > Cc: PETSc Users List > > > > If you can only generate the nonzero allocation sequentially you can only solve sequentially which means your matrix is MATSEQAIJ and your vector is VECSEQ and your communicator is PETSC_COMM_SELF. > > If you pass and array for nnz, what you pass for nz is irrelevant, you might as well pass 0. > > Barry > > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: > > > > More questions about matrix and vector type selection for my application: > > > > My starting point is a huge sparse matrix which can be symmetric or asymmetric and a rhs vector. There's no defined local or block structure at all, just row and column indices and the values and an array style rhs vector together describing the entire linear system to be solved. With quite some effort, I should be able to create an array nnz[N] containing the number of nonzeros per row in the global matrix for memory allocation which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I assume here, that 0 indicates different numbers of nonzero values in each row, the exact number being stored in the nnz array. Regarding this detail but one example assume a constant number of nz per row so I am not sure whether I should write 0 or NULL for nz? > > > > I started with: > > > > MatCreate(PETSC_COMM_WORLD, &M); > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > > MatSetFromOptions(M); > > > > taken from a paper and assume, the latter would set the matrix type to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe decompositioning took place at an earlier stage and the authors of the paper were able to retrieve the local data and structure. > > > > What type of matrix and vector should I use for my application e.g. MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, nnz); for efficient memory allocation? > > > > In this case, where would the decompositioning / MPI process allocation take place? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 20 10:12:22 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 20 Apr 2018 15:12:22 +0000 Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: <1215145994.3164869.1524236744544@mail.yahoo.com> References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> <1215145994.3164869.1524236744544@mail.yahoo.com> Message-ID: For square matrices almost always n is the same as m. On different processes m can be different. You get to decide what makes sense for each processes what its m should be. Barry > On Apr 20, 2018, at 10:05 AM, Klaus Burkart wrote: > > I think I understood the matrix structure for parallel computation with the rows, diagonal (d) and off-diagonal (o) structure, where I have problems is how to do the setup including memory allocation in PETSc: > > Lets assume, I use a 16 core workstation (=16 processes) and the number of nonzeros varies in each row for both d and o and the number of rows assigned to each process differs too - at least for the nth process. > > Looking at the manual and http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateAIJ.html#MatCreateAIJ, I don't understand how to enter a global matrix when n is NOT the same for each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not arrays? > > MatCreate(...,&A); > > MatSetType(A,MATMPIAIJ); > > MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same for each process which isn't even the case in the example on the page mentioned above?! > > MatMPIAIJSetPreallocation(A,...); > > > How can I enter the parallel global-local matrix structure? > > How can the memory preallocation be done? > > Klaus > > Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. Folgendes geschrieben: > > > > > > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote: > > > > So, practically speaking, l should invent routines to decompose the matrix e.g. into a block matrix structure to be able to make real use of PETSc ie. be able to solve a linear system using more than one process/core? > > To really use PETSc efficiently/effectively you need to generate your matrix in parallel. > > Barry > > > > > Klaus > > > > Von meinem Huawei-Mobiltelefon gesendet > > > > > > -------- Originalnachricht -------- > > Betreff: Re: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? > > Von: "Smith, Barry F." > > An: Klaus Burkart > > Cc: PETSc Users List > > > > > > > > If you can only generate the nonzero allocation sequentially you can only solve sequentially which means your matrix is MATSEQAIJ and your vector is VECSEQ and your communicator is PETSC_COMM_SELF. > > > > If you pass and array for nnz, what you pass for nz is irrelevant, you might as well pass 0. > > > > Barry > > > > > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: > > > > > > More questions about matrix and vector type selection for my application: > > > > > > My starting point is a huge sparse matrix which can be symmetric or asymmetric and a rhs vector. There's no defined local or block structure at all, just row and column indices and the values and an array style rhs vector together describing the entire linear system to be solved. With quite some effort, I should be able to create an array nnz[N] containing the number of nonzeros per row in the global matrix for memory allocation which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I assume here, that 0 indicates different numbers of nonzero values in each row, the exact number being stored in the nnz array. Regarding this detail but one example assume a constant number of nz per row so I am not sure whether I should write 0 or NULL for nz? > > > > > > I started with: > > > > > > MatCreate(PETSC_COMM_WORLD, &M); > > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > > > MatSetFromOptions(M); > > > > > > taken from a paper and assume, the latter would set the matrix type to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe decompositioning took place at an earlier stage and the authors of the paper were able to retrieve the local data and structure. > > > > > > What type of matrix and vector should I use for my application e.g. MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, nnz); for efficient memory allocation? > > > > > > In this case, where would the decompositioning / MPI process allocation take place? > > From knepley at gmail.com Fri Apr 20 11:14:12 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 20 Apr 2018 12:14:12 -0400 Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: <1215145994.3164869.1524236744544@mail.yahoo.com> References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> <1215145994.3164869.1524236744544@mail.yahoo.com> Message-ID: On Fri, Apr 20, 2018 at 11:05 AM, Klaus Burkart wrote: > I think I understood the matrix structure for parallel computation with > the rows, diagonal (d) and off-diagonal (o) structure, where I have > problems is how to do the setup including memory allocation in PETSc: > > Lets assume, I use a 16 core workstation (=16 processes) and the number of > nonzeros varies in each row for both d and o and the number of rows > assigned to each process differs too - at least for the nth process. > > Looking at the manual and http://www.mcs.anl.gov/petsc/petsc-current/docs/ > manualpages/Mat/MatCreateAIJ.html#MatCreateAIJ, I don't understand how to > enter a global matrix when n is NOT the same for each process as e.g. in > MatSetSizes(A, m,n,M,N); n and m are integers, not arrays? > This is SPMD programming, so the call on each process gives a different n, and collectively you have 16 different n values. "Using MPI" is a great book for understanding this. Thanks, Matt > MatCreate(...,&A); > > MatSetType(A,MATMPIAIJ); > > MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same for > each process which isn't even the case in the example on the page mentioned > above?! > > MatMPIAIJSetPreallocation(A,...); > > > How can I enter the parallel global-local matrix structure? > > How can the memory preallocation be done? > > Klaus > > Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. < > bsmith at mcs.anl.gov> Folgendes geschrieben: > > > > > > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote: > > > > So, practically speaking, l should invent routines to decompose the > matrix e.g. into a block matrix structure to be able to make real use of > PETSc ie. be able to solve a linear system using more than one process/core? > > To really use PETSc efficiently/effectively you need to generate your > matrix in parallel. > > Barry > > > > > Klaus > > > > Von meinem Huawei-Mobiltelefon gesendet > > > > > > -------- Originalnachricht -------- > > Betreff: Re: [petsc-users] Matrix and vector type selection & memory > allocation for efficient matrix import? > > Von: "Smith, Barry F." > > An: Klaus Burkart > > Cc: PETSc Users List > > > > > > > > If you can only generate the nonzero allocation sequentially you can > only solve sequentially which means your matrix is MATSEQAIJ and your > vector is VECSEQ and your communicator is PETSC_COMM_SELF. > > > > If you pass and array for nnz, what you pass for nz is irrelevant, you > might as well pass 0. > > > > Barry > > > > > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: > > > > > > More questions about matrix and vector type selection for my > application: > > > > > > My starting point is a huge sparse matrix which can be symmetric or > asymmetric and a rhs vector. There's no defined local or block structure at > all, just row and column indices and the values and an array style rhs > vector together describing the entire linear system to be solved. With > quite some effort, I should be able to create an array nnz[N] containing > the number of nonzeros per row in the global matrix for memory allocation > which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only > option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I > assume here, that 0 indicates different numbers of nonzero values in each > row, the exact number being stored in the nnz array. Regarding this detail > but one example assume a constant number of nz per row so I am not sure > whether I should write 0 or NULL for nz? > > > > > > I started with: > > > > > > MatCreate(PETSC_COMM_WORLD, &M); > > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > > > MatSetFromOptions(M); > > > > > > taken from a paper and assume, the latter would set the matrix type to > MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe > decompositioning took place at an earlier stage and the authors of the > paper were able to retrieve the local data and structure. > > > > > > What type of matrix and vector should I use for my application e.g. > MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, > nnz); for efficient memory allocation? > > > > > > In this case, where would the decompositioning / MPI process > allocation take place? > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From k_burkart at yahoo.com Fri Apr 20 12:30:54 2018 From: k_burkart at yahoo.com (Klaus Burkart) Date: Fri, 20 Apr 2018 17:30:54 +0000 (UTC) Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> <1215145994.3164869.1524236744544@mail.yahoo.com> Message-ID: <897258597.3227571.1524245454851@mail.yahoo.com> In my case N=M but n for process 0, 1, 2, 3,... no_processes-1 can be different from the nth process like in the example where the nth process=Proc2 and has only two rows while all other processes have three rows: Example from the PETSc webpage mentioned before: 1 2 0 | 0 3 0 | 0 4 Proc0 0 5 6 | 7 0 0 | 8 0 9 0 10 | 11 0 0 | 12 0 ------------------------------------- 13 0 14 | 15 16 17 | 0 0 Proc1 0 18 0 | 19 20 21 | 0 0 0 0 0 | 22 23 0 | 24 0 ------------------------------------- Proc2 25 26 27 | 0 0 28 | 29 0 30 0 0 | 31 32 33 | 0 34and I need to enter different values for d_nnz and o_nnz for each row somewhere too proc0: d_nnz = [2,2,2] and o_nnz = [2,2,2] proc1: d_nnz = [3,3,2] and o_nnz = [2,1,1] proc2: d_nnz = [1,1] and o_nnz = [4,4] I simply can't identify the function(s) used to set the values for n, d_nnz and o_nnz for the individual local matrices allocated to all the processes if n isn't the same for all processes and d_nnz and o_nnz are different for each local matrix? Approach described on the PETSc webpage: MatCreate(...,&A); MatSetType(A,MATMPIAIJ); MatSetSizes(A, m,n,M,N); // for the example above using this function would set the no. of rows for Proc2 to 3 but it's 2 MatMPIAIJSetPreallocation(A,...); // this function can be used to set values for ONE local matrix only In addition to that I don't know which functions to use to preallocate memory for ALL local matrices when each of them has different values for d_nnz and o_nnz. I other words, what's the code for the 3 process example above? (entering the matrix structure and allocating memory) Klaus Am Freitag, 20. April 2018, 17:13:26 MESZ hat Smith, Barry F. Folgendes geschrieben: ? For square matrices almost always n is the same as m. On different processes m can be different. You get to decide what makes sense for each processes what its m should be. ? Barry > On Apr 20, 2018, at 10:05 AM, Klaus Burkart wrote: > > I think I understood the matrix structure for parallel computation with the rows, diagonal (d) and off-diagonal (o) structure, where I have problems is how to do the setup including memory allocation in PETSc: > > Lets assume, I use a 16 core workstation (=16 processes) and the number of nonzeros varies in each row for both d and o and the number of rows assigned to each process differs too - at least for the nth process. > > Looking at the manual and http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateAIJ.html#MatCreateAIJ, I don't understand how to enter a global matrix when n is NOT the same for each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not arrays? > >? ? MatCreate(...,&A); > >? ? MatSetType(A,MATMPIAIJ); > >? ? MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same for each process which isn't even the case in the example on the page mentioned above?! > >? ? MatMPIAIJSetPreallocation(A,...); > > > How can I enter the parallel global-local matrix structure? > > How can the memory preallocation be done? > > Klaus > > Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. Folgendes geschrieben: > > > > > > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote: > > > > So, practically speaking, l should invent routines to decompose the matrix e.g. into a block matrix structure to be able to make real use of PETSc ie. be able to solve a linear system using more than one process/core? > >? To really use PETSc efficiently/effectively you need to generate your matrix in parallel. > >? Barry > > > > > Klaus > > > > Von meinem Huawei-Mobiltelefon gesendet > > > > > > -------- Originalnachricht -------- > > Betreff: Re: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? > > Von: "Smith, Barry F." > > An: Klaus Burkart > > Cc: PETSc Users List > > > > > > > > If you can only generate the nonzero allocation sequentially you can only solve sequentially which means your matrix is MATSEQAIJ and your vector is VECSEQ and your communicator is PETSC_COMM_SELF. > > > > If you pass and array for nnz, what you pass for nz is irrelevant, you might as well pass 0. > > > > Barry > > > > > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: > > > > > > More questions about matrix and vector type selection for my application: > > > > > > My starting point is a huge sparse matrix which can be symmetric or asymmetric and a rhs vector. There's no defined local or block structure at all, just row and column indices and the values and an array style rhs vector together describing the entire linear system to be solved. With quite some effort, I should be able to create an array nnz[N] containing the number of nonzeros per row in the global matrix for memory allocation which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I assume here, that 0 indicates different numbers of nonzero values in each row, the exact number being stored in the nnz array. Regarding this detail but one example assume a constant number of nz per row so I am not sure whether I should write 0 or NULL for nz? > > > > > > I started with: > > > > > > MatCreate(PETSC_COMM_WORLD, &M); > > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > > > MatSetFromOptions(M); > > > > > > taken from a paper and assume, the latter would set the matrix type to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe decompositioning took place at an earlier stage and the authors of the paper were able to retrieve the local data and structure. > > > > > > What type of matrix and vector should I use for my application e.g. MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, nnz); for efficient memory allocation? > > > > > > In this case, where would the decompositioning / MPI process allocation take place? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 20 12:52:08 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 20 Apr 2018 13:52:08 -0400 Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: <897258597.3227571.1524245454851@mail.yahoo.com> References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> <1215145994.3164869.1524236744544@mail.yahoo.com> <897258597.3227571.1524245454851@mail.yahoo.com> Message-ID: On Fri, Apr 20, 2018 at 1:30 PM, Klaus Burkart wrote: > In my case N=M but n for process 0, 1, 2, 3,... no_processes-1 can be > different from the nth process like in the example where the nth > process=Proc2 and has only two rows while all other processes have three > rows: > Yes. > Example from the PETSc webpage mentioned before: > > 1 2 0 | 0 3 0 | 0 4 > Proc0 0 5 6 | 7 0 0 | 8 0 > 9 0 10 | 11 0 0 | 12 0 > ------------------------------------- > 13 0 14 | 15 16 17 | 0 0 > Proc1 0 18 0 | 19 20 21 | 0 0 > 0 0 0 | 22 23 0 | 24 0 > ------------------------------------- > Proc2 25 26 27 | 0 0 28 | 29 0 > 30 0 0 | 31 32 33 | 0 34 > > and I need to enter different values for d_nnz and o_nnz for each row > somewhere too > > proc0: d_nnz = [2,2,2] and o_nnz = [2,2,2] > proc1: d_nnz = [3,3,2] and o_nnz = [2,1,1] > proc2: d_nnz = [1,1] and o_nnz = [4,4] > > Each process only sets d_nnz and o_nnz for its LOCAL rows. Thus, it is exactly as shown above. Proc1 sets values only for rows 3-5. Thanks, Matt > I simply can't identify the function(s) used to set the values for n, > d_nnz and o_nnz for the individual local matrices allocated to all the > processes if n isn't the same for all processes and d_nnz and o_nnz are > different for each local matrix? > > Approach described on the PETSc webpage: > > MatCreate (...,&A); > > > MatSetType (A,MATMPIAIJ ); > > > MatSetSizes (A, m,n,M,N); // for the example above using this function would set the no. of rows for Proc2 to 3 but it's 2 > > > > MatMPIAIJSetPreallocation (A,...); // this function can be used to set values for ONE local matrix only > > > In addition to that I don't know which functions to use to preallocate memory for ALL local matrices when each of them has different values for d_nnz and o_nnz. > > I other words, what's the code for the 3 process example above? (entering the matrix structure and allocating memory) > > Klaus > > Am Freitag, 20. April 2018, 17:13:26 MESZ hat Smith, Barry F. < > bsmith at mcs.anl.gov> Folgendes geschrieben: > > > > For square matrices almost always n is the same as m. On different > processes m can be different. You get to decide what makes sense for each > processes what its m should be. > > Barry > > > > On Apr 20, 2018, at 10:05 AM, Klaus Burkart wrote: > > > > I think I understood the matrix structure for parallel computation with > the rows, diagonal (d) and off-diagonal (o) structure, where I have > problems is how to do the setup including memory allocation in PETSc: > > > > Lets assume, I use a 16 core workstation (=16 processes) and the number > of nonzeros varies in each row for both d and o and the number of rows > assigned to each process differs too - at least for the nth process. > > > > Looking at the manual and http://www.mcs.anl.gov/petsc/ > petsc-current/docs/manualpages/Mat/MatCreateAIJ.html#MatCreateAIJ, I > don't understand how to enter a global matrix when n is NOT the same for > each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not > arrays? > > > > MatCreate(...,&A); > > > > MatSetType(A,MATMPIAIJ); > > > > MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same for > each process which isn't even the case in the example on the page mentioned > above?! > > > > MatMPIAIJSetPreallocation(A,...); > > > > > > How can I enter the parallel global-local matrix structure? > > > > How can the memory preallocation be done? > > > > Klaus > > > > Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. < > bsmith at mcs.anl.gov> Folgendes geschrieben: > > > > > > > > > > > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote: > > > > > > So, practically speaking, l should invent routines to decompose the > matrix e.g. into a block matrix structure to be able to make real use of > PETSc ie. be able to solve a linear system using more than one process/core? > > > > To really use PETSc efficiently/effectively you need to generate your > matrix in parallel. > > > > Barry > > > > > > > > Klaus > > > > > > Von meinem Huawei-Mobiltelefon gesendet > > > > > > > > > -------- Originalnachricht -------- > > > Betreff: Re: [petsc-users] Matrix and vector type selection & memory > allocation for efficient matrix import? > > > Von: "Smith, Barry F." > > > An: Klaus Burkart > > > Cc: PETSc Users List > > > > > > > > > > > > If you can only generate the nonzero allocation sequentially you can > only solve sequentially which means your matrix is MATSEQAIJ and your > vector is VECSEQ and your communicator is PETSC_COMM_SELF. > > > > > > If you pass and array for nnz, what you pass for nz is irrelevant, you > might as well pass 0. > > > > > > Barry > > > > > > > > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: > > > > > > > > More questions about matrix and vector type selection for my > application: > > > > > > > > My starting point is a huge sparse matrix which can be symmetric or > asymmetric and a rhs vector. There's no defined local or block structure at > all, just row and column indices and the values and an array style rhs > vector together describing the entire linear system to be solved. With > quite some effort, I should be able to create an array nnz[N] containing > the number of nonzeros per row in the global matrix for memory allocation > which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only > option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I > assume here, that 0 indicates different numbers of nonzero values in each > row, the exact number being stored in the nnz array. Regarding this detail > but one example assume a constant number of nz per row so I am not sure > whether I should write 0 or NULL for nz? > > > > > > > > I started with: > > > > > > > > MatCreate(PETSC_COMM_WORLD, &M); > > > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > > > > MatSetFromOptions(M); > > > > > > > > taken from a paper and assume, the latter would set the matrix type > to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe > decompositioning took place at an earlier stage and the authors of the > paper were able to retrieve the local data and structure. > > > > > > > > What type of matrix and vector should I use for my application e.g. > MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, > nnz); for efficient memory allocation? > > > > > > > > In this case, where would the decompositioning / MPI process > allocation take place? > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jczhang at mcs.anl.gov Fri Apr 20 13:55:36 2018 From: jczhang at mcs.anl.gov (Junchao Zhang) Date: Fri, 20 Apr 2018 13:55:36 -0500 Subject: [petsc-users] How to adapt vectors to matrices loaded from a viewer Message-ID: In a PETSc example (ex10.c ) one MatLoad A and then VecLoad b from a viewer. Since one can change mat block size through option -matload_block_size, the code tries to pad b if A and b's sizes mismatch, using the following test 186: MatGetSize(A,&M,NULL);187: VecGetSize(b,&m);188: if (M != m) { /* Create a new vector b by padding the old one */ I think the code is wrong. One has to compare local column size of A and local size of b, and if there is a mismatch on ANY processor, then one has to create a new b. My questions are: Are users supposed to take this complexity? Does PETSc provide a neat way to do that, for example, MatVecLoad(A,b,viewer) Thanks. --Junchao Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From k_burkart at yahoo.com Fri Apr 20 15:07:50 2018 From: k_burkart at yahoo.com (Klaus Burkart) Date: Fri, 20 Apr 2018 20:07:50 +0000 (UTC) Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> <1215145994.3164869.1524236744544@mail.yahoo.com> <897258597.3227571.1524245454851@mail.yahoo.com> Message-ID: <270008272.3263240.1524254870285@mail.yahoo.com> Sorry, I don't understand you: The example says: Step 1: Create the gobal matrix ??? MatCreate(...,&A); // no problem Step 2: Make it a parallel matrix ??? MatSetType(A,MATMPIAIJ); // no problem Step 3: Define the size of the global matrix and the number of rows per process IF this number is the same for all processes ??? MatSetSizes(A, N,n,N,N); In the example, I have the problem with n which is 3 or 2 depending on the process but I can only set n to 3 or 2 so it will be the wrong for at least one process Step 4: Preallocate memory for the d_nnz and o_nnz values which are different for each row (for each n) ??? MatMPIAIJSetPreallocation(A,0,d_nnz[n],0,o_nnz[n]); // How to do this repeatedly for all processes especially when using PETEC_DECIDE for n and m as shown in many examples in which case I don't even know the n per process I have to retrieve the relevant values (content and structure) from an application and I absolutely don't understand how to enter the exact same matrix structure as shown in the example (only much larger) into PETSc using the retrieved data? How would assign the d_nnz and o_nnz values of a global matrix on a 32 core workstation to the 32 processes using MatMPIAIJSetPreallocation(A,0,d_nnz[n],0,o_nnz[n]);? (Which means in my case assigning 64 arrays with different content containing the d_nnz and o_nnz? values for the 32 processes) Would I use a for loop - but then how to get hold of the individual processes? Same for Step 3? I don't get the underlying idea about how to create a parallel matrix structure in PETSc - don't get me wrong I understand the layout "simply" not how to enter more than one local matrix it into PETSc. Klaus Am Freitag, 20. April 2018, 19:52:11 MESZ hat Matthew Knepley Folgendes geschrieben: On Fri, Apr 20, 2018 at 1:30 PM, Klaus Burkart wrote: In my case N=M but n for process 0, 1, 2, 3,... no_processes-1 can be different from the nth process like in the example where the nth process=Proc2 and has only two rows while all other processes have three rows: Yes.? Example from the PETSc webpage mentioned before: 1 2 0 | 0 3 0 | 0 4 Proc0 0 5 6 | 7 0 0 | 8 0 9 0 10 | 11 0 0 | 12 0 ------------------------------ ------- 13 0 14 | 15 16 17 | 0 0 Proc1 0 18 0 | 19 20 21 | 0 0 0 0 0 | 22 23 0 | 24 0 ------------------------------ ------- Proc2 25 26 27 | 0 0 28 | 29 0 30 0 0 | 31 32 33 | 0 34and I need to enter different values for d_nnz and o_nnz for each row somewhere too proc0: d_nnz = [2,2,2] and o_nnz = [2,2,2] proc1: d_nnz = [3,3,2] and o_nnz = [2,1,1] proc2: d_nnz = [1,1] and o_nnz = [4,4] Each process only sets d_nnz and o_nnz for its LOCAL rows. Thus, it is exactly as shown above. Proc1 sets valuesonly for rows 3-5. ? Thanks, ? ? Matt? I simply can't identify the function(s) used to set the values for n, d_nnz and o_nnz for the individual local matrices allocated to all the processes if n isn't the same for all processes and d_nnz and o_nnz are different for each local matrix? Approach described on the PETSc webpage: MatCreate(...,&A); MatSetType(A,MATMPIAIJ); MatSetSizes(A, m,n,M,N); // for the example above using this function would set the no. of rows for Proc2 to 3 but it's 2 MatMPIAIJSetPreallocation(A,.. .); // this function can be used to set values for ONE local matrix only In addition to that I don't know which functions to use to preallocate memory for ALL local matrices when each of them has different values for d_nnz and o_nnz. I other words, what's the code for the 3 process example above? (entering the matrix structure and allocating memory) Klaus Am Freitag, 20. April 2018, 17:13:26 MESZ hat Smith, Barry F. Folgendes geschrieben: ? For square matrices almost always n is the same as m. On different processes m can be different. You get to decide what makes sense for each processes what its m should be. ? Barry > On Apr 20, 2018, at 10:05 AM, Klaus Burkart wrote: > > I think I understood the matrix structure for parallel computation with the rows, diagonal (d) and off-diagonal (o) structure, where I have problems is how to do the setup including memory allocation in PETSc: > > Lets assume, I use a 16 core workstation (=16 processes) and the number of nonzeros varies in each row for both d and o and the number of rows assigned to each process differs too - at least for the nth process. > > Looking at the manual and http://www.mcs.anl.gov/petsc/ petsc-current/docs/ manualpages/Mat/MatCreateAIJ. html#MatCreateAIJ, I don't understand how to enter a global matrix when n is NOT the same for each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not arrays? > >? ? MatCreate(...,&A); > >? ? MatSetType(A,MATMPIAIJ); > >? ? MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same for each process which isn't even the case in the example on the page mentioned above?! > >? ? MatMPIAIJSetPreallocation(A,.. .); > > > How can I enter the parallel global-local matrix structure? > > How can the memory preallocation be done? > > Klaus > > Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. Folgendes geschrieben: > > > > > > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote: > > > > So, practically speaking, l should invent routines to decompose the matrix e.g. into a block matrix structure to be able to make real use of PETSc ie. be able to solve a linear system using more than one process/core? > >? To really use PETSc efficiently/effectively you need to generate your matrix in parallel. > >? Barry > > > > > Klaus > > > > Von meinem Huawei-Mobiltelefon gesendet > > > > > > -------- Originalnachricht -------- > > Betreff: Re: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? > > Von: "Smith, Barry F." > > An: Klaus Burkart > > Cc: PETSc Users List > > > > > > > > If you can only generate the nonzero allocation sequentially you can only solve sequentially which means your matrix is MATSEQAIJ and your vector is VECSEQ and your communicator is PETSC_COMM_SELF. > > > > If you pass and array for nnz, what you pass for nz is irrelevant, you might as well pass 0. > > > > Barry > > > > > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: > > > > > > More questions about matrix and vector type selection for my application: > > > > > > My starting point is a huge sparse matrix which can be symmetric or asymmetric and a rhs vector. There's no defined local or block structure at all, just row and column indices and the values and an array style rhs vector together describing the entire linear system to be solved. With quite some effort, I should be able to create an array nnz[N] containing the number of nonzeros per row in the global matrix for memory allocation which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I assume here, that 0 indicates different numbers of nonzero values in each row, the exact number being stored in the nnz array. Regarding this detail but one example assume a constant number of nz per row so I am not sure whether I should write 0 or NULL for nz? > > > > > > I started with: > > > > > > MatCreate(PETSC_COMM_WORLD, &M); > > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > > > MatSetFromOptions(M); > > > > > > taken from a paper and assume, the latter would set the matrix type to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe decompositioning took place at an earlier stage and the authors of the paper were able to retrieve the local data and structure. > > > > > > What type of matrix and vector should I use for my application e.g. MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, nnz); for efficient memory allocation? > > > > > > In this case, where would the decompositioning / MPI process allocation take place? > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 20 15:17:02 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 20 Apr 2018 16:17:02 -0400 Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: <270008272.3263240.1524254870285@mail.yahoo.com> References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> <1215145994.3164869.1524236744544@mail.yahoo.com> <897258597.3227571.1524245454851@mail.yahoo.com> <270008272.3263240.1524254870285@mail.yahoo.com> Message-ID: On Fri, Apr 20, 2018 at 4:07 PM, Klaus Burkart wrote: > Sorry, I don't understand you: > > The example says: > > Step 1: Create the gobal matrix > > MatCreate(...,&A); // no problem > > Step 2: Make it a parallel matrix > > MatSetType(A,MATMPIAIJ); // no problem > > Step 3: Define the size of the global matrix and the number of rows per > process IF this number is the same for all processes > > MatSetSizes(A, N,n,N,N); In the example, I have the problem with n > which is 3 or 2 depending on the process but I can only set n to 3 or 2 so > it will be the wrong for at least one process > 1) Get the book "Using MPI" just like I suggested. It explains this part of parallel programming that you do not understand. 2) Suppose we have two processes P0 and P1. Here are the call made on both processes for a matrix with 5 rows split 3, 2: P0: MatSetSizes(A, 3, 3, 5, 5); P1: MatSetSizes(A, 2, 2, 5, 5); See how different processes give different numbers to the routine? This is what SPMD programming is all about. Thanks, Matt > Step 4: Preallocate memory for the d_nnz and o_nnz values which are > different for each row (for each n) > > MatMPIAIJSetPreallocation(A,0,d_nnz[n],0,o_nnz[n]); // How to do this > repeatedly for all processes especially when using PETEC_DECIDE for n and m > as shown in many examples in which case I don't even know the n per process > > I have to retrieve the relevant values (content and structure) from an > application and I absolutely don't understand how to enter the exact same > matrix structure as shown in the example (only much larger) into PETSc > using the retrieved data? > > How would assign the d_nnz and o_nnz values of a global matrix on a 32 > core workstation to the 32 processes using MatMPIAIJSetPreallocation(A,0,d_nnz[n],0,o_nnz[n]);? > (Which means in my case assigning 64 arrays with different content > containing the d_nnz and o_nnz values for the 32 processes) > > Would I use a for loop - but then how to get hold of the individual > processes? Same for Step 3? > > I don't get the underlying idea about how to create a parallel matrix > structure in PETSc - don't get me wrong I understand the layout "simply" > not how to enter more than one local matrix it into PETSc. > > Klaus > > Am Freitag, 20. April 2018, 19:52:11 MESZ hat Matthew Knepley < > knepley at gmail.com> Folgendes geschrieben: > > > On Fri, Apr 20, 2018 at 1:30 PM, Klaus Burkart > wrote: > > In my case N=M but n for process 0, 1, 2, 3,... no_processes-1 can be > different from the nth process like in the example where the nth > process=Proc2 and has only two rows while all other processes have three > rows: > > > Yes. > > > Example from the PETSc webpage mentioned before: > > 1 2 0 | 0 3 0 | 0 4 > Proc0 0 5 6 | 7 0 0 | 8 0 > 9 0 10 | 11 0 0 | 12 0 > ------------------------------ ------- > 13 0 14 | 15 16 17 | 0 0 > Proc1 0 18 0 | 19 20 21 | 0 0 > 0 0 0 | 22 23 0 | 24 0 > ------------------------------ ------- > Proc2 25 26 27 | 0 0 28 | 29 0 > 30 0 0 | 31 32 33 | 0 34 > > and I need to enter different values for d_nnz and o_nnz for each row > somewhere too > > proc0: d_nnz = [2,2,2] and o_nnz = [2,2,2] > proc1: d_nnz = [3,3,2] and o_nnz = [2,1,1] > proc2: d_nnz = [1,1] and o_nnz = [4,4] > > Each process only sets d_nnz and o_nnz for its LOCAL rows. Thus, it is > exactly as shown above. Proc1 sets values > only for rows 3-5. > > Thanks, > > Matt > > > I simply can't identify the function(s) used to set the values for n, > d_nnz and o_nnz for the individual local matrices allocated to all the > processes if n isn't the same for all processes and d_nnz and o_nnz are > different for each local matrix? > > Approach described on the PETSc webpage: > > MatCreate (...,&A); > > > > MatSetType (A,MATMPIAIJ ); > > > > MatSetSizes (A, m,n,M,N); // for the example above using this function would set the no. of rows for Proc2 to 3 but it's 2 > > > > > MatMPIAIJSetPreallocation (A,.. .); // this function can be used to set values for ONE local matrix only > > > > In addition to that I don't know which functions to use to preallocate > memory for ALL local matrices when each of them has different values for > d_nnz and o_nnz. > > I other words, what's the code for the 3 process example above? (entering > the matrix structure and allocating memory) > > Klaus > > Am Freitag, 20. April 2018, 17:13:26 MESZ hat Smith, Barry F. < > bsmith at mcs.anl.gov> Folgendes geschrieben: > > > > For square matrices almost always n is the same as m. On different > processes m can be different. You get to decide what makes sense for each > processes what its m should be. > > Barry > > > > On Apr 20, 2018, at 10:05 AM, Klaus Burkart wrote: > > > > I think I understood the matrix structure for parallel computation with > the rows, diagonal (d) and off-diagonal (o) structure, where I have > problems is how to do the setup including memory allocation in PETSc: > > > > Lets assume, I use a 16 core workstation (=16 processes) and the number > of nonzeros varies in each row for both d and o and the number of rows > assigned to each process differs too - at least for the nth process. > > > > Looking at the manual and http://www.mcs.anl.gov/petsc/ > petsc-current/docs/ manualpages/Mat/MatCreateAIJ. html#MatCreateAIJ, > I > don't understand how to enter a global matrix when n is NOT the same for > each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not > arrays? > > > > MatCreate(...,&A); > > > > MatSetType(A,MATMPIAIJ); > > > > MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same for > each process which isn't even the case in the example on the page mentioned > above?! > > > > MatMPIAIJSetPreallocation(A,.. .); > > > > > > How can I enter the parallel global-local matrix structure? > > > > How can the memory preallocation be done? > > > > Klaus > > > > Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. < > bsmith at mcs.anl.gov> Folgendes geschrieben: > > > > > > > > > > > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote: > > > > > > So, practically speaking, l should invent routines to decompose the > matrix e.g. into a block matrix structure to be able to make real use of > PETSc ie. be able to solve a linear system using more than one process/core? > > > > To really use PETSc efficiently/effectively you need to generate your > matrix in parallel. > > > > Barry > > > > > > > > Klaus > > > > > > Von meinem Huawei-Mobiltelefon gesendet > > > > > > > > > -------- Originalnachricht -------- > > > Betreff: Re: [petsc-users] Matrix and vector type selection & memory > allocation for efficient matrix import? > > > Von: "Smith, Barry F." > > > An: Klaus Burkart > > > Cc: PETSc Users List > > > > > > > > > > > > If you can only generate the nonzero allocation sequentially you can > only solve sequentially which means your matrix is MATSEQAIJ and your > vector is VECSEQ and your communicator is PETSC_COMM_SELF. > > > > > > If you pass and array for nnz, what you pass for nz is irrelevant, you > might as well pass 0. > > > > > > Barry > > > > > > > > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: > > > > > > > > More questions about matrix and vector type selection for my > application: > > > > > > > > My starting point is a huge sparse matrix which can be symmetric or > asymmetric and a rhs vector. There's no defined local or block structure at > all, just row and column indices and the values and an array style rhs > vector together describing the entire linear system to be solved. With > quite some effort, I should be able to create an array nnz[N] containing > the number of nonzeros per row in the global matrix for memory allocation > which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only > option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I > assume here, that 0 indicates different numbers of nonzero values in each > row, the exact number being stored in the nnz array. Regarding this detail > but one example assume a constant number of nz per row so I am not sure > whether I should write 0 or NULL for nz? > > > > > > > > I started with: > > > > > > > > MatCreate(PETSC_COMM_WORLD, &M); > > > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > > > > MatSetFromOptions(M); > > > > > > > > taken from a paper and assume, the latter would set the matrix type > to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe > decompositioning took place at an earlier stage and the authors of the > paper were able to retrieve the local data and structure. > > > > > > > > What type of matrix and vector should I use for my application e.g. > MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, > nnz); for efficient memory allocation? > > > > > > > > > In this case, where would the decompositioning / MPI process > allocation take place? > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ling.zou at inl.gov Fri Apr 20 15:28:30 2018 From: ling.zou at inl.gov (Zou, Ling) Date: Fri, 20 Apr 2018 14:28:30 -0600 Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> <1215145994.3164869.1524236744544@mail.yahoo.com> <897258597.3227571.1524245454851@mail.yahoo.com> <270008272.3263240.1524254870285@mail.yahoo.com> Message-ID: Mat, is this book you recommended? https://www.amazon.com/Using-MPI-Programming-Message-Passing-Engineering/dp/0262527391/ref=pd_lpo_sbs_14_img_0?_encoding=UTF8&psc=1&refRID=EYCV0H0J5EQ9M0GDKWFT Thanks, Ling On Fri, Apr 20, 2018 at 2:17 PM, Matthew Knepley wrote: > On Fri, Apr 20, 2018 at 4:07 PM, Klaus Burkart > wrote: > >> Sorry, I don't understand you: >> >> The example says: >> >> Step 1: Create the gobal matrix >> >> MatCreate(...,&A); // no problem >> >> Step 2: Make it a parallel matrix >> >> MatSetType(A,MATMPIAIJ); // no problem >> >> Step 3: Define the size of the global matrix and the number of rows per >> process IF this number is the same for all processes >> >> MatSetSizes(A, N,n,N,N); In the example, I have the problem with n >> which is 3 or 2 depending on the process but I can only set n to 3 or 2 so >> it will be the wrong for at least one process >> > > 1) Get the book "Using MPI" just like I suggested. It explains this part > of parallel programming that you do not understand. > > 2) Suppose we have two processes P0 and P1. Here are the call made on both > processes for a matrix with 5 rows split 3, 2: > > P0: MatSetSizes(A, 3, 3, 5, 5); > P1: MatSetSizes(A, 2, 2, 5, 5); > > See how different processes give different numbers to the routine? This is > what SPMD programming is all about. > > Thanks, > > Matt > > >> Step 4: Preallocate memory for the d_nnz and o_nnz values which are >> different for each row (for each n) >> >> MatMPIAIJSetPreallocation(A,0,d_nnz[n],0,o_nnz[n]); // How to do >> this repeatedly for all processes especially when using PETEC_DECIDE for n >> and m as shown in many examples in which case I don't even know the n per >> process >> >> I have to retrieve the relevant values (content and structure) from an >> application and I absolutely don't understand how to enter the exact same >> matrix structure as shown in the example (only much larger) into PETSc >> using the retrieved data? >> >> How would assign the d_nnz and o_nnz values of a global matrix on a 32 >> core workstation to the 32 processes using MatMPIAIJSetPreallocation(A,0,d_nnz[n],0,o_nnz[n]);? >> (Which means in my case assigning 64 arrays with different content >> containing the d_nnz and o_nnz values for the 32 processes) >> >> Would I use a for loop - but then how to get hold of the individual >> processes? Same for Step 3? >> >> I don't get the underlying idea about how to create a parallel matrix >> structure in PETSc - don't get me wrong I understand the layout "simply" >> not how to enter more than one local matrix it into PETSc. >> >> Klaus >> >> Am Freitag, 20. April 2018, 19:52:11 MESZ hat Matthew Knepley < >> knepley at gmail.com> Folgendes geschrieben: >> >> >> On Fri, Apr 20, 2018 at 1:30 PM, Klaus Burkart >> wrote: >> >> In my case N=M but n for process 0, 1, 2, 3,... no_processes-1 can be >> different from the nth process like in the example where the nth >> process=Proc2 and has only two rows while all other processes have three >> rows: >> >> >> Yes. >> >> >> Example from the PETSc webpage mentioned before: >> >> 1 2 0 | 0 3 0 | 0 4 >> Proc0 0 5 6 | 7 0 0 | 8 0 >> 9 0 10 | 11 0 0 | 12 0 >> ------------------------------ ------- >> 13 0 14 | 15 16 17 | 0 0 >> Proc1 0 18 0 | 19 20 21 | 0 0 >> 0 0 0 | 22 23 0 | 24 0 >> ------------------------------ ------- >> Proc2 25 26 27 | 0 0 28 | 29 0 >> 30 0 0 | 31 32 33 | 0 34 >> >> and I need to enter different values for d_nnz and o_nnz for each row >> somewhere too >> >> proc0: d_nnz = [2,2,2] and o_nnz = [2,2,2] >> proc1: d_nnz = [3,3,2] and o_nnz = [2,1,1] >> proc2: d_nnz = [1,1] and o_nnz = [4,4] >> >> Each process only sets d_nnz and o_nnz for its LOCAL rows. Thus, it is >> exactly as shown above. Proc1 sets values >> only for rows 3-5. >> >> Thanks, >> >> Matt >> >> >> I simply can't identify the function(s) used to set the values for n, >> d_nnz and o_nnz for the individual local matrices allocated to all the >> processes if n isn't the same for all processes and d_nnz and o_nnz are >> different for each local matrix? >> >> Approach described on the PETSc webpage: >> >> MatCreate (...,&A); >> >> >> >> MatSetType (A,MATMPIAIJ ); >> >> >> >> MatSetSizes (A, m,n,M,N); // for the example above using this function would set the no. of rows for Proc2 to 3 but it's 2 >> >> >> >> >> MatMPIAIJSetPreallocation (A,.. .); // this function can be used to set values for ONE local matrix only >> >> >> >> In addition to that I don't know which functions to use to preallocate >> memory for ALL local matrices when each of them has different values for >> d_nnz and o_nnz. >> >> I other words, what's the code for the 3 process example above? (entering >> the matrix structure and allocating memory) >> >> Klaus >> >> Am Freitag, 20. April 2018, 17:13:26 MESZ hat Smith, Barry F. < >> bsmith at mcs.anl.gov> Folgendes geschrieben: >> >> >> >> For square matrices almost always n is the same as m. On different >> processes m can be different. You get to decide what makes sense for each >> processes what its m should be. >> >> Barry >> >> >> > On Apr 20, 2018, at 10:05 AM, Klaus Burkart >> wrote: >> > >> > I think I understood the matrix structure for parallel computation with >> the rows, diagonal (d) and off-diagonal (o) structure, where I have >> problems is how to do the setup including memory allocation in PETSc: >> > >> > Lets assume, I use a 16 core workstation (=16 processes) and the number >> of nonzeros varies in each row for both d and o and the number of rows >> assigned to each process differs too - at least for the nth process. >> > >> > Looking at the manual and http://www.mcs.anl.gov/petsc/ >> petsc-current/docs/ manualpages/Mat/MatCreateAIJ. html#MatCreateAIJ, >> I >> don't understand how to enter a global matrix when n is NOT the same for >> each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not >> arrays? >> > >> > MatCreate(...,&A); >> > >> > MatSetType(A,MATMPIAIJ); >> > >> > MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same for >> each process which isn't even the case in the example on the page mentioned >> above?! >> > >> > MatMPIAIJSetPreallocation(A,.. .); >> > >> > >> > How can I enter the parallel global-local matrix structure? >> > >> > How can the memory preallocation be done? >> > >> > Klaus >> > >> > Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. < >> bsmith at mcs.anl.gov> Folgendes geschrieben: >> > >> > >> > >> > >> > > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote: >> > > >> > > So, practically speaking, l should invent routines to decompose the >> matrix e.g. into a block matrix structure to be able to make real use of >> PETSc ie. be able to solve a linear system using more than one process/core? >> > >> > To really use PETSc efficiently/effectively you need to generate your >> matrix in parallel. >> > >> > Barry >> > >> > > >> > > Klaus >> > > >> > > Von meinem Huawei-Mobiltelefon gesendet >> > > >> > > >> > > -------- Originalnachricht -------- >> > > Betreff: Re: [petsc-users] Matrix and vector type selection & memory >> allocation for efficient matrix import? >> > > Von: "Smith, Barry F." >> > > An: Klaus Burkart >> > > Cc: PETSc Users List >> > > >> > > >> > > >> > > If you can only generate the nonzero allocation sequentially you can >> only solve sequentially which means your matrix is MATSEQAIJ and your >> vector is VECSEQ and your communicator is PETSC_COMM_SELF. >> > > >> > > If you pass and array for nnz, what you pass for nz is irrelevant, >> you might as well pass 0. >> > > >> > > Barry >> > > >> > > >> > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: >> > > > >> > > > More questions about matrix and vector type selection for my >> application: >> > > > >> > > > My starting point is a huge sparse matrix which can be symmetric or >> asymmetric and a rhs vector. There's no defined local or block structure at >> all, just row and column indices and the values and an array style rhs >> vector together describing the entire linear system to be solved. With >> quite some effort, I should be able to create an array nnz[N] containing >> the number of nonzeros per row in the global matrix for memory allocation >> which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only >> option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I >> assume here, that 0 indicates different numbers of nonzero values in each >> row, the exact number being stored in the nnz array. Regarding this detail >> but one example assume a constant number of nz per row so I am not sure >> whether I should write 0 or NULL for nz? >> > > > >> > > > I started with: >> > > > >> > > > MatCreate(PETSC_COMM_WORLD, &M); >> > > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); >> > > > MatSetFromOptions(M); >> > > > >> > > > taken from a paper and assume, the latter would set the matrix type >> to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe >> decompositioning took place at an earlier stage and the authors of the >> paper were able to retrieve the local data and structure. >> > > > >> > > > What type of matrix and vector should I use for my application e.g. >> MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, >> nnz); for efficient memory allocation? >> > > > >> >> > > > In this case, where would the decompositioning / MPI process >> allocation take place? >> > > >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 20 15:34:01 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 20 Apr 2018 20:34:01 +0000 Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: <270008272.3263240.1524254870285@mail.yahoo.com> References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> <1215145994.3164869.1524236744544@mail.yahoo.com> <897258597.3227571.1524245454851@mail.yahoo.com> <270008272.3263240.1524254870285@mail.yahoo.com> Message-ID: > On Apr 20, 2018, at 3:07 PM, Klaus Burkart wrote: > > Sorry, I don't understand you: > > The example says: > > Step 1: Create the gobal matrix > > MatCreate(...,&A); // no problem > > Step 2: Make it a parallel matrix > > MatSetType(A,MATMPIAIJ); // no problem > > Step 3: Define the size of the global matrix and the number of rows per process IF this number is the same for all processes > > MatSetSizes(A, N,n,N,N); In the example, I have the problem with n which is 3 or 2 depending on the process but I can only set n to 3 or 2 so it will be the wrong for at least one process N,n,N,N doesn't make much sense. Likely you mean n,n,N,N. Different MPI processes have different values for n. > > Step 4: Preallocate memory for the d_nnz and o_nnz values which are different for each row (for each n) > > MatMPIAIJSetPreallocation(A,0,d_nnz[n],0,o_nnz[n]); // How to do this repeatedly for all processes especially when using PETEC_DECIDE for n and m as shown in many examples in which case I don't even know the n per process You cannot do preallocation if you use PETSC_DECIDE for m/n > > I have to retrieve the relevant values (content and structure) from an application and I absolutely don't understand how to enter the exact same matrix structure as shown in the example (only much larger) into PETSc using the retrieved data? > > How would assign the d_nnz and o_nnz values of a global matrix on a 32 core workstation to the 32 processes using MatMPIAIJSetPreallocation(A,0,d_nnz[n],0,o_nnz[n]);? (Which means in my case assigning 64 arrays with different content containing the d_nnz and o_nnz values for the 32 processes) Each process has its own array that contains the information just for that one process. > > Would I use a for loop - but then how to get hold of the individual processes? Same for Step 3? > > I don't get the underlying idea about how to create a parallel matrix structure in PETSc - don't get me wrong I understand the layout "simply" not how to enter more than one local matrix it into PETSc. > > Klaus > > Am Freitag, 20. April 2018, 19:52:11 MESZ hat Matthew Knepley Folgendes geschrieben: > > > On Fri, Apr 20, 2018 at 1:30 PM, Klaus Burkart wrote: > In my case N=M but n for process 0, 1, 2, 3,... no_processes-1 can be different from the nth process like in the example where the nth process=Proc2 and has only two rows while all other processes have three rows: > > Yes. > > Example from the PETSc webpage mentioned before: > 1 2 0 | 0 3 0 | 0 4 > Proc0 0 5 6 | 7 0 0 | 8 0 > 9 0 10 | 11 0 0 | 12 0 > ------------------------------ ------- > 13 0 14 | 15 16 17 | 0 0 > Proc1 0 18 0 | 19 20 21 | 0 0 > 0 0 0 | 22 23 0 | 24 0 > ------------------------------ ------- > Proc2 25 26 27 | 0 0 28 | 29 0 > 30 0 0 | 31 32 33 | 0 34 > > and I need to enter different values for d_nnz and o_nnz for each row somewhere too > proc0: d_nnz = [2,2,2] and o_nnz = [2,2,2] > proc1: d_nnz = [3,3,2] and o_nnz = [2,1,1] > proc2: d_nnz = [1,1] and o_nnz = [4,4] > > Each process only sets d_nnz and o_nnz for its LOCAL rows. Thus, it is exactly as shown above. Proc1 sets values > only for rows 3-5. > > Thanks, > > Matt > > I simply can't identify the function(s) used to set the values for n, d_nnz and o_nnz for the individual local matrices allocated to all the processes if n isn't the same for all processes and d_nnz and o_nnz are different for each local matrix? > > Approach described on the PETSc webpage: > MatCreate > (...,&A); > > > > MatSetType(A,MATMPIAIJ > ); > > > > MatSetSizes > (A, m,n,M,N); // for the example above using this function would set the no. of rows for Proc2 to 3 but it's 2 > > > > > MatMPIAIJSetPreallocation(A,.. .); // this function can be used to set values for ONE local matrix only > > > In addition to that I don't know which functions to use to preallocate memory for ALL local matrices when each of them has different values for d_nnz and o_nnz. > > I other words, what's the code for the 3 process example above? (entering the matrix structure and allocating memory) > > Klaus > > Am Freitag, 20. April 2018, 17:13:26 MESZ hat Smith, Barry F. Folgendes geschrieben: > > > > For square matrices almost always n is the same as m. On different processes m can be different. You get to decide what makes sense for each processes what its m should be. > > Barry > > > > On Apr 20, 2018, at 10:05 AM, Klaus Burkart wrote: > > > > I think I understood the matrix structure for parallel computation with the rows, diagonal (d) and off-diagonal (o) structure, where I have problems is how to do the setup including memory allocation in PETSc: > > > > Lets assume, I use a 16 core workstation (=16 processes) and the number of nonzeros varies in each row for both d and o and the number of rows assigned to each process differs too - at least for the nth process. > > > > Looking at the manual and http://www.mcs.anl.gov/petsc/ petsc-current/docs/ manualpages/Mat/MatCreateAIJ. html#MatCreateAIJ, I don't understand how to enter a global matrix when n is NOT the same for each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not arrays? > > > > MatCreate(...,&A); > > > > MatSetType(A,MATMPIAIJ); > > > > MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same for each process which isn't even the case in the example on the page mentioned above?! > > > > MatMPIAIJSetPreallocation(A,.. .); > > > > > > How can I enter the parallel global-local matrix structure? > > > > How can the memory preallocation be done? > > > > Klaus > > > > Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. Folgendes geschrieben: > > > > > > > > > > > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote: > > > > > > So, practically speaking, l should invent routines to decompose the matrix e.g. into a block matrix structure to be able to make real use of PETSc ie. be able to solve a linear system using more than one process/core? > > > > To really use PETSc efficiently/effectively you need to generate your matrix in parallel. > > > > Barry > > > > > > > > Klaus > > > > > > Von meinem Huawei-Mobiltelefon gesendet > > > > > > > > > -------- Originalnachricht -------- > > > Betreff: Re: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? > > > Von: "Smith, Barry F." > > > An: Klaus Burkart > > > Cc: PETSc Users List > > > > > > > > > > > > If you can only generate the nonzero allocation sequentially you can only solve sequentially which means your matrix is MATSEQAIJ and your vector is VECSEQ and your communicator is PETSC_COMM_SELF. > > > > > > If you pass and array for nnz, what you pass for nz is irrelevant, you might as well pass 0. > > > > > > Barry > > > > > > > > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: > > > > > > > > More questions about matrix and vector type selection for my application: > > > > > > > > My starting point is a huge sparse matrix which can be symmetric or asymmetric and a rhs vector. There's no defined local or block structure at all, just row and column indices and the values and an array style rhs vector together describing the entire linear system to be solved. With quite some effort, I should be able to create an array nnz[N] containing the number of nonzeros per row in the global matrix for memory allocation which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I assume here, that 0 indicates different numbers of nonzero values in each row, the exact number being stored in the nnz array. Regarding this detail but one example assume a constant number of nz per row so I am not sure whether I should write 0 or NULL for nz? > > > > > > > > I started with: > > > > > > > > MatCreate(PETSC_COMM_WORLD, &M); > > > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); > > > > MatSetFromOptions(M); > > > > > > > > taken from a paper and assume, the latter would set the matrix type to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe decompositioning took place at an earlier stage and the authors of the paper were able to retrieve the local data and structure. > > > > > > > > What type of matrix and vector should I use for my application e.g. MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, nnz); for efficient memory allocation? > > > > > > > > > In this case, where would the decompositioning / MPI process allocation take place? > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > From knepley at gmail.com Fri Apr 20 15:37:43 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 20 Apr 2018 16:37:43 -0400 Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> <1215145994.3164869.1524236744544@mail.yahoo.com> <897258597.3227571.1524245454851@mail.yahoo.com> <270008272.3263240.1524254870285@mail.yahoo.com> Message-ID: On Fri, Apr 20, 2018 at 4:28 PM, Zou, Ling wrote: > Mat, is this book you recommended? > > https://www.amazon.com/Using-MPI-Programming-Message- > Passing-Engineering/dp/0262527391/ref=pd_lpo_sbs_14_ > img_0?_encoding=UTF8&psc=1&refRID=EYCV0H0J5EQ9M0GDKWFT > Yep. I think its the best one, although Victor's book is also nice http://pages.tacc.utexas.edu/~eijkhout/istc/istc.html Matt > Thanks, > > Ling > > On Fri, Apr 20, 2018 at 2:17 PM, Matthew Knepley > wrote: > >> On Fri, Apr 20, 2018 at 4:07 PM, Klaus Burkart >> wrote: >> >>> Sorry, I don't understand you: >>> >>> The example says: >>> >>> Step 1: Create the gobal matrix >>> >>> MatCreate(...,&A); // no problem >>> >>> Step 2: Make it a parallel matrix >>> >>> MatSetType(A,MATMPIAIJ); // no problem >>> >>> Step 3: Define the size of the global matrix and the number of rows per >>> process IF this number is the same for all processes >>> >>> MatSetSizes(A, N,n,N,N); In the example, I have the problem with n >>> which is 3 or 2 depending on the process but I can only set n to 3 or 2 so >>> it will be the wrong for at least one process >>> >> >> 1) Get the book "Using MPI" just like I suggested. It explains this part >> of parallel programming that you do not understand. >> >> 2) Suppose we have two processes P0 and P1. Here are the call made on >> both processes for a matrix with 5 rows split 3, 2: >> >> P0: MatSetSizes(A, 3, 3, 5, 5); >> P1: MatSetSizes(A, 2, 2, 5, 5); >> >> See how different processes give different numbers to the routine? This >> is what SPMD programming is all about. >> >> Thanks, >> >> Matt >> >> >>> Step 4: Preallocate memory for the d_nnz and o_nnz values which are >>> different for each row (for each n) >>> >>> MatMPIAIJSetPreallocation(A,0,d_nnz[n],0,o_nnz[n]); // How to do >>> this repeatedly for all processes especially when using PETEC_DECIDE for n >>> and m as shown in many examples in which case I don't even know the n per >>> process >>> >>> I have to retrieve the relevant values (content and structure) from an >>> application and I absolutely don't understand how to enter the exact same >>> matrix structure as shown in the example (only much larger) into PETSc >>> using the retrieved data? >>> >>> How would assign the d_nnz and o_nnz values of a global matrix on a 32 >>> core workstation to the 32 processes using MatMPIAIJSetPreallocation(A,0,d_nnz[n],0,o_nnz[n]);? >>> (Which means in my case assigning 64 arrays with different content >>> containing the d_nnz and o_nnz values for the 32 processes) >>> >>> Would I use a for loop - but then how to get hold of the individual >>> processes? Same for Step 3? >>> >>> I don't get the underlying idea about how to create a parallel matrix >>> structure in PETSc - don't get me wrong I understand the layout "simply" >>> not how to enter more than one local matrix it into PETSc. >>> >>> Klaus >>> >>> Am Freitag, 20. April 2018, 19:52:11 MESZ hat Matthew Knepley < >>> knepley at gmail.com> Folgendes geschrieben: >>> >>> >>> On Fri, Apr 20, 2018 at 1:30 PM, Klaus Burkart >>> wrote: >>> >>> In my case N=M but n for process 0, 1, 2, 3,... no_processes-1 can be >>> different from the nth process like in the example where the nth >>> process=Proc2 and has only two rows while all other processes have three >>> rows: >>> >>> >>> Yes. >>> >>> >>> Example from the PETSc webpage mentioned before: >>> >>> 1 2 0 | 0 3 0 | 0 4 >>> Proc0 0 5 6 | 7 0 0 | 8 0 >>> 9 0 10 | 11 0 0 | 12 0 >>> ------------------------------ ------- >>> 13 0 14 | 15 16 17 | 0 0 >>> Proc1 0 18 0 | 19 20 21 | 0 0 >>> 0 0 0 | 22 23 0 | 24 0 >>> ------------------------------ ------- >>> Proc2 25 26 27 | 0 0 28 | 29 0 >>> 30 0 0 | 31 32 33 | 0 34 >>> >>> and I need to enter different values for d_nnz and o_nnz for each row >>> somewhere too >>> >>> proc0: d_nnz = [2,2,2] and o_nnz = [2,2,2] >>> proc1: d_nnz = [3,3,2] and o_nnz = [2,1,1] >>> proc2: d_nnz = [1,1] and o_nnz = [4,4] >>> >>> Each process only sets d_nnz and o_nnz for its LOCAL rows. Thus, it is >>> exactly as shown above. Proc1 sets values >>> only for rows 3-5. >>> >>> Thanks, >>> >>> Matt >>> >>> >>> I simply can't identify the function(s) used to set the values for n, >>> d_nnz and o_nnz for the individual local matrices allocated to all the >>> processes if n isn't the same for all processes and d_nnz and o_nnz are >>> different for each local matrix? >>> >>> Approach described on the PETSc webpage: >>> >>> MatCreate (...,&A); >>> >>> >>> >>> MatSetType (A,MATMPIAIJ ); >>> >>> >>> >>> MatSetSizes (A, m,n,M,N); // for the example above using this function would set the no. of rows for Proc2 to 3 but it's 2 >>> >>> >>> >>> >>> MatMPIAIJSetPreallocation (A,.. .); // this function can be used to set values for ONE local matrix only >>> >>> >>> >>> In addition to that I don't know which functions to use to preallocate >>> memory for ALL local matrices when each of them has different values for >>> d_nnz and o_nnz. >>> >>> I other words, what's the code for the 3 process example above? >>> (entering the matrix structure and allocating memory) >>> >>> Klaus >>> >>> Am Freitag, 20. April 2018, 17:13:26 MESZ hat Smith, Barry F. < >>> bsmith at mcs.anl.gov> Folgendes geschrieben: >>> >>> >>> >>> For square matrices almost always n is the same as m. On different >>> processes m can be different. You get to decide what makes sense for each >>> processes what its m should be. >>> >>> Barry >>> >>> >>> > On Apr 20, 2018, at 10:05 AM, Klaus Burkart >>> wrote: >>> > >>> > I think I understood the matrix structure for parallel computation >>> with the rows, diagonal (d) and off-diagonal (o) structure, where I have >>> problems is how to do the setup including memory allocation in PETSc: >>> > >>> > Lets assume, I use a 16 core workstation (=16 processes) and the >>> number of nonzeros varies in each row for both d and o and the number of >>> rows assigned to each process differs too - at least for the nth process. >>> > >>> > Looking at the manual and http://www.mcs.anl.gov/petsc/ >>> petsc-current/docs/ manualpages/Mat/MatCreateAIJ. html#MatCreateAIJ, >>> I >>> don't understand how to enter a global matrix when n is NOT the same for >>> each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not >>> arrays? >>> > >>> > MatCreate(...,&A); >>> > >>> > MatSetType(A,MATMPIAIJ); >>> > >>> > MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same >>> for each process which isn't even the case in the example on the page >>> mentioned above?! >>> > >>> > MatMPIAIJSetPreallocation(A,.. .); >>> > >>> > >>> > How can I enter the parallel global-local matrix structure? >>> > >>> > How can the memory preallocation be done? >>> > >>> > Klaus >>> > >>> > Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. < >>> bsmith at mcs.anl.gov> Folgendes geschrieben: >>> > >>> > >>> > >>> > >>> > > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote: >>> > > >>> > > So, practically speaking, l should invent routines to decompose the >>> matrix e.g. into a block matrix structure to be able to make real use of >>> PETSc ie. be able to solve a linear system using more than one process/core? >>> > >>> > To really use PETSc efficiently/effectively you need to generate your >>> matrix in parallel. >>> > >>> > Barry >>> > >>> > > >>> > > Klaus >>> > > >>> > > Von meinem Huawei-Mobiltelefon gesendet >>> > > >>> > > >>> > > -------- Originalnachricht -------- >>> > > Betreff: Re: [petsc-users] Matrix and vector type selection & memory >>> allocation for efficient matrix import? >>> > > Von: "Smith, Barry F." >>> > > An: Klaus Burkart >>> > > Cc: PETSc Users List >>> > > >>> > > >>> > > >>> > > If you can only generate the nonzero allocation sequentially you can >>> only solve sequentially which means your matrix is MATSEQAIJ and your >>> vector is VECSEQ and your communicator is PETSC_COMM_SELF. >>> > > >>> > > If you pass and array for nnz, what you pass for nz is irrelevant, >>> you might as well pass 0. >>> > > >>> > > Barry >>> > > >>> > > >>> > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: >>> > > > >>> > > > More questions about matrix and vector type selection for my >>> application: >>> > > > >>> > > > My starting point is a huge sparse matrix which can be symmetric >>> or asymmetric and a rhs vector. There's no defined local or block structure >>> at all, just row and column indices and the values and an array style rhs >>> vector together describing the entire linear system to be solved. With >>> quite some effort, I should be able to create an array nnz[N] containing >>> the number of nonzeros per row in the global matrix for memory allocation >>> which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only >>> option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I >>> assume here, that 0 indicates different numbers of nonzero values in each >>> row, the exact number being stored in the nnz array. Regarding this detail >>> but one example assume a constant number of nz per row so I am not sure >>> whether I should write 0 or NULL for nz? >>> > > > >>> > > > I started with: >>> > > > >>> > > > MatCreate(PETSC_COMM_WORLD, &M); >>> > > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); >>> > > > MatSetFromOptions(M); >>> > > > >>> > > > taken from a paper and assume, the latter would set the matrix >>> type to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe >>> decompositioning took place at an earlier stage and the authors of the >>> paper were able to retrieve the local data and structure. >>> > > > >>> > > > What type of matrix and vector should I use for my application >>> e.g. MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, >>> nnz); for efficient memory allocation? >>> > > > >>> >>> > > > In this case, where would the decompositioning / MPI process >>> allocation take place? >>> > > >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ling.zou at inl.gov Fri Apr 20 15:40:05 2018 From: ling.zou at inl.gov (Zou, Ling) Date: Fri, 20 Apr 2018 14:40:05 -0600 Subject: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import? In-Reply-To: References: <246102299.2103671.1524066532893.ref@mail.yahoo.com> <246102299.2103671.1524066532893@mail.yahoo.com> <5071855b-1da1-42ae-924b-ceefda2e3b45@LUCKMAN.anl.gov> <1215145994.3164869.1524236744544@mail.yahoo.com> <897258597.3227571.1524245454851@mail.yahoo.com> <270008272.3263240.1524254870285@mail.yahoo.com> Message-ID: Thanks, will take a look. Ling On Fri, Apr 20, 2018 at 2:37 PM, Matthew Knepley wrote: > On Fri, Apr 20, 2018 at 4:28 PM, Zou, Ling wrote: > >> Mat, is this book you recommended? >> >> https://www.amazon.com/Using-MPI-Programming-Message-Passing >> -Engineering/dp/0262527391/ref=pd_lpo_sbs_14_img_0?_ >> encoding=UTF8&psc=1&refRID=EYCV0H0J5EQ9M0GDKWFT >> >> > > Yep. I think its the best one, although Victor's book is also nice > > http://pages.tacc.utexas.edu/~eijkhout/istc/istc.html > > > Matt > > >> Thanks, >> >> Ling >> >> On Fri, Apr 20, 2018 at 2:17 PM, Matthew Knepley >> wrote: >> >>> On Fri, Apr 20, 2018 at 4:07 PM, Klaus Burkart >>> wrote: >>> >>>> Sorry, I don't understand you: >>>> >>>> The example says: >>>> >>>> Step 1: Create the gobal matrix >>>> >>>> MatCreate(...,&A); // no problem >>>> >>>> Step 2: Make it a parallel matrix >>>> >>>> MatSetType(A,MATMPIAIJ); // no problem >>>> >>>> Step 3: Define the size of the global matrix and the number of rows per >>>> process IF this number is the same for all processes >>>> >>>> MatSetSizes(A, N,n,N,N); In the example, I have the problem with n >>>> which is 3 or 2 depending on the process but I can only set n to 3 or 2 so >>>> it will be the wrong for at least one process >>>> >>> >>> 1) Get the book "Using MPI" just like I suggested. It explains this part >>> of parallel programming that you do not understand. >>> >>> 2) Suppose we have two processes P0 and P1. Here are the call made on >>> both processes for a matrix with 5 rows split 3, 2: >>> >>> P0: MatSetSizes(A, 3, 3, 5, 5); >>> P1: MatSetSizes(A, 2, 2, 5, 5); >>> >>> See how different processes give different numbers to the routine? This >>> is what SPMD programming is all about. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Step 4: Preallocate memory for the d_nnz and o_nnz values which are >>>> different for each row (for each n) >>>> >>>> MatMPIAIJSetPreallocation(A,0,d_nnz[n],0,o_nnz[n]); // How to do >>>> this repeatedly for all processes especially when using PETEC_DECIDE for n >>>> and m as shown in many examples in which case I don't even know the n per >>>> process >>>> >>>> I have to retrieve the relevant values (content and structure) from an >>>> application and I absolutely don't understand how to enter the exact same >>>> matrix structure as shown in the example (only much larger) into PETSc >>>> using the retrieved data? >>>> >>>> How would assign the d_nnz and o_nnz values of a global matrix on a 32 >>>> core workstation to the 32 processes using MatMPIAIJSetPreallocation(A,0,d_nnz[n],0,o_nnz[n]);? >>>> (Which means in my case assigning 64 arrays with different content >>>> containing the d_nnz and o_nnz values for the 32 processes) >>>> >>>> Would I use a for loop - but then how to get hold of the individual >>>> processes? Same for Step 3? >>>> >>>> I don't get the underlying idea about how to create a parallel matrix >>>> structure in PETSc - don't get me wrong I understand the layout "simply" >>>> not how to enter more than one local matrix it into PETSc. >>>> >>>> Klaus >>>> >>>> Am Freitag, 20. April 2018, 19:52:11 MESZ hat Matthew Knepley < >>>> knepley at gmail.com> Folgendes geschrieben: >>>> >>>> >>>> On Fri, Apr 20, 2018 at 1:30 PM, Klaus Burkart >>>> wrote: >>>> >>>> In my case N=M but n for process 0, 1, 2, 3,... no_processes-1 can be >>>> different from the nth process like in the example where the nth >>>> process=Proc2 and has only two rows while all other processes have three >>>> rows: >>>> >>>> >>>> Yes. >>>> >>>> >>>> Example from the PETSc webpage mentioned before: >>>> >>>> 1 2 0 | 0 3 0 | 0 4 >>>> Proc0 0 5 6 | 7 0 0 | 8 0 >>>> 9 0 10 | 11 0 0 | 12 0 >>>> ------------------------------ ------- >>>> 13 0 14 | 15 16 17 | 0 0 >>>> Proc1 0 18 0 | 19 20 21 | 0 0 >>>> 0 0 0 | 22 23 0 | 24 0 >>>> ------------------------------ ------- >>>> Proc2 25 26 27 | 0 0 28 | 29 0 >>>> 30 0 0 | 31 32 33 | 0 34 >>>> >>>> and I need to enter different values for d_nnz and o_nnz for each row >>>> somewhere too >>>> >>>> proc0: d_nnz = [2,2,2] and o_nnz = [2,2,2] >>>> proc1: d_nnz = [3,3,2] and o_nnz = [2,1,1] >>>> proc2: d_nnz = [1,1] and o_nnz = [4,4] >>>> >>>> Each process only sets d_nnz and o_nnz for its LOCAL rows. Thus, it is >>>> exactly as shown above. Proc1 sets values >>>> only for rows 3-5. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>> I simply can't identify the function(s) used to set the values for n, >>>> d_nnz and o_nnz for the individual local matrices allocated to all the >>>> processes if n isn't the same for all processes and d_nnz and o_nnz are >>>> different for each local matrix? >>>> >>>> Approach described on the PETSc webpage: >>>> >>>> MatCreate (...,&A); >>>> >>>> >>>> >>>> MatSetType (A,MATMPIAIJ ); >>>> >>>> >>>> >>>> MatSetSizes (A, m,n,M,N); // for the example above using this function would set the no. of rows for Proc2 to 3 but it's 2 >>>> >>>> >>>> >>>> >>>> MatMPIAIJSetPreallocation (A,.. .); // this function can be used to set values for ONE local matrix only >>>> >>>> >>>> >>>> In addition to that I don't know which functions to use to preallocate >>>> memory for ALL local matrices when each of them has different values for >>>> d_nnz and o_nnz. >>>> >>>> I other words, what's the code for the 3 process example above? >>>> (entering the matrix structure and allocating memory) >>>> >>>> Klaus >>>> >>>> Am Freitag, 20. April 2018, 17:13:26 MESZ hat Smith, Barry F. < >>>> bsmith at mcs.anl.gov> Folgendes geschrieben: >>>> >>>> >>>> >>>> For square matrices almost always n is the same as m. On different >>>> processes m can be different. You get to decide what makes sense for each >>>> processes what its m should be. >>>> >>>> Barry >>>> >>>> >>>> > On Apr 20, 2018, at 10:05 AM, Klaus Burkart >>>> wrote: >>>> > >>>> > I think I understood the matrix structure for parallel computation >>>> with the rows, diagonal (d) and off-diagonal (o) structure, where I have >>>> problems is how to do the setup including memory allocation in PETSc: >>>> > >>>> > Lets assume, I use a 16 core workstation (=16 processes) and the >>>> number of nonzeros varies in each row for both d and o and the number of >>>> rows assigned to each process differs too - at least for the nth process. >>>> > >>>> > Looking at the manual and http://www.mcs.anl.gov/petsc/ >>>> petsc-current/docs/ manualpages/Mat/MatCreateAIJ. html#MatCreateAIJ, >>>> I >>>> don't understand how to enter a global matrix when n is NOT the same for >>>> each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not >>>> arrays? >>>> > >>>> > MatCreate(...,&A); >>>> > >>>> > MatSetType(A,MATMPIAIJ); >>>> > >>>> > MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same >>>> for each process which isn't even the case in the example on the page >>>> mentioned above?! >>>> > >>>> > MatMPIAIJSetPreallocation(A,.. .); >>>> > >>>> > >>>> > How can I enter the parallel global-local matrix structure? >>>> > >>>> > How can the memory preallocation be done? >>>> > >>>> > Klaus >>>> > >>>> > Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. < >>>> bsmith at mcs.anl.gov> Folgendes geschrieben: >>>> > >>>> > >>>> > >>>> > >>>> > > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote: >>>> > > >>>> > > So, practically speaking, l should invent routines to decompose the >>>> matrix e.g. into a block matrix structure to be able to make real use of >>>> PETSc ie. be able to solve a linear system using more than one process/core? >>>> > >>>> > To really use PETSc efficiently/effectively you need to generate >>>> your matrix in parallel. >>>> > >>>> > Barry >>>> > >>>> > > >>>> > > Klaus >>>> > > >>>> > > Von meinem Huawei-Mobiltelefon gesendet >>>> > > >>>> > > >>>> > > -------- Originalnachricht -------- >>>> > > Betreff: Re: [petsc-users] Matrix and vector type selection & >>>> memory allocation for efficient matrix import? >>>> > > Von: "Smith, Barry F." >>>> > > An: Klaus Burkart >>>> > > Cc: PETSc Users List >>>> > > >>>> > > >>>> > > >>>> > > If you can only generate the nonzero allocation sequentially you >>>> can only solve sequentially which means your matrix is MATSEQAIJ and your >>>> vector is VECSEQ and your communicator is PETSC_COMM_SELF. >>>> > > >>>> > > If you pass and array for nnz, what you pass for nz is irrelevant, >>>> you might as well pass 0. >>>> > > >>>> > > Barry >>>> > > >>>> > > >>>> > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote: >>>> > > > >>>> > > > More questions about matrix and vector type selection for my >>>> application: >>>> > > > >>>> > > > My starting point is a huge sparse matrix which can be symmetric >>>> or asymmetric and a rhs vector. There's no defined local or block structure >>>> at all, just row and column indices and the values and an array style rhs >>>> vector together describing the entire linear system to be solved. With >>>> quite some effort, I should be able to create an array nnz[N] containing >>>> the number of nonzeros per row in the global matrix for memory allocation >>>> which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only >>>> option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I >>>> assume here, that 0 indicates different numbers of nonzero values in each >>>> row, the exact number being stored in the nnz array. Regarding this detail >>>> but one example assume a constant number of nz per row so I am not sure >>>> whether I should write 0 or NULL for nz? >>>> > > > >>>> > > > I started with: >>>> > > > >>>> > > > MatCreate(PETSC_COMM_WORLD, &M); >>>> > > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N); >>>> > > > MatSetFromOptions(M); >>>> > > > >>>> > > > taken from a paper and assume, the latter would set the matrix >>>> type to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe >>>> decompositioning took place at an earlier stage and the authors of the >>>> paper were able to retrieve the local data and structure. >>>> > > > >>>> > > > What type of matrix and vector should I use for my application >>>> e.g. MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, >>>> nnz); for efficient memory allocation? >>>> > > > >>>> >>>> > > > In this case, where would the decompositioning / MPI process >>>> allocation take place? >>>> > > >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Fri Apr 20 16:08:05 2018 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Fri, 20 Apr 2018 16:08:05 -0500 Subject: [petsc-users] Fwd: How to adapt vectors to matrices loaded from a viewer In-Reply-To: References: Message-ID: (resend from my gmail) In a PETSc example (ex10.c ) one MatLoad A and then VecLoad b from a viewer. Since one can change mat block size through option -matload_block_size, the code tries to pad b if A and b's sizes mismatch, using the following test 186: MatGetSize(A,&M,NULL); 187: VecGetSize(b,&m); 188: if (M != m) { /* Create a new vector b by padding the old one */ I think the code is wrong. One has to compare local column size of A and local size of b, and if there is a mismatch on ANY processor, then one has to create a new b. My questions are: Are users supposed to take this complexity? Does PETSc provide a neat way to do that, for example, MatVecLoad(A,b,viewer) Thanks. --Junchao Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sat Apr 21 10:34:22 2018 From: zonexo at gmail.com (TAY wee-beng) Date: Sat, 21 Apr 2018 23:34:22 +0800 Subject: [petsc-users] Scaling problem when cores > 600 In-Reply-To: <9346FFA5-9385-4EDD-9C4D-2232CCB3F9F9@mcs.anl.gov> References: <26ed65df-90d0-1f27-5d66-b6488bef3b90@gmail.com> <5a76371a-0eb1-f204-4974-549865ce726d@gmail.com> <35ace3a0-1e35-1e66-6482-98fe2483f24e@gmail.com> <16EEAEE0-BF68-4DF9-B0BE-8DF5D25ECF63@mcs.anl.gov> <4043d81a-8bbf-03be-c536-ae28d6ebefb2@gmail.com> <9346FFA5-9385-4EDD-9C4D-2232CCB3F9F9@mcs.anl.gov> Message-ID: Hi, I have found some time to work on this scaling problem again. I am now using: mpirun ./a.out -log_view -poisson_pc_type gamg -poisson_pc_gamg_agg_nsmooths 1 I have attached the log_view output for 288, 600, 960, 1440 procs for comparison. Please give some comments. Thank you very much Yours sincerely, ================================================ TAY Wee-Beng ??? (Zheng Weiming) Personal research webpage: http://tayweebeng.wixsite.com/website Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA linkedin: www.linkedin.com/in/tay-weebeng ================================================ On 7/3/2018 11:58 PM, Smith, Barry F. wrote: > What are you using for Poisson log. > > If it is a Poisson problem then almost for sure you should be using Hypre BoomerAMG?. > > It sounds like your matrix does not change. You will need to discuss the scaling with the hypre people. > > Barry > > >> On Mar 7, 2018, at 5:38 AM, TAY wee-beng wrote: >> >> >> On 7/3/2018 6:22 AM, Smith, Barry F. wrote: >>> The speed up for "Poisson log" is 1.6425364214878704 = 5.0848e+02/3.0957e+02 >>> >>> This is lower than I would expect for Hypre BoomerAMG? >>> >>> Are you doing multiple solves with the same matrix with hypre or is each solve a new matrix? If each solve is a new matrix then you may be getting expected behavior since the multigrid AMG construction process does not scale as well as the application of AMG once it is constructed. >>> >>> I am forwarding to the hypre team since this is their expertise not ours >>> >>> Barry >>> >> Hi, >> >> My LHS of the eqn does not change. Only the RHS changes at each time step. So should this be expected? >> >> So maybe I should change to BoomerAMG and compare? >> >> Will PETSc GAMG give better performance? >> >> Also, I must add that I only partition in the x and y direction. Will this be a factor? >> >> Thanks. >> >>>> On Mar 5, 2018, at 11:19 PM, TAY wee-beng wrote: >>>> >>>> >>>> On 5/3/2018 11:43 AM, Smith, Barry F. wrote: >>>>> 360 process >>>>> >>>>> KSPSolve 99 1.0 2.6403e+02 1.0 6.67e+10 1.1 2.7e+05 9.9e+05 5.1e+02 15100 17 42 19 15100 17 42 19 87401 >>>>> >>>>> 1920 processes >>>>> >>>>> KSPSolve 99 1.0 2.3184e+01 1.0 1.32e+10 1.2 1.5e+06 4.3e+05 5.1e+02 4100 17 42 19 4100 17 42 19 967717 >>>>> >>>>> >>>>> Ratio of number of processes 5.33 ratio of time for KSPSolve 11.388 so the time for the solve is scaling very well (extremely well actually). The problem is >>>>> due to "other time" that is not in KSP solve. Note that the percentage of the total time in KSPSolve went from 15 percent of the runtime to 4 percent. This means something outside of KSPSolve is scaling very poorly. You will need to profile the rest of the code to determine where the time is being spent. PetscLogEventRegister() and PetscLogEventBegin/End() will be needed in your code. Already with 360 processes the linear solver is only taking 15 percent of the time. >>>>> >>>>> Barry >>>>> >>>> Hi, >>>> >>>> I have attached the new logging results with the HYPRE Poisson eqn solver. However, due to some problems, I am now using Intel 2018. Should be quite similar to 2016 in terms of runtime. Using 360 processes can't work this time, and I'm not sure why though. >>>>>> On Mar 4, 2018, at 9:23 PM, TAY wee-beng wrote: >>>>>> >>>>>> >>>>>> On 1/3/2018 12:14 PM, Smith, Barry F. wrote: >>>>>>>> On Feb 28, 2018, at 8:01 PM, TAY wee-beng wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 1/3/2018 12:10 AM, Matthew Knepley wrote: >>>>>>>>> On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I have a CFD code which uses PETSc and HYPRE. I found that for a certain case with grid size of 192,570,048, I encounter scaling problem when my cores > 600. At 600 cores, the code took 10min for 100 time steps. At 960, 1440 and 2880 cores, it still takes around 10min. At 360 cores, it took 15min. >>>>>>>>> >>>>>>>>> So how can I find the bottleneck? Any recommended steps? >>>>>>>>> >>>>>>>>> For any performance question, we need to see the output of -log_view for all test cases. >>>>>>>> Hi, >>>>>>>> >>>>>>>> To be more specific, I use PETSc KSPBCGS and HYPRE geometric multigrid (entirely based on HYPRE, no PETSc) for the momentum and Poisson eqns in my code. >>>>>>>> >>>>>>>> So can log_view be used in this case to give a meaningful? Since part of the code uses HYPRE? >>>>>>> Yes, just send the logs. >>>>>>> >>>>>> Hi, >>>>>> >>>>>> I have attached the logs, with the number indicating the no. of cores used. Some of the new results are different from the previous runs, although I'm using the same cluster. >>>>>> >>>>>> Thanks for the help. >>>>>>>> I also program another subroutine in the past which uses PETSc to solve the Poisson eqn. It uses either HYPRE's boomeramg, KSPBCGS or KSPGMRES. >>>>>>>> >>>>>>>> If I use boomeramg, can log_view be used in this case? >>>>>>>> >>>>>>>> Or do I have to use KSPBCGS or KSPGMRES, which is directly from PETSc? However, I ran KSPGMRES yesterday with the Poisson eqn and my ans didn't converge. >>>>>>>> >>>>>>>> Thanks. >>>>>>>>> I must also mention that I partition my grid only in the x and y direction. There is no partitioning in the z direction due to limited code development. I wonder if there is a strong effect in this case. >>>>>>>>> >>>>>>>>> Maybe. Usually what happens is you fill up memory with a z-column and cannot scale further. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> -- >>>>>>>>> Thank you very much >>>>>>>>> >>>>>>>>> Yours sincerely, >>>>>>>>> >>>>>>>>> ================================================ >>>>>>>>> TAY Wee-Beng ??? (Zheng Weiming) >>>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>>>>> ================================================ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>> -------------- next part -------------- 0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.400000000000000 0 -400000 AB,AA,BB,CC -3.30400006193668 3.62600007723086 3.22400005813688 3.22400005813688 size_x,size_y,size_z 718x718x285 total grid size = 146924340 recommended cores (50k / core) = 2938.48680000000 min_area,max_area,min_grid_area,ratio 9.229464518072358E-005 0.108728370237650 4.000000000000000E-004 271.820925594124 ratio bet max_area,min_grid_area not ideal max element length should be 2.828427124746190E-002 body_cg_ini -2.750916668409921E-003 2.695116216584833E-004 6.75873529067005 Warning - length difference between element and cell max_element_length,min_element_length,min_delta 0.558213960965656 4.851715434749100E-003 2.000000000000000E-002 maximum ngh_surfaces and ngh_vertics are 4 3 minimum ngh_surfaces and ngh_vertics are 1 1 min IIB_cell_no 0 max IIB_cell_no 435 IIB_cell_no_sum 71637 min equal_size 1560 max equal_size 1820 min I_cell_no 0 max I_cell_no 2310 I_cell_no_sum 264465 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz e(IIB_global_cell_u),size(I_global_cell_u) 435 2310 435 2310 71637 264465 IIB_equal_cell_no_u1_max 435 I_equal_cell_no_u1_max 2310 IIB_I_cell_no_uvw_total1 71637 0 0 264465 0 0 size(IIB_cell_u),IIB_cell_no_max_cur 435 17201 local IIB_cells size exceed, to increase size size(I_cell_u),I_cell_no_max_cur 2310 84869 local I_cells size exceed, to increase size IIB_cell_no_u1_max,I_cell_no_u1_max 25801 127303 size(IIB_cell_u),IIB_cell_no_max_cur 435 845 IIB_cells size exceed, to increase size IIB_equal_cell_no_u1_max 1267 size(I_cell_u),I_cell_no_max_cur 2310 5060 I_cells size exceed, to increase size I_equal_cell_no_u1_max 7590 size(I_global_cell_u1),I_global_cell_no_u1_max_cur 264465 264586 I global cells size exceed, to increase size IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 25801 127303 time,IIB_I_cell_no_uvw_total1 2 71589 0 0 264586 0 0 IIB_equal_cell_no_u1_max 1267 size(I_cell_u),I_cell_no_max_cur 7590 7728 I_cells size exceed, to increase size I_equal_cell_no_u1_max 11592 IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 25801 127303 time,IIB_I_cell_no_uvw_total1 3 71237 0 0 264577 0 0 IIB_equal_cell_no_u1_max 1267 I_equal_cell_no_u1_max 11592 IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 25801 127303 time,IIB_I_cell_no_uvw_total1 4 70676 0 0 264504 0 0 IIB_equal_cell_no_u1_max 1267 I_equal_cell_no_u1_max 11592 IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 25801 127303 time,IIB_I_cell_no_uvw_total1 5 70909 0 0 264557 0 0 escape_time reached, so abort Recommended no for final_IIB_equal_no1,final_IIB_no1,final_IIB_global_no1,final _I_equal_no1,final_I_no1,final_I_global_no1 1583 32251 89546 14490 159128 496098 0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.400000000000000 0 -400000 AB,AA,BB,CC -3.30400006193668 3.62600007723086 3.22400005813688 3.22400005813688 size_x,size_y,size_z 718x718x285 total grid size = 146924340 recommended cores (50k / core) = 2938.48680000000 min_area,max_area,min_grid_area,ratio 9.229464518072358E-005 0.108728370237650 4.000000000000000E-004 271.820925594124 ratio bet max_area,min_grid_area not ideal max element length should be 2.828427124746190E-002 body_cg_ini -2.750916668409921E-003 2.695116216584833E-004 6.75873529067005 Warning - length difference between element and cell max_element_length,min_element_length,min_delta 0.558213960965656 4.851715434749100E-003 2.000000000000000E-002 maximum ngh_surfaces and ngh_vertics are 4 3 minimum ngh_surfaces and ngh_vertics are 1 1 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz e(IIB_global_cell_u),size(I_global_cell_u) 32251 159128 1583 14490 89546 496098 IIB_I_cell_no_uvw_total1 71637 71704 71732 264465 264616 264595 1 0.00150000 0.00000000 0.00000000 0.00000000 0.00000000E+00 0.00000000E+00 0.00000000E+00 escape_time reached, so abort cd_cl_cs_mom_implicit1 -0.203649100207704 -3.78579022968484 -142.426095309958 3.49125655195658 -18.5255013193465 194.110055674799 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a named std0771 with 600 processors, by tsltaywb Sat Apr 21 00:23:03 2018 Using Petsc Release Version 3.9.0, Apr, 07, 2018 Max Max/Min Avg Total Time (sec): 1.293e+02 1.00002 1.293e+02 Objects: 8.450e+02 1.00118 8.440e+02 Flop: 2.402e+10 1.16509 2.255e+10 1.353e+13 Flop/sec: 1.858e+08 1.16510 1.744e+08 1.046e+11 MPI Messages: 1.668e+05 3.35829 1.236e+05 7.419e+07 MPI Message Lengths: 1.466e+09 2.16566 1.103e+04 8.180e+11 MPI Reductions: 1.451e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.2932e+02 100.0% 1.3533e+13 100.0% 7.419e+07 100.0% 1.103e+04 100.0% 1.442e+03 99.4% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSided 5 1.0 1.1753e-02 5.7 0.00e+00 0.0 2.9e+04 4.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 BuildTwoSidedF 67 1.0 1.6764e+00 4.1 0.00e+00 0.0 1.3e+05 1.4e+05 0.0e+00 1 0 0 2 0 1 0 0 2 0 0 VecDot 8 1.0 1.9643e-01 3.4 1.24e+07 1.1 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 1 0 0 0 0 1 35902 VecDotNorm2 4 1.0 1.4356e-0119.8 1.24e+07 1.1 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 49124 VecMDot 100 1.0 3.0183e-01 2.7 6.25e+07 1.1 0.0e+00 0.0e+00 1.0e+02 0 0 0 0 7 0 0 0 0 7 117627 VecTDot 248 1.0 1.0376e+00 2.0 1.28e+08 1.1 0.0e+00 0.0e+00 2.5e+02 1 1 0 0 17 1 1 0 0 17 70231 VecNorm 247 1.0 4.7946e+00 8.8 9.16e+07 1.1 0.0e+00 0.0e+00 2.5e+02 1 0 0 0 17 1 0 0 0 17 10858 VecScale 110 1.0 6.2954e-03 2.2 6.25e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 563963 VecCopy 672 1.0 2.3515e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 2431 1.0 1.4385e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 258 1.0 2.8743e-01 1.2 1.29e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 255784 VecAYPX 5280 1.0 1.2274e+00 2.0 4.28e+08 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 198344 VecAXPBYCZ 2588 1.0 7.0963e-01 1.5 7.58e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 606609 VecWAXPY 8 1.0 4.5665e-02 1.4 1.24e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 154437 VecMAXPY 110 1.0 9.0745e-02 1.2 7.38e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 462386 VecAssemblyBegin 34 1.0 2.7867e-01 3.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 34 1.0 6.3419e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 55 1.0 1.9400e-02 3.0 3.12e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 91507 VecScatterBegin 5448 1.0 1.1685e+00 2.6 0.00e+00 0.0 7.0e+07 1.1e+04 0.0e+00 1 0 94 94 0 1 0 94 94 0 0 VecScatterEnd 5448 1.0 1.1562e+01 3.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 0 VecSetRandom 5 1.0 7.0169e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 110 1.0 1.0416e-01 2.0 1.87e+07 1.1 0.0e+00 0.0e+00 1.1e+02 0 0 0 0 8 0 0 0 0 8 102254 MatMult 4102 1.0 2.2797e+01 1.2 1.15e+10 1.2 6.2e+07 1.1e+04 0.0e+00 16 48 84 86 0 16 48 84 86 0 283196 MatMultAdd 645 1.0 2.0686e+00 1.9 5.99e+08 1.2 3.8e+06 3.7e+03 0.0e+00 1 2 5 2 0 1 2 5 2 0 163213 MatMultTranspose 645 1.0 2.3167e+00 2.2 5.99e+08 1.2 3.8e+06 3.7e+03 0.0e+00 1 2 5 2 0 1 2 5 2 0 145734 MatSolve 14111.8 9.9322e-01 1.2 6.80e+08 1.1 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 388324 MatSOR 3925 1.0 2.3818e+01 1.4 8.20e+09 1.1 0.0e+00 0.0e+00 0.0e+00 17 34 0 0 0 17 34 0 0 0 194060 MatLUFactorSym 1 1.0 3.9697e-0457.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 2 1.0 5.6432e-01 1.4 2.52e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 252133 MatILUFactorSym 1 1.0 2.5929e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatConvert 5 1.0 6.1710e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 16 1.0 1.8764e-01 1.3 4.80e+07 1.2 7.8e+04 1.0e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 144625 MatResidual 645 1.0 3.9151e+00 1.5 1.69e+09 1.2 1.0e+07 1.0e+04 0.0e+00 2 7 14 13 0 2 7 14 13 0 243152 MatAssemblyBegin 102 1.0 1.4931e+00 2.5 0.00e+00 0.0 1.3e+05 1.4e+05 0.0e+00 1 0 0 2 0 1 0 0 2 0 0 MatAssemblyEnd 102 1.0 6.1704e+00 1.0 0.00e+00 0.0 6.2e+05 2.7e+03 2.2e+02 5 0 1 0 15 5 0 1 0 15 0 MatGetRow 852087 1.1 1.0557e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 2 2.0 3.6955e-0538.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCreateSubMat 4 1.0 1.3800e+00 1.0 0.00e+00 0.0 1.3e+04 1.6e+02 6.4e+01 1 0 0 0 4 1 0 0 0 4 0 MatGetOrdering 2 2.0 1.5060e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 5 1.0 1.3929e-01 1.1 0.00e+00 0.0 2.6e+06 1.3e+03 1.0e+02 0 0 4 0 7 0 0 4 0 7 0 MatZeroEntries 5 1.0 8.9250e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAXPY 5 1.0 7.8287e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 5 1.0 2.4963e+00 1.0 1.31e+07 1.2 4.6e+05 5.3e+03 6.2e+01 2 0 1 0 4 2 0 1 0 4 2956 MatMatMultSym 5 1.0 2.3848e+00 1.0 0.00e+00 0.0 3.8e+05 4.3e+03 6.0e+01 2 0 1 0 4 2 0 1 0 4 0 MatMatMultNum 5 1.0 1.0086e-01 1.0 1.31e+07 1.2 7.8e+04 1.0e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 73170 MatPtAP 5 1.0 6.8432e+00 1.0 4.11e+08 1.5 8.6e+05 4.6e+04 7.7e+01 5 2 1 5 5 5 2 1 5 5 33013 MatPtAPSymbolic 5 1.0 1.8684e+00 1.0 0.00e+00 0.0 4.5e+05 4.6e+04 3.5e+01 1 0 1 3 2 1 0 1 3 2 0 MatPtAPNumeric 5 1.0 4.9728e+00 1.0 4.11e+08 1.5 4.1e+05 4.6e+04 4.0e+01 4 2 1 2 3 4 2 1 2 3 45430 MatTrnMatMult 1 1.0 9.3095e-01 1.0 2.61e+07 1.2 1.8e+04 1.7e+05 1.6e+01 1 0 0 0 1 1 0 0 0 1 15887 MatTrnMatMultSym 1 1.0 6.6997e-01 1.0 0.00e+00 0.0 1.6e+04 8.7e+04 1.6e+01 1 0 0 0 1 1 0 0 0 1 0 MatTrnMatMultNum 1 1.0 2.6105e-01 1.0 2.61e+07 1.2 2.3e+03 7.7e+05 0.0e+00 0 0 0 0 0 0 0 0 0 0 56658 MatGetLocalMat 17 1.0 7.8735e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 15 1.0 1.2452e-01 1.8 0.00e+00 0.0 5.5e+05 3.3e+04 0.0e+00 0 0 1 2 0 0 0 1 2 0 0 SFSetGraph 5 1.0 4.0531e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFSetUp 5 1.0 1.6538e-02 1.9 0.00e+00 0.0 8.6e+04 4.0e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFBcastBegin 114 1.0 1.0221e-02 4.0 0.00e+00 0.0 2.5e+06 1.2e+03 0.0e+00 0 0 3 0 0 0 0 3 0 0 0 SFBcastEnd 114 1.0 3.8953e-02 5.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 100 1.0 3.6761e-01 2.0 1.25e+08 1.1 0.0e+00 0.0e+00 1.0e+02 0 1 0 0 7 0 1 0 0 7 193158 KSPSetUp 20 1.0 4.3537e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.8e+01 0 0 0 0 2 0 0 0 0 2 0 KSPSolve 9 1.0 6.5676e+01 1.0 2.40e+10 1.2 7.4e+07 1.1e+04 1.2e+03 51100100 96 83 51100100 96 83 205797 PCGAMGGraph_AGG 5 1.0 6.6401e-01 1.0 1.31e+07 1.2 1.9e+05 5.4e+03 6.0e+01 1 0 0 0 4 1 0 0 0 4 11114 PCGAMGCoarse_AGG 5 1.0 1.1636e+00 1.0 2.61e+07 1.2 2.7e+06 3.3e+03 1.2e+02 1 0 4 1 9 1 0 4 1 9 12711 PCGAMGProl_AGG 5 1.0 4.3104e-01 1.0 0.00e+00 0.0 1.7e+05 9.3e+03 8.0e+01 0 0 0 0 6 0 0 0 0 6 0 PCGAMGPOpt_AGG 5 1.0 3.0145e+00 1.0 2.28e+08 1.2 1.2e+06 8.4e+03 2.1e+02 2 1 2 1 14 2 1 2 1 14 42573 GAMG: createProl 5 1.0 5.2617e+00 1.0 2.67e+08 1.2 4.3e+06 5.1e+03 4.7e+02 4 1 6 3 32 4 1 6 3 33 28604 Graph 10 1.0 6.5672e-01 1.0 1.31e+07 1.2 1.9e+05 5.4e+03 6.0e+01 1 0 0 0 4 1 0 0 0 4 11237 MIS/Agg 5 1.0 1.3933e-01 1.1 0.00e+00 0.0 2.6e+06 1.3e+03 1.0e+02 0 0 4 0 7 0 0 4 0 7 0 SA: col data 5 1.0 8.9589e-02 1.1 0.00e+00 0.0 1.1e+05 1.2e+04 2.0e+01 0 0 0 0 1 0 0 0 0 1 0 SA: frmProl0 5 1.0 3.3868e-01 1.0 0.00e+00 0.0 5.2e+04 3.5e+03 4.0e+01 0 0 0 0 3 0 0 0 0 3 0 SA: smooth 5 1.0 2.6009e+00 1.0 1.78e+07 1.2 4.6e+05 5.3e+03 7.2e+01 2 0 1 0 5 2 0 1 0 5 3844 GAMG: partLevel 5 1.0 8.3094e+00 1.0 4.11e+08 1.5 8.8e+05 4.5e+04 1.8e+02 6 2 1 5 12 6 2 1 5 12 27188 repartition 3 1.0 2.4631e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 1 0 0 0 0 1 0 Invert-Sort 2 1.0 7.8353e-0291.3 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 1 0 0 0 0 1 0 Move A 2 1.0 1.1592e-02 1.2 0.00e+00 0.0 2.0e+03 8.5e+02 3.4e+01 0 0 0 0 2 0 0 0 0 2 0 Move P 2 1.0 1.3714e+00 1.0 0.00e+00 0.0 1.1e+04 2.8e+01 3.4e+01 1 0 0 0 2 1 0 0 0 2 0 PCSetUp 4 1.0 1.4434e+01 1.0 9.30e+08 1.3 5.1e+06 1.2e+04 6.8e+02 11 4 7 7 47 11 4 7 7 47 35935 PCSetUpOnBlocks 133 1.0 8.3417e-01 1.3 2.52e+08 1.2 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 170569 PCApply 141 1.0 4.7204e+01 1.1 2.13e+10 1.2 6.9e+07 9.5e+03 1.0e+02 36 89 92 80 7 36 89 92 80 7 254450 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 363 363 223074648 0. Matrix 150 150 1237480116 0. Matrix Coarsen 5 5 3180 0. Distributed Mesh 18 18 94176 0. Index Set 128 128 23259280 0. IS L to G Mapping 18 18 10397096 0. Star Forest Graph 41 41 35424 0. Discrete System 18 18 16632 0. Vec Scatter 58 58 17003440 0. Krylov Solver 20 20 314880 0. Preconditioner 15 15 15212 0. PetscRandom 10 10 6460 0. Viewer 1 0 0 0. ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 1.64032e-05 Average time for zero size MPI_Send(): 5.4582e-06 #PETSc Option Table entries: -log_view -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-mpi-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64 --with-blaslapack-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 --download-hypre=/home/users/nus/tsltaywb/scratch/source/v2.14.0.tar.gz --download-ml=/home/users/nus/tsltaywb/scratch/source/ml-v6.2-p4.tar.gz --with-debugging=0 --prefix=/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel --with-shared-libraries=0 --known-mpi-shared-libraries=0 --with-fortran-interfaces=1 --CFLAGS="-xHost -g -O2" --CXXFLAGS="-xHost -g -O2" --FFLAGS="-xHost -g -O2" ----------------------------------------- Libraries compiled on 2018-04-17 05:50:35 on nus02 Machine characteristics: Linux-2.6.32-696.18.7.el6.x86_64-x86_64-with-redhat-6.9-Santiago Using PETSc directory: /home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel Using PETSc arch: ----------------------------------------- Using C compiler: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpicc -xHost -g -O2 Using Fortran compiler: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 -xHost -g -O2 ----------------------------------------- Using include paths: -I/home/users/nus/tsltaywb/lib//include -I/home/users/nus/tsltaywb/lib///include -I/home/users/nus/tsltaywb/lib//include -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include ----------------------------------------- Using C linker: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpicc Using Fortran linker: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 Using libraries: -Wl,-rpath,/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -L/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -lpetsc -Wl,-rpath,/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -L/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -Wl,-rpath,/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib/debug_mt -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/ippcp/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/ipp/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64_lin -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64/gcc4.4 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/daal/lib/intel64_lin -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64_lin/gcc4.4 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib -lHYPRE -lml -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lX11 -lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lstdc++ -ldl ----------------------------------------- ====================================================================================== Resource Usage on 2018-04-21 00:23:08.128316: JobId: 6808395.wlm01 Project: 11000324 Exit Status: 0 NCPUs Requested: 600 NCPUs Used: 600 CPU Time Used: 00:50:53 Memory Requested: 2400gb Memory Used: 53653576kb Vmem Used: 75059520kb Walltime requested: 00:05:00 Walltime Used: 00:02:25 Execution Nodes Used: (std0771:ncpus=24:mem=100663296kb)+(std0772:ncpus=24:mem=100663296kb)+(std0866:ncpus=24:mem=100663296kb)+(std0867:ncpus=24:mem=100663296kb)+(std0871:ncpus=24:mem=100663296kb)+(std0872:ncpus=24:mem=100663296kb)+(std0901:ncpus=24:mem=100663296kb)+(std0902:ncpus=24:mem=100663296kb)+(std0903:ncpus=24:mem=100663296kb)+(std0904:ncpus=24:mem=100663296kb)+(std0905:ncpus=24:mem=100663296kb)+(std0906:ncpus=24:mem=100663296kb)+(std0907:ncpus=24:mem=100663296kb)+(std0908:ncpus=24:mem=100663296kb)+(std0909:ncpus=24:mem=100663296kb)+(std0910:ncpus=24:mem=100663296kb)+(std0911:ncpus=24:mem=100663296kb)+(std0912:ncpus=24:mem=100663296kb)+(std0913:ncpus=24:mem=100663296kb)+(std0914:ncpus=24:mem=100663296kb)+(std0915:ncpus=24:mem=100663296kb)+(std0916:ncpus=24:mem=100663296kb)+(std0917:ncpus=24:mem=100663296kb)+(std0918:ncpus=24:mem=100663296kb)+(std0919:ncpus=24:mem=100663296kb) ====================================================================================== -------------- next part -------------- 0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.400000000000000 0 -400000 AB,AA,BB,CC -3.30400006193668 3.62600007723086 3.22400005813688 3.22400005813688 size_x,size_y,size_z 718x718x285 total grid size = 146924340 recommended cores (50k / core) = 2938.48680000000 min_area,max_area,min_grid_area,ratio 9.229464518072358E-005 0.108728370237650 4.000000000000000E-004 271.820925594124 ratio bet max_area,min_grid_area not ideal max element length should be 2.828427124746190E-002 body_cg_ini -2.750916668409921E-003 2.695116216584833E-004 6.75873529067005 Warning - length difference between element and cell max_element_length,min_element_length,min_delta 0.558213960965656 4.851715434749100E-003 2.000000000000000E-002 maximum ngh_surfaces and ngh_vertics are 4 4 minimum ngh_surfaces and ngh_vertics are 1 1 min IIB_cell_no 0 max IIB_cell_no 592 IIB_cell_no_sum 71637 min equal_size 3600 max equal_size 4096 min I_cell_no 0 max I_cell_no 4352 I_cell_no_sum 264465 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz e(IIB_global_cell_u),size(I_global_cell_u) 592 4352 592 4352 71637 264465 IIB_equal_cell_no_u1_max 592 I_equal_cell_no_u1_max 4352 IIB_I_cell_no_uvw_total1 71637 0 0 264465 0 0 size(IIB_cell_u),IIB_cell_no_max_cur 592 18100 local IIB_cells size exceed, to increase size size(I_cell_u),I_cell_no_max_cur 4352 70137 local I_cells size exceed, to increase size IIB_cell_no_u1_max,I_cell_no_u1_max 27150 105205 size(IIB_cell_u),IIB_cell_no_max_cur 592 1296 IIB_cells size exceed, to increase size IIB_equal_cell_no_u1_max 1944 size(I_cell_u),I_cell_no_max_cur 4352 9324 I_cells size exceed, to increase size I_equal_cell_no_u1_max 13986 size(I_global_cell_u1),I_global_cell_no_u1_max_cur 264465 264586 I global cells size exceed, to increase size IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 27150 105205 time,IIB_I_cell_no_uvw_total1 2 71589 0 0 264586 0 0 size(IIB_cell_u),IIB_cell_no_max_cur 1944 2004 IIB_cells size exceed, to increase size IIB_equal_cell_no_u1_max 3006 size(I_cell_u),I_cell_no_max_cur 13986 13992 I_cells size exceed, to increase size I_equal_cell_no_u1_max 20988 IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 27150 105205 time,IIB_I_cell_no_uvw_total1 3 71237 0 0 264577 0 0 IIB_equal_cell_no_u1_max 3006 I_equal_cell_no_u1_max 20988 IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 27150 105205 time,IIB_I_cell_no_uvw_total1 4 70676 0 0 264504 0 0 IIB_equal_cell_no_u1_max 3006 I_equal_cell_no_u1_max 20988 IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 27150 105205 time,IIB_I_cell_no_uvw_total1 5 70909 0 0 264557 0 0 escape_time reached, so abort Recommended no for final_IIB_equal_no1,final_IIB_no1,final_IIB_global_no1,final _I_equal_no1,final_I_no1,final_I_global_no1 3757 33937 89546 26235 131506 496098 0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.400000000000000 0 -400000 AB,AA,BB,CC -3.30400006193668 3.62600007723086 3.22400005813688 3.22400005813688 size_x,size_y,size_z 718x718x285 total grid size = 146924340 recommended cores (50k / core) = 2938.48680000000 min_area,max_area,min_grid_area,ratio 9.229464518072358E-005 0.108728370237650 4.000000000000000E-004 271.820925594124 ratio bet max_area,min_grid_area not ideal max element length should be 2.828427124746190E-002 body_cg_ini -2.750916668409921E-003 2.695116216584833E-004 6.75873529067005 Warning - length difference between element and cell max_element_length,min_element_length,min_delta 0.558213960965656 4.851715434749100E-003 2.000000000000000E-002 maximum ngh_surfaces and ngh_vertics are 4 4 minimum ngh_surfaces and ngh_vertics are 1 1 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz e(IIB_global_cell_u),size(I_global_cell_u) 33937 131506 3757 26235 89546 496098 IIB_I_cell_no_uvw_total1 71637 71704 71732 264465 264616 264595 1 0.00150000 0.00000000 0.00000000 0.00000000 0.00000000E+00 0.00000000E+00 0.00000000E+00 escape_time reached, so abort cd_cl_cs_mom_implicit1 -0.203707530878565 -3.78586926105787 -142.425572181451 3.49109646861912 -18.5191982260976 194.106345316573 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a named std0442 with 288 processors, by tsltaywb Fri Apr 20 22:40:33 2018 Using Petsc Release Version 3.9.0, Apr, 07, 2018 Max Max/Min Avg Total Time (sec): 2.523e+02 1.00002 2.523e+02 Objects: 8.450e+02 1.00118 8.440e+02 Flop: 4.819e+10 1.10663 4.730e+10 1.362e+13 Flop/sec: 1.910e+08 1.10663 1.875e+08 5.400e+10 MPI Messages: 9.780e+04 3.18949 7.382e+04 2.126e+07 MPI Message Lengths: 1.951e+09 2.12277 2.441e+04 5.190e+11 MPI Reductions: 1.416e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 2.5229e+02 100.0% 1.3623e+13 100.0% 2.126e+07 100.0% 2.441e+04 100.0% 1.407e+03 99.4% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSided 5 1.0 3.7102e-0213.7 0.00e+00 0.0 8.5e+03 4.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 BuildTwoSidedF 67 1.0 8.4729e+00 2.0 0.00e+00 0.0 4.9e+04 2.5e+05 0.0e+00 3 0 0 2 0 3 0 0 2 0 0 VecDot 8 1.0 7.0657e-01 5.3 2.48e+07 1.1 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 1 0 0 0 0 1 9981 VecDotNorm2 4 1.0 4.9834e-0155.8 2.48e+07 1.1 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 14152 VecMDot 100 1.0 1.2717e+00 2.4 1.25e+08 1.1 0.0e+00 0.0e+00 1.0e+02 0 0 0 0 7 0 0 0 0 7 27905 VecTDot 244 1.0 2.3305e+00 3.3 2.52e+08 1.1 0.0e+00 0.0e+00 2.4e+02 1 1 0 0 17 1 1 0 0 17 30766 VecNorm 245 1.0 4.0297e+0117.5 1.81e+08 1.1 0.0e+00 0.0e+00 2.4e+02 9 0 0 0 17 9 0 0 0 17 1277 VecScale 110 1.0 4.8598e-02 2.2 1.25e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 73019 VecCopy 662 1.0 7.8418e-01 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 2395 1.0 6.5509e-01 2.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 254 1.0 1.0951e+00 2.1 2.55e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 66060 VecAYPX 5198 1.0 2.9301e+00 2.4 8.43e+08 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 81748 VecAXPBYCZ 2548 1.0 1.9207e+00 2.1 1.49e+09 1.1 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 220650 VecWAXPY 8 1.0 1.9802e-01 2.7 2.48e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 35614 VecMAXPY 110 1.0 2.6582e-01 1.6 1.48e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 157767 VecAssemblyBegin 34 1.0 4.0356e-01 4.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 34 1.0 6.3658e-05 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 55 1.0 4.0215e-02 1.7 6.24e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 44121 VecScatterBegin 5366 1.0 1.6781e+00 3.0 0.00e+00 0.0 2.0e+07 2.4e+04 0.0e+00 0 0 95 94 0 0 0 95 94 0 0 VecScatterEnd 5366 1.0 7.0576e+01 7.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 17 0 0 0 0 0 VecSetRandom 5 1.0 1.4006e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 110 1.0 1.1545e+00 4.5 3.75e+07 1.1 0.0e+00 0.0e+00 1.1e+02 0 0 0 0 8 0 0 0 0 8 9221 MatMult 4040 1.0 8.7491e+01 1.8 2.25e+10 1.1 1.7e+07 2.6e+04 0.0e+00 25 46 81 86 0 25 46 81 86 0 72363 MatMultAdd 635 1.0 1.3872e+01 7.7 1.17e+09 1.1 1.4e+06 6.2e+03 0.0e+00 2 2 7 2 0 2 2 7 2 0 23814 MatMultTranspose 635 1.0 1.4661e+01 7.5 1.17e+09 1.1 1.4e+06 6.2e+03 0.0e+00 2 2 7 2 0 2 2 7 2 0 22532 MatSolve 13911.6 2.7498e+00 1.6 1.38e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 142785 MatSOR 3865 1.0 7.5148e+01 2.3 1.71e+10 1.1 0.0e+00 0.0e+00 0.0e+00 18 36 0 0 0 18 36 0 0 0 64645 MatLUFactorSym 1 1.0 5.1308e-0474.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 2 1.0 1.8094e+00 2.2 5.16e+08 1.1 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 80741 MatILUFactorSym 1 1.0 6.5920e+0014.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatConvert 5 1.0 4.5344e-01 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 16 1.0 5.1112e-01 1.5 9.57e+07 1.1 2.2e+04 2.3e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 52984 MatResidual 635 1.0 1.6918e+01 3.2 3.31e+09 1.1 2.8e+06 2.3e+04 0.0e+00 4 7 13 12 0 4 7 13 12 0 55100 MatAssemblyBegin 102 1.0 8.2371e+00 1.8 0.00e+00 0.0 4.9e+04 2.5e+05 0.0e+00 3 0 0 2 0 3 0 0 2 0 0 MatAssemblyEnd 102 1.0 6.0627e+00 1.2 0.00e+00 0.0 2.4e+05 4.6e+03 2.2e+02 2 0 1 0 15 2 0 1 0 15 0 MatGetRow 1702611 1.1 1.6791e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 2 2.0 5.6982e-04597.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCreateSubMat 4 1.0 3.1587e-01 1.0 0.00e+00 0.0 6.8e+03 3.7e+02 6.4e+01 0 0 0 0 5 0 0 0 0 5 0 MatGetOrdering 2 2.0 4.2314e-0117.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 5 1.0 1.6996e-01 1.2 0.00e+00 0.0 5.9e+05 3.1e+03 7.5e+01 0 0 3 0 5 0 0 3 0 5 0 MatZeroEntries 5 1.0 1.8760e-0112.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAXPY 5 1.0 2.1740e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 5 1.0 1.3280e+00 1.0 2.61e+07 1.1 1.4e+05 1.1e+04 6.2e+01 1 0 1 0 4 1 0 1 0 4 5527 MatMatMultSym 5 1.0 1.1175e+00 1.0 0.00e+00 0.0 1.2e+05 8.9e+03 6.0e+01 0 0 1 0 4 0 0 1 0 4 0 MatMatMultNum 5 1.0 2.0309e-01 1.1 2.61e+07 1.1 2.2e+04 2.3e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 36141 MatPtAP 5 1.0 1.2759e+01 1.0 8.09e+08 1.3 3.2e+05 7.9e+04 7.7e+01 5 2 1 5 5 5 2 1 5 5 17465 MatPtAPSymbolic 5 1.0 3.8273e+00 1.0 0.00e+00 0.0 1.4e+05 9.0e+04 3.5e+01 1 0 1 2 2 1 0 1 2 2 0 MatPtAPNumeric 5 1.0 9.0744e+00 1.0 8.09e+08 1.3 1.7e+05 7.0e+04 4.0e+01 4 2 1 2 3 4 2 1 2 3 24556 MatTrnMatMult 1 1.0 4.0050e+00 1.0 5.17e+07 1.1 8.5e+03 2.4e+05 1.6e+01 2 0 0 0 1 2 0 0 0 1 3656 MatTrnMatMultSym 1 1.0 2.7167e+00 1.0 0.00e+00 0.0 7.4e+03 1.2e+05 1.6e+01 1 0 0 0 1 1 0 0 0 1 0 MatTrnMatMultNum 1 1.0 1.2892e+00 1.0 5.17e+07 1.1 1.1e+03 1.1e+06 0.0e+00 1 0 0 0 0 1 0 0 0 0 11357 MatGetLocalMat 17 1.0 2.1107e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 15 1.0 2.0231e-01 2.1 0.00e+00 0.0 1.5e+05 7.5e+04 0.0e+00 0 0 1 2 0 0 0 1 2 0 0 SFSetGraph 5 1.0 5.2452e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFSetUp 5 1.0 4.1750e-02 3.9 0.00e+00 0.0 2.5e+04 8.7e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFBcastBegin 85 1.0 8.3704e-03 4.0 0.00e+00 0.0 5.6e+05 2.9e+03 0.0e+00 0 0 3 0 0 0 0 3 0 0 0 SFBcastEnd 85 1.0 2.3875e-02 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 100 1.0 1.4133e+00 2.0 2.50e+08 1.1 0.0e+00 0.0e+00 1.0e+02 0 1 0 0 7 0 1 0 0 7 50219 KSPSetUp 20 1.0 2.1678e+00 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.8e+01 1 0 0 0 2 1 0 0 0 2 0 KSPSolve 9 1.0 1.8116e+02 1.0 4.81e+10 1.1 2.1e+07 2.4e+04 1.2e+03 72100100 96 82 72100100 96 83 75103 PCGAMGGraph_AGG 5 1.0 3.1132e+00 1.0 2.61e+07 1.1 5.4e+04 1.2e+04 6.0e+01 1 0 0 0 4 1 0 0 0 4 2358 PCGAMGCoarse_AGG 5 1.0 4.3671e+00 1.0 5.17e+07 1.1 6.1e+05 9.0e+03 9.5e+01 2 0 3 1 7 2 0 3 1 7 3352 PCGAMGProl_AGG 5 1.0 4.7506e-01 1.0 0.00e+00 0.0 5.5e+04 1.8e+04 8.0e+01 0 0 0 0 6 0 0 0 0 6 0 PCGAMGPOpt_AGG 5 1.0 2.6141e+00 1.0 4.53e+08 1.1 3.6e+05 1.9e+04 2.1e+02 1 1 2 1 15 1 1 2 1 15 48912 GAMG: createProl 5 1.0 1.0569e+01 1.0 5.31e+08 1.1 1.1e+06 1.3e+04 4.4e+02 4 1 5 3 31 4 1 5 3 31 14178 Graph 10 1.0 3.1086e+00 1.0 2.61e+07 1.1 5.4e+04 1.2e+04 6.0e+01 1 0 0 0 4 1 0 0 0 4 2361 MIS/Agg 5 1.0 1.7001e-01 1.2 0.00e+00 0.0 5.9e+05 3.1e+03 7.5e+01 0 0 3 0 5 0 0 3 0 5 0 SA: col data 5 1.0 6.0319e-02 1.1 0.00e+00 0.0 3.4e+04 2.6e+04 2.0e+01 0 0 0 0 1 0 0 0 0 1 0 SA: frmProl0 5 1.0 3.9480e-01 1.0 0.00e+00 0.0 2.1e+04 5.6e+03 4.0e+01 0 0 0 0 3 0 0 0 0 3 0 SA: smooth 5 1.0 1.6144e+00 1.0 3.53e+07 1.1 1.4e+05 1.1e+04 7.2e+01 1 0 1 0 5 1 0 1 0 5 6158 GAMG: partLevel 5 1.0 1.3111e+01 1.0 8.09e+08 1.3 3.2e+05 7.7e+04 1.8e+02 5 2 2 5 13 5 2 2 5 13 16996 repartition 2 1.0 1.3661e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 1 0 0 0 0 1 0 Invert-Sort 2 1.0 3.1821e-0235.4 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 1 0 0 0 0 1 0 Move A 2 1.0 8.8933e-03 1.1 0.00e+00 0.0 1.5e+03 1.5e+03 3.4e+01 0 0 0 0 2 0 0 0 0 2 0 Move P 2 1.0 3.0897e-01 1.0 0.00e+00 0.0 5.3e+03 3.4e+01 3.4e+01 0 0 0 0 2 0 0 0 0 2 0 PCSetUp 4 1.0 3.1552e+01 1.2 1.86e+09 1.2 1.4e+06 2.8e+04 6.5e+02 11 4 7 7 46 11 4 7 7 46 16441 PCSetUpOnBlocks 131 1.0 7.7258e+00 5.2 5.16e+08 1.1 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 18910 PCApply 139 1.0 1.3946e+02 1.3 4.28e+10 1.1 2.0e+07 2.1e+04 1.0e+02 49 89 92 79 7 49 89 92 79 7 86847 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 363 363 438680928 0. Matrix 150 150 2506664764 0. Matrix Coarsen 5 5 3180 0. Distributed Mesh 18 18 94176 0. Index Set 128 128 44378656 0. IS L to G Mapping 18 18 19585328 0. Star Forest Graph 41 41 35424 0. Discrete System 18 18 16632 0. Vec Scatter 58 58 33956736 0. Krylov Solver 20 20 314880 0. Preconditioner 15 15 15212 0. PetscRandom 10 10 6460 0. Viewer 1 0 0 0. ======================================================================================================================== Average time to get PetscTime(): 0. Average time for MPI_Barrier(): 1.54018e-05 Average time for zero size MPI_Send(): 5.56641e-06 #PETSc Option Table entries: -log_view -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-mpi-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64 --with-blaslapack-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 --download-hypre=/home/users/nus/tsltaywb/scratch/source/v2.14.0.tar.gz --download-ml=/home/users/nus/tsltaywb/scratch/source/ml-v6.2-p4.tar.gz --with-debugging=0 --prefix=/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel --with-shared-libraries=0 --known-mpi-shared-libraries=0 --with-fortran-interfaces=1 --CFLAGS="-xHost -g -O2" --CXXFLAGS="-xHost -g -O2" --FFLAGS="-xHost -g -O2" ----------------------------------------- Libraries compiled on 2018-04-17 05:50:35 on nus02 Machine characteristics: Linux-2.6.32-696.18.7.el6.x86_64-x86_64-with-redhat-6.9-Santiago Using PETSc directory: /home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel Using PETSc arch: ----------------------------------------- Using C compiler: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpicc -xHost -g -O2 Using Fortran compiler: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 -xHost -g -O2 ----------------------------------------- Using include paths: -I/home/users/nus/tsltaywb/lib//include -I/home/users/nus/tsltaywb/lib///include -I/home/users/nus/tsltaywb/lib//include -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include ----------------------------------------- Using C linker: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpicc Using Fortran linker: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 Using libraries: -Wl,-rpath,/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -L/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -lpetsc -Wl,-rpath,/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -L/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -Wl,-rpath,/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib/debug_mt -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/ippcp/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/ipp/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64_lin -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64/gcc4.4 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/daal/lib/intel64_lin -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64_lin/gcc4.4 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib -lHYPRE -lml -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lX11 -lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lstdc++ -ldl ----------------------------------------- ====================================================================================== Resource Usage on 2018-04-20 22:40:36.157718: JobId: 6800949.wlm01 Project: 11000324 Exit Status: 0 NCPUs Requested: 288 NCPUs Used: 288 CPU Time Used: 01:39:53 Memory Requested: 1152gb Memory Used: 97346692kb Vmem Used: 117893608kb Walltime requested: 00:05:00 Walltime Used: 00:04:26 Execution Nodes Used: (std0442:ncpus=24:mem=100663296kb)+(std0116:ncpus=24:mem=100663296kb)+(std0162:ncpus=24:mem=100663296kb)+(std0207:ncpus=24:mem=100663296kb)+(std0215:ncpus=24:mem=100663296kb)+(std0218:ncpus=24:mem=100663296kb)+(std0227:ncpus=24:mem=100663296kb)+(std0254:ncpus=24:mem=100663296kb)+(std0256:ncpus=24:mem=100663296kb)+(std0303:ncpus=24:mem=100663296kb)+(std0313:ncpus=24:mem=100663296kb)+(std0363:ncpus=24:mem=100663296kb) ====================================================================================== -------------- next part -------------- 0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.400000000000000 0 -400000 AB,AA,BB,CC -3.30400006193668 3.62600007723086 3.22400005813688 3.22400005813688 size_x,size_y,size_z 718x718x285 total grid size = 146924340 recommended cores (50k / core) = 2938.48680000000 min_area,max_area,min_grid_area,ratio 9.229464518072358E-005 0.108728370237650 4.000000000000000E-004 271.820925594124 ratio bet max_area,min_grid_area not ideal max element length should be 2.828427124746190E-002 body_cg_ini -2.750916668409921E-003 2.695116216584833E-004 6.75873529067005 Warning - length difference between element and cell max_element_length,min_element_length,min_delta 0.558213960965656 4.851715434749100E-003 2.000000000000000E-002 maximum ngh_surfaces and ngh_vertics are 4 3 minimum ngh_surfaces and ngh_vertics are 1 3 min IIB_cell_no 0 max IIB_cell_no 227 IIB_cell_no_sum 71637 min equal_size 560 max equal_size 792 min I_cell_no 0 max I_cell_no 960 I_cell_no_sum 264465 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz e(IIB_global_cell_u),size(I_global_cell_u) 227 960 227 960 71637 264465 IIB_equal_cell_no_u1_max 227 I_equal_cell_no_u1_max 960 IIB_I_cell_no_uvw_total1 71637 0 0 264465 0 0 size(IIB_cell_u),IIB_cell_no_max_cur 227 6585 local IIB_cells size exceed, to increase size size(I_cell_u),I_cell_no_max_cur 960 36719 local I_cells size exceed, to increase size IIB_cell_no_u1_max,I_cell_no_u1_max 9877 55078 size(IIB_cell_u),IIB_cell_no_max_cur 227 385 IIB_cells size exceed, to increase size IIB_equal_cell_no_u1_max 577 size(I_cell_u),I_cell_no_max_cur 960 2002 I_cells size exceed, to increase size I_equal_cell_no_u1_max 3003 size(I_global_cell_u1),I_global_cell_no_u1_max_cur 264465 264586 I global cells size exceed, to increase size IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 9877 55078 time,IIB_I_cell_no_uvw_total1 2 71589 0 0 264586 0 0 size(IIB_cell_u),IIB_cell_no_max_cur 577 606 IIB_cells size exceed, to increase size IIB_equal_cell_no_u1_max 909 size(I_cell_u),I_cell_no_max_cur 3003 3328 I_cells size exceed, to increase size I_equal_cell_no_u1_max 4992 IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 9877 55078 time,IIB_I_cell_no_uvw_total1 3 71237 0 0 264577 0 0 IIB_equal_cell_no_u1_max 909 I_equal_cell_no_u1_max 4992 IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 9877 55078 time,IIB_I_cell_no_uvw_total1 4 70676 0 0 264504 0 0 IIB_equal_cell_no_u1_max 909 I_equal_cell_no_u1_max 4992 IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 9877 55078 time,IIB_I_cell_no_uvw_total1 5 70909 0 0 264557 0 0 escape_time reached, so abort Recommended no for final_IIB_equal_no1,final_IIB_no1,final_IIB_global_no1,final _I_equal_no1,final_I_no1,final_I_global_no1 1136 12346 89546 6240 68847 496098 0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.400000000000000 0 -400000 AB,AA,BB,CC -3.30400006193668 3.62600007723086 3.22400005813688 3.22400005813688 size_x,size_y,size_z 718x718x285 total grid size = 146924340 recommended cores (50k / core) = 2938.48680000000 min_area,max_area,min_grid_area,ratio 9.229464518072358E-005 0.108728370237650 4.000000000000000E-004 271.820925594124 ratio bet max_area,min_grid_area not ideal max element length should be 2.828427124746190E-002 body_cg_ini -2.750916668409921E-003 2.695116216584833E-004 6.75873529067005 Warning - length difference between element and cell max_element_length,min_element_length,min_delta 0.558213960965656 4.851715434749100E-003 2.000000000000000E-002 maximum ngh_surfaces and ngh_vertics are 4 3 minimum ngh_surfaces and ngh_vertics are 1 3 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz e(IIB_global_cell_u),size(I_global_cell_u) 12346 68847 1136 6240 89546 496098 IIB_I_cell_no_uvw_total1 71637 71704 71732 264465 264616 264595 1 0.00150000 0.00000000 0.00000000 0.00000000 0.00000000E+00 0.00000000E+00 0.00000000E+00 escape_time reached, so abort cd_cl_cs_mom_implicit1 -0.203573203414587 -3.78836120073860 -142.428735523722 3.49076187897078 -18.4949348357545 194.047090449719 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a named std0930 with 1440 processors, by tsltaywb Sat Apr 21 20:13:18 2018 Using Petsc Release Version 3.9.0, Apr, 07, 2018 Max Max/Min Avg Total Time (sec): 1.077e+02 1.00011 1.077e+02 Objects: 8.780e+02 1.00114 8.770e+02 Flop: 9.807e+09 1.27406 9.576e+09 1.379e+13 Flop/sec: 9.107e+07 1.27409 8.893e+07 1.281e+11 MPI Messages: 1.693e+05 8.99238 7.795e+04 1.122e+08 MPI Message Lengths: 1.007e+09 2.35317 1.203e+04 1.351e+12 MPI Reductions: 1.583e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.0768e+02 100.0% 1.3789e+13 100.0% 1.122e+08 100.0% 1.203e+04 100.0% 1.574e+03 99.4% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSided 5 1.0 2.0495e-02 4.7 0.00e+00 0.0 4.0e+04 4.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 BuildTwoSidedF 69 1.0 1.4551e+00 2.8 0.00e+00 0.0 2.4e+05 1.3e+05 0.0e+00 1 0 0 2 0 1 0 0 2 0 0 VecDot 8 1.0 2.0908e-0121.8 4.96e+06 1.2 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 1 0 0 0 0 1 33730 VecDotNorm2 4 1.0 1.9284e-0193.1 4.96e+06 1.2 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 36571 VecMDot 100 1.0 2.6590e-01 2.0 2.51e+07 1.2 0.0e+00 0.0e+00 1.0e+02 0 0 0 0 6 0 0 0 0 6 133692 VecTDot 260 1.0 7.6737e-01 1.7 5.38e+07 1.2 0.0e+00 0.0e+00 2.6e+02 0 1 0 0 16 0 1 0 0 17 99561 VecNorm 253 1.0 3.7218e+00 3.7 3.79e+07 1.2 0.0e+00 0.0e+00 2.5e+02 2 0 0 0 16 2 0 0 0 16 14464 VecScale 110 1.0 2.4757e-03 2.5 2.51e+06 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1435901 VecCopy 702 1.0 7.9543e-02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 2542 1.0 5.8362e-02 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 270 1.0 1.0637e-01 1.5 5.42e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 724347 VecAYPX 5526 1.0 3.6515e-01 2.1 1.80e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 698807 VecAXPBYCZ 2708 1.0 3.5396e-01 2.3 3.17e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 1272430 VecWAXPY 8 1.0 2.0493e-02 1.7 4.96e+06 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 344131 VecMAXPY 110 1.0 4.1608e-02 1.9 2.96e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1009728 VecAssemblyBegin 35 1.0 2.0360e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 35 1.0 6.2466e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 55 1.0 7.8759e-03 3.4 1.25e+06 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 225683 VecScatterBegin 5695 1.0 9.3908e-01 3.8 0.00e+00 0.0 1.0e+08 1.2e+04 0.0e+00 1 0 93 94 0 1 0 93 94 0 0 VecScatterEnd 5695 1.0 8.3244e+00 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 VecSetRandom 5 1.0 2.8300e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 110 1.0 2.0694e-01 1.6 7.52e+06 1.2 0.0e+00 0.0e+00 1.1e+02 0 0 0 0 7 0 0 0 0 7 51535 MatMult 4288 1.0 1.1565e+01 1.4 4.89e+09 1.4 8.9e+07 1.3e+04 0.0e+00 9 49 79 86 0 9 49 79 86 0 589627 MatMultAdd 675 1.0 2.5995e+00 5.2 2.54e+08 1.3 7.6e+06 3.1e+03 0.0e+00 2 3 7 2 0 2 3 7 2 0 137560 MatMultTranspose 675 1.0 1.8251e+00 4.0 2.54e+08 1.3 7.6e+06 3.1e+03 0.0e+00 1 3 7 2 0 1 3 7 2 0 195931 MatSolve 14712.2 7.5874e-01 2.6 2.64e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 494325 MatSOR 4105 1.0 9.8198e+00 2.0 3.15e+09 1.2 0.0e+00 0.0e+00 0.0e+00 8 32 0 0 0 8 32 0 0 0 452389 MatLUFactorSym 1 1.0 5.6100e-0481.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 2 1.0 4.5086e-01 3.0 9.66e+07 1.4 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 302823 MatILUFactorSym 1 1.0 3.1375e-01 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatConvert 5 1.0 2.7598e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 16 1.0 7.8282e-02 1.7 1.93e+07 1.3 1.1e+05 1.2e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 348309 MatResidual 675 1.0 2.4201e+00 2.1 7.25e+08 1.4 1.4e+07 1.2e+04 0.0e+00 1 7 13 13 0 1 7 13 13 0 417100 MatAssemblyBegin 106 1.0 1.3056e+00 2.1 0.00e+00 0.0 2.4e+05 1.3e+05 0.0e+00 1 0 0 2 0 1 0 0 2 0 0 MatAssemblyEnd 106 1.0 7.0611e+00 1.0 0.00e+00 0.0 1.5e+06 1.8e+03 2.3e+02 7 0 1 0 15 7 0 1 0 15 0 MatGetRow 341706 1.2 3.7706e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 2 2.0 3.4308e-04359.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCreateSubMat 6 1.0 2.0243e+00 1.0 0.00e+00 0.0 1.6e+05 1.1e+03 9.6e+01 2 0 0 0 6 2 0 0 0 6 0 MatGetOrdering 2 2.0 1.5898e-02 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 5 1.0 1.1609e-01 1.1 0.00e+00 0.0 4.7e+06 1.8e+03 1.7e+02 0 0 4 1 11 0 0 4 1 11 0 MatZeroEntries 5 1.0 4.3550e-03 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAXPY 5 1.0 5.6949e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 5 1.0 1.1438e+00 1.0 5.37e+06 1.4 6.8e+05 5.7e+03 6.2e+01 1 0 1 0 4 1 0 1 0 4 6537 MatMatMultSym 5 1.0 1.0687e+00 1.0 0.00e+00 0.0 5.7e+05 4.6e+03 6.0e+01 1 0 1 0 4 1 0 1 0 4 0 MatMatMultNum 5 1.0 6.2605e-02 1.1 5.37e+06 1.4 1.1e+05 1.2e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 119436 MatPtAP 5 1.0 5.8088e+00 1.0 1.73e+08 1.9 1.7e+06 3.8e+04 7.7e+01 5 2 2 5 5 5 2 2 5 5 40397 MatPtAPSymbolic 5 1.0 9.7437e-01 1.0 0.00e+00 0.0 7.0e+05 4.9e+04 3.5e+01 1 0 1 3 2 1 0 1 3 2 0 MatPtAPNumeric 5 1.0 4.8295e+00 1.0 1.73e+08 1.9 1.0e+06 3.0e+04 4.0e+01 4 2 1 2 3 4 2 1 2 3 48588 MatTrnMatMult 1 1.0 5.1030e-01 1.0 1.06e+07 1.3 4.4e+04 1.1e+05 1.6e+01 0 0 0 0 1 0 0 0 0 1 29433 MatTrnMatMultSym 1 1.0 3.3892e-01 1.0 0.00e+00 0.0 3.9e+04 5.5e+04 1.6e+01 0 0 0 0 1 0 0 0 0 1 0 MatTrnMatMultNum 1 1.0 1.7566e-01 1.0 1.06e+07 1.3 5.6e+03 4.8e+05 0.0e+00 0 0 0 0 0 0 0 0 0 0 85507 MatGetLocalMat 17 1.0 3.4708e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 15 1.0 9.4534e-02 2.1 0.00e+00 0.0 7.4e+05 4.1e+04 0.0e+00 0 0 1 2 0 0 0 1 2 0 0 SFSetGraph 5 1.0 5.0068e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFSetUp 5 1.0 2.3274e-02 2.1 0.00e+00 0.0 1.2e+05 4.5e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFBcastBegin 177 1.0 9.6838e-03 4.3 0.00e+00 0.0 4.6e+06 1.7e+03 0.0e+00 0 0 4 1 0 0 0 4 1 0 0 SFBcastEnd 177 1.0 2.8875e-02 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 100 1.0 2.8578e-01 1.8 5.01e+07 1.2 0.0e+00 0.0e+00 1.0e+02 0 1 0 0 6 0 1 0 0 6 248784 KSPSetUp 20 1.0 4.2640e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.8e+01 0 0 0 0 2 0 0 0 0 2 0 KSPSolve 9 1.0 3.5276e+01 1.0 9.79e+09 1.3 1.1e+08 1.2e+04 1.3e+03 33100100 97 84 33100100 97 84 390407 PCGAMGGraph_AGG 5 1.0 3.0630e-01 1.0 5.37e+06 1.4 2.5e+05 6.3e+03 6.0e+01 0 0 0 0 4 0 0 0 0 4 24412 PCGAMGCoarse_AGG 5 1.0 6.6444e-01 1.0 1.06e+07 1.3 4.8e+06 3.5e+03 1.9e+02 1 0 4 1 12 1 0 4 1 12 22605 PCGAMGProl_AGG 5 1.0 1.0120e-01 1.0 0.00e+00 0.0 2.7e+05 9.1e+03 8.0e+01 0 0 0 0 5 0 0 0 0 5 0 PCGAMGPOpt_AGG 5 1.0 1.4647e+00 1.0 9.23e+07 1.3 1.7e+06 9.5e+03 2.1e+02 1 1 2 1 13 1 1 2 1 13 88410 GAMG: createProl 5 1.0 2.5337e+00 1.0 1.08e+08 1.3 7.1e+06 5.3e+03 5.3e+02 2 1 6 3 34 2 1 6 3 34 59989 Graph 10 1.0 3.0478e-01 1.0 5.37e+06 1.4 2.5e+05 6.3e+03 6.0e+01 0 0 0 0 4 0 0 0 0 4 24533 MIS/Agg 5 1.0 1.1613e-01 1.1 0.00e+00 0.0 4.7e+06 1.8e+03 1.7e+02 0 0 4 1 11 0 0 4 1 11 0 SA: col data 5 1.0 1.8581e-02 1.1 0.00e+00 0.0 1.6e+05 1.3e+04 2.0e+01 0 0 0 0 1 0 0 0 0 1 0 SA: frmProl0 5 1.0 7.7043e-02 1.0 0.00e+00 0.0 1.1e+05 2.5e+03 4.0e+01 0 0 0 0 3 0 0 0 0 3 0 SA: smooth 5 1.0 1.2110e+00 1.0 7.25e+06 1.4 6.8e+05 5.7e+03 7.2e+01 1 0 1 0 5 1 0 1 0 5 8362 GAMG: partLevel 5 1.0 8.0089e+00 1.0 1.73e+08 1.9 1.9e+06 3.5e+04 2.3e+02 7 2 2 5 15 7 2 2 5 15 29299 repartition 3 1.0 5.4121e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 1 0 0 0 0 1 0 Invert-Sort 3 1.0 1.5763e-0122.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 1 0 0 0 0 1 0 Move A 3 1.0 8.4985e-01 1.0 0.00e+00 0.0 9.2e+04 1.8e+03 5.1e+01 1 0 0 0 3 1 0 0 0 3 0 Move P 3 1.0 1.1894e+00 1.0 0.00e+00 0.0 6.4e+04 6.2e+01 5.1e+01 1 0 0 0 3 1 0 0 0 3 0 PCSetUp 4 1.0 1.1349e+01 1.0 3.78e+08 1.5 9.0e+06 1.1e+04 7.9e+02 10 4 8 8 50 10 4 8 8 50 46098 PCSetUpOnBlocks 139 1.0 7.7229e-01 3.2 9.66e+07 1.4 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 176788 PCApply 147 1.0 2.1472e+01 1.1 8.70e+09 1.3 1.0e+08 1.1e+04 1.0e+02 19 89 91 80 7 19 89 91 80 7 569588 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 369 369 92175624 0. Matrix 158 158 487572844 0. Matrix Coarsen 5 5 3180 0. Distributed Mesh 18 18 94176 0. Index Set 144 144 10541668 0. IS L to G Mapping 18 18 4620592 0. Star Forest Graph 41 41 35424 0. Discrete System 18 18 16632 0. Vec Scatter 61 61 6861288 0. Krylov Solver 20 20 314880 0. Preconditioner 15 15 15212 0. PetscRandom 10 10 6460 0. Viewer 1 0 0 0. ======================================================================================================================== Average time to get PetscTime(): 0. Average time for MPI_Barrier(): 0.000318766 Average time for zero size MPI_Send(): 8.26998e-06 #PETSc Option Table entries: -log_view -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-mpi-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64 --with-blaslapack-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 --download-hypre=/home/users/nus/tsltaywb/scratch/source/v2.14.0.tar.gz --download-ml=/home/users/nus/tsltaywb/scratch/source/ml-v6.2-p4.tar.gz --with-debugging=0 --prefix=/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel --with-shared-libraries=0 --known-mpi-shared-libraries=0 --with-fortran-interfaces=1 --CFLAGS="-xHost -g -O2" --CXXFLAGS="-xHost -g -O2" --FFLAGS="-xHost -g -O2" ----------------------------------------- Libraries compiled on 2018-04-17 05:50:35 on nus02 Machine characteristics: Linux-2.6.32-696.18.7.el6.x86_64-x86_64-with-redhat-6.9-Santiago Using PETSc directory: /home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel Using PETSc arch: ----------------------------------------- Using C compiler: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpicc -xHost -g -O2 Using Fortran compiler: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 -xHost -g -O2 ----------------------------------------- Using include paths: -I/home/users/nus/tsltaywb/lib//include -I/home/users/nus/tsltaywb/lib///include -I/home/users/nus/tsltaywb/lib//include -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include ----------------------------------------- Using C linker: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpicc Using Fortran linker: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 Using libraries: -Wl,-rpath,/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -L/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -lpetsc -Wl,-rpath,/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -L/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -Wl,-rpath,/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib/debug_mt -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/ippcp/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/ipp/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64_lin -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64/gcc4.4 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/daal/lib/intel64_lin -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64_lin/gcc4.4 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib -lHYPRE -lml -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lX11 -lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lstdc++ -ldl ----------------------------------------- ====================================================================================== Resource Usage on 2018-04-21 20:13:23.909068: JobId: 6809210.wlm01 Project: 11000324 Exit Status: 0 NCPUs Requested: 1440 NCPUs Used: 1440 CPU Time Used: 00:42:12 Memory Requested: 5760gb Memory Used: 29886936kb Vmem Used: 53296508kb Walltime requested: 00:05:00 Walltime Used: 00:02:04 Execution Nodes Used: (std0930:ncpus=24:mem=100663296kb)+(std0931:ncpus=24:mem=100663296kb)+(std0932:ncpus=24:mem=100663296kb)+(std0933:ncpus=24:mem=100663296kb)+(std0934:ncpus=24:mem=100663296kb)+(std0935:ncpus=24:mem=100663296kb)+(std0936:ncpus=24:mem=100663296kb)+(std0937:ncpus=24:mem=100663296kb)+(std0938:ncpus=24:mem=100663296kb)+(std0939:ncpus=24:mem=100663296kb)+(std0940:ncpus=24:mem=100663296kb)+(std0941:ncpus=24:mem=100663296kb)+(std0942:ncpus=24:mem=100663296kb)+(std0953:ncpus=24:mem=100663296kb)+(std0954:ncpus=24:mem=100663296kb)+(std0955:ncpus=24:mem=100663296kb)+(std0956:ncpus=24:mem=100663296kb)+(std0957:ncpus=24:mem=100663296kb)+(std0958:ncpus=24:mem=100663296kb)+(std0959:ncpus=24:mem=100663296kb)+(std0960:ncpus=24:mem=100663296kb)+(std0961:ncpus=24:mem=100663296kb)+(std0962:ncpus=24:mem=100663296kb)+(std0963:ncpus=24:mem=100663296kb)+(std1101:ncpus=24:mem=100663296kb)+(std1103:ncpus=24:mem=100663296kb)+(std1105:ncpus=24:mem=100663296kb)+(std1108:ncpus=24:mem=100663296kb)+(std1109:ncpus=24:mem=100663296kb)+(std1111:ncpus=24:mem=100663296kb)+(std1113:ncpus=24:mem=100663296kb)+(std1114:ncpus=24:mem=100663296kb)+(std1116:ncpus=24:mem=100663296kb)+(std1118:ncpus=24:mem=100663296kb)+(std1120:ncpus=24:mem=100663296kb)+(std1121:ncpus=24:mem=100663296kb)+(std1127:ncpus=24:mem=100663296kb)+(std1129:ncpus=24:mem=100663296kb)+(std1131:ncpus=24:mem=100663296kb)+(std1136:ncpus=24:mem=100663296kb)+(std1138:ncpus=24:mem=100663296kb)+(std1139:ncpus=24:mem=100663296kb)+(std1140:ncpus=24:mem=100663296kb)+(std1141:ncpus=24:mem=100663296kb)+(std1154:ncpus=24:mem=100663296kb)+(std1247:ncpus=24:mem=100663296kb)+(std1248:ncpus=24:mem=100663296kb)+(std1250:ncpus=24:mem=100663296kb)+(std1251:ncpus=24:mem=100663296kb)+(std1253:ncpus=24:mem=100663296kb)+(std1255:ncpus=24:mem=100663296kb)+(std1256:ncpus=24:mem=100663296kb)+(std1262:ncpus=24:mem=100663296kb)+(std1263:ncpus=24:mem=100663296kb)+(std1264:ncpus=24:mem=100663296kb)+(std1265:ncpus=24:mem=100663296kb)+(std1268:ncpus=24:mem=100663296kb)+(std1269:ncpus=24:mem=100663296kb)+(std1466:ncpus=24:mem=100663296kb)+(std0630:ncpus=24:mem=100663296kb) ====================================================================================== -------------- next part -------------- 0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.400000000000000 0 -400000 AB,AA,BB,CC -3.30400006193668 3.62600007723086 3.22400005813688 3.22400005813688 size_x,size_y,size_z 718x718x285 total grid size = 146924340 recommended cores (50k / core) = 2938.48680000000 min_area,max_area,min_grid_area,ratio 9.229464518072358E-005 0.108728370237650 4.000000000000000E-004 271.820925594124 ratio bet max_area,min_grid_area not ideal max element length should be 2.828427124746190E-002 body_cg_ini -2.750916668409921E-003 2.695116216584833E-004 6.75873529067005 Warning - length difference between element and cell max_element_length,min_element_length,min_delta 0.558213960965656 4.851715434749100E-003 2.000000000000000E-002 maximum ngh_surfaces and ngh_vertics are 4 3 minimum ngh_surfaces and ngh_vertics are 1 3 min IIB_cell_no 0 max IIB_cell_no 324 IIB_cell_no_sum 71637 min equal_size 910 max equal_size 1232 min I_cell_no 0 max I_cell_no 1440 I_cell_no_sum 264465 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz e(IIB_global_cell_u),size(I_global_cell_u) 324 1440 324 1440 71637 264465 IIB_equal_cell_no_u1_max 324 I_equal_cell_no_u1_max 1440 IIB_I_cell_no_uvw_total1 71637 0 0 264465 0 0 size(IIB_cell_u),IIB_cell_no_max_cur 324 11334 local IIB_cells size exceed, to increase size size(I_cell_u),I_cell_no_max_cur 1440 62615 local I_cells size exceed, to increase size IIB_cell_no_u1_max,I_cell_no_u1_max 17001 93922 size(IIB_cell_u),IIB_cell_no_max_cur 324 577 IIB_cells size exceed, to increase size IIB_equal_cell_no_u1_max 865 size(I_cell_u),I_cell_no_max_cur 1440 3168 I_cells size exceed, to increase size I_equal_cell_no_u1_max 4752 size(I_global_cell_u1),I_global_cell_no_u1_max_cur 264465 264586 I global cells size exceed, to increase size IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 17001 93922 time,IIB_I_cell_no_uvw_total1 2 71589 0 0 264586 0 0 IIB_equal_cell_no_u1_max 865 size(I_cell_u),I_cell_no_max_cur 4752 4896 I_cells size exceed, to increase size I_equal_cell_no_u1_max 7344 IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 17001 93922 time,IIB_I_cell_no_uvw_total1 3 71237 0 0 264577 0 0 IIB_equal_cell_no_u1_max 865 I_equal_cell_no_u1_max 7344 IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 17001 93922 time,IIB_I_cell_no_uvw_total1 4 70676 0 0 264504 0 0 IIB_equal_cell_no_u1_max 865 I_equal_cell_no_u1_max 7344 IIB_global_cell_no_u1_max,I_global_cell_no_u1_max 71637 396879 IIB_cell_no_u1_max,I_cell_no_u1_max 17001 93922 time,IIB_I_cell_no_uvw_total1 5 70909 0 0 264557 0 0 escape_time reached, so abort Recommended no for final_IIB_equal_no1,final_IIB_no1,final_IIB_global_no1,final _I_equal_no1,final_I_no1,final_I_global_no1 1081 21251 89546 9180 117402 496098 0.000000000000000E+000 3.700000000000000E-002 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.400000000000000 0 -400000 AB,AA,BB,CC -3.30400006193668 3.62600007723086 3.22400005813688 3.22400005813688 size_x,size_y,size_z 718x718x285 total grid size = 146924340 recommended cores (50k / core) = 2938.48680000000 min_area,max_area,min_grid_area,ratio 9.229464518072358E-005 0.108728370237650 4.000000000000000E-004 271.820925594124 ratio bet max_area,min_grid_area not ideal max element length should be 2.828427124746190E-002 body_cg_ini -2.750916668409921E-003 2.695116216584833E-004 6.75873529067005 Warning - length difference between element and cell max_element_length,min_element_length,min_delta 0.558213960965656 4.851715434749100E-003 2.000000000000000E-002 maximum ngh_surfaces and ngh_vertics are 4 3 minimum ngh_surfaces and ngh_vertics are 1 3 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u),siz e(IIB_global_cell_u),size(I_global_cell_u) 21251 117402 1081 9180 89546 496098 IIB_I_cell_no_uvw_total1 71637 71704 71732 264465 264616 264595 1 0.00150000 0.00000000 0.00000000 0.00000000 0.00000000E+00 0.00000000E+00 0.00000000E+00 escape_time reached, so abort cd_cl_cs_mom_implicit1 -0.203534875831958 -3.78723081611176 -142.427697164364 3.49017516030323 -18.5098557775327 194.067816292935 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a named std0864 with 960 processors, by tsltaywb Sat Apr 21 01:26:50 2018 Using Petsc Release Version 3.9.0, Apr, 07, 2018 Max Max/Min Avg Total Time (sec): 1.168e+02 1.00008 1.168e+02 Objects: 8.780e+02 1.00114 8.770e+02 Flop: 1.506e+10 1.20490 1.414e+10 1.358e+13 Flop/sec: 1.289e+08 1.20492 1.211e+08 1.162e+11 MPI Messages: 1.469e+05 9.74171 7.679e+04 7.372e+07 MPI Message Lengths: 1.183e+09 2.27657 1.417e+04 1.044e+12 MPI Reductions: 1.547e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.1681e+02 100.0% 1.3576e+13 100.0% 7.372e+07 100.0% 1.417e+04 100.0% 1.538e+03 99.4% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSided 5 1.0 1.2648e-02 7.1 0.00e+00 0.0 3.0e+04 4.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 BuildTwoSidedF 69 1.0 1.2455e+00 4.1 0.00e+00 0.0 1.5e+05 1.6e+05 0.0e+00 0 0 0 2 0 0 0 0 2 0 0 VecDot 8 1.0 1.1479e-01 9.0 7.75e+06 1.1 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 1 0 0 0 0 1 61436 VecDotNorm2 4 1.0 1.0597e-0111.8 7.75e+06 1.1 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 66552 VecMDot 100 1.0 2.2530e-01 3.0 3.91e+07 1.1 0.0e+00 0.0e+00 1.0e+02 0 0 0 0 6 0 0 0 0 7 157690 VecTDot 252 1.0 9.9791e-01 1.7 8.14e+07 1.1 0.0e+00 0.0e+00 2.5e+02 1 1 0 0 16 1 1 0 0 16 74204 VecNorm 249 1.0 3.8511e+00 4.9 5.79e+07 1.1 0.0e+00 0.0e+00 2.5e+02 2 0 0 0 16 2 0 0 0 16 13672 VecScale 110 1.0 2.6612e-03 1.8 3.91e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1334996 VecCopy 682 1.0 1.4146e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 2470 1.0 7.4235e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 262 1.0 1.7395e-01 1.2 8.21e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 429416 VecAYPX 5362 1.0 6.6165e-01 2.3 2.72e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 373910 VecAXPBYCZ 2628 1.0 3.9758e-01 1.5 4.81e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 1099671 VecWAXPY 8 1.0 3.3081e-02 1.7 7.75e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 213183 VecMAXPY 110 1.0 5.6687e-02 1.3 4.62e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 740673 VecAssemblyBegin 35 1.0 1.7896e-01 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 35 1.0 6.3896e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 55 1.0 1.2317e-02 3.1 1.95e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 144216 VecScatterBegin 5531 1.0 1.0172e+00 2.6 0.00e+00 0.0 6.9e+07 1.4e+04 0.0e+00 1 0 93 94 0 1 0 93 94 0 0 VecScatterEnd 5531 1.0 9.3496e+00 2.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 VecSetRandom 5 1.0 4.4341e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 110 1.0 7.5254e-02 1.7 1.17e+07 1.1 0.0e+00 0.0e+00 1.1e+02 0 0 0 0 7 0 0 0 0 7 141630 MatMult 4164 1.0 1.5945e+01 1.4 7.33e+09 1.3 5.9e+07 1.5e+04 0.0e+00 12 49 80 86 0 12 49 80 86 0 413057 MatMultAdd 655 1.0 3.4669e+00 5.3 3.82e+08 1.2 4.8e+06 3.8e+03 0.0e+00 2 3 6 2 0 2 3 6 2 0 99404 MatMultTranspose 655 1.0 1.7476e+00 3.1 3.82e+08 1.2 4.8e+06 3.8e+03 0.0e+00 1 3 6 2 0 1 3 6 2 0 197203 MatSolve 14311.9 6.1919e-01 1.3 4.20e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 615183 MatSOR 3985 1.0 1.3915e+01 1.5 5.00e+09 1.2 0.0e+00 0.0e+00 0.0e+00 11 33 0 0 0 11 33 0 0 0 324223 MatLUFactorSym 1 1.0 4.4298e-0464.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 2 1.0 3.3439e-01 1.4 1.55e+08 1.3 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 417768 MatILUFactorSym 1 1.0 1.6124e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatConvert 5 1.0 3.8676e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 16 1.0 1.1365e-01 1.3 3.01e+07 1.2 7.2e+04 1.4e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 239293 MatResidual 655 1.0 2.7324e+00 1.8 1.08e+09 1.3 9.5e+06 1.4e+04 0.0e+00 2 7 13 13 0 2 7 13 13 0 355949 MatAssemblyBegin 106 1.0 1.1344e+00 2.3 0.00e+00 0.0 1.5e+05 1.6e+05 0.0e+00 1 0 0 2 0 1 0 0 2 0 0 MatAssemblyEnd 106 1.0 5.9141e+00 1.0 0.00e+00 0.0 9.2e+05 2.3e+03 2.3e+02 5 0 1 0 15 5 0 1 0 15 0 MatGetRow 533004 1.1 5.9415e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 2 2.0 3.6955e-0538.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCreateSubMat 6 1.0 1.6832e+00 1.0 0.00e+00 0.0 1.3e+05 1.2e+03 9.6e+01 1 0 0 0 6 1 0 0 0 6 0 MatGetOrdering 2 2.0 9.6362e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 5 1.0 9.3870e-02 1.1 0.00e+00 0.0 3.1e+06 1.8e+03 1.4e+02 0 0 4 1 9 0 0 4 1 9 0 MatZeroEntries 5 1.0 5.8362e-03 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAXPY 5 1.0 7.1930e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 5 1.0 1.8777e+00 1.0 8.28e+06 1.3 4.5e+05 6.8e+03 6.2e+01 2 0 1 0 4 2 0 1 0 4 3954 MatMatMultSym 5 1.0 1.8023e+00 1.0 0.00e+00 0.0 3.8e+05 5.5e+03 6.0e+01 2 0 1 0 4 2 0 1 0 4 0 MatMatMultNum 5 1.0 6.5588e-02 1.0 8.28e+06 1.3 7.2e+04 1.4e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 113198 MatPtAP 5 1.0 5.6372e+00 1.0 2.61e+08 1.6 1.0e+06 4.9e+04 7.7e+01 5 2 1 5 5 5 2 1 5 5 40626 MatPtAPSymbolic 5 1.0 1.2509e+00 1.0 0.00e+00 0.0 4.6e+05 5.7e+04 3.5e+01 1 0 1 3 2 1 0 1 3 2 0 MatPtAPNumeric 5 1.0 4.3676e+00 1.0 2.61e+08 1.6 5.7e+05 4.2e+04 4.0e+01 4 2 1 2 3 4 2 1 2 3 52436 MatTrnMatMult 1 1.0 6.8610e-01 1.0 1.64e+07 1.2 2.9e+04 1.3e+05 1.6e+01 1 0 0 0 1 1 0 0 0 1 21707 MatTrnMatMultSym 1 1.0 5.0253e-01 1.0 0.00e+00 0.0 2.6e+04 6.7e+04 1.6e+01 0 0 0 0 1 0 0 0 0 1 0 MatTrnMatMultNum 1 1.0 1.8408e-01 1.0 1.64e+07 1.2 3.7e+03 5.9e+05 0.0e+00 0 0 0 0 0 0 0 0 0 0 80908 MatGetLocalMat 17 1.0 4.8088e-02 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 15 1.0 1.0984e-01 2.0 0.00e+00 0.0 5.1e+05 4.6e+04 0.0e+00 0 0 1 2 0 0 0 1 2 0 0 SFSetGraph 5 1.0 4.5300e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFSetUp 5 1.0 1.6025e-02 2.4 0.00e+00 0.0 9.1e+04 4.7e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFBcastBegin 153 1.0 9.5887e-03 3.8 0.00e+00 0.0 3.0e+06 1.7e+03 0.0e+00 0 0 4 0 0 0 0 4 0 0 0 SFBcastEnd 153 1.0 1.4909e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 100 1.0 2.6370e-01 2.2 7.82e+07 1.1 0.0e+00 0.0e+00 1.0e+02 0 1 0 0 6 0 1 0 0 7 269452 KSPSetUp 20 1.0 2.7623e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.8e+01 0 0 0 0 2 0 0 0 0 2 0 KSPSolve 9 1.0 4.4800e+01 1.0 1.50e+10 1.2 7.3e+07 1.4e+04 1.3e+03 38100100 96 84 38100100 96 84 302653 PCGAMGGraph_AGG 5 1.0 3.6210e-01 1.0 8.28e+06 1.3 1.9e+05 6.8e+03 6.0e+01 0 0 0 0 4 0 0 0 0 4 20504 PCGAMGCoarse_AGG 5 1.0 8.4045e-01 1.0 1.64e+07 1.2 3.1e+06 4.0e+03 1.6e+02 1 0 4 1 11 1 0 4 1 11 17721 PCGAMGProl_AGG 5 1.0 1.0871e-01 1.0 0.00e+00 0.0 1.9e+05 9.9e+03 8.0e+01 0 0 0 0 5 0 0 0 0 5 0 PCGAMGPOpt_AGG 5 1.0 2.2764e+00 1.0 1.43e+08 1.2 1.2e+06 1.1e+04 2.1e+02 2 1 2 1 13 2 1 2 1 13 56612 GAMG: createProl 5 1.0 3.5835e+00 1.0 1.68e+08 1.2 4.7e+06 6.1e+03 5.1e+02 3 1 6 3 33 3 1 6 3 33 42190 Graph 10 1.0 3.5973e-01 1.0 8.28e+06 1.3 1.9e+05 6.8e+03 6.0e+01 0 0 0 0 4 0 0 0 0 4 20639 MIS/Agg 5 1.0 9.3908e-02 1.1 0.00e+00 0.0 3.1e+06 1.8e+03 1.4e+02 0 0 4 1 9 0 0 4 1 9 0 SA: col data 5 1.0 1.9782e-02 1.1 0.00e+00 0.0 1.2e+05 1.4e+04 2.0e+01 0 0 0 0 1 0 0 0 0 1 0 SA: frmProl0 5 1.0 8.2424e-02 1.0 0.00e+00 0.0 7.3e+04 3.0e+03 4.0e+01 0 0 0 0 3 0 0 0 0 3 0 SA: smooth 5 1.0 1.9676e+00 1.0 1.12e+07 1.3 4.5e+05 6.8e+03 7.2e+01 2 0 1 0 5 2 0 1 0 5 5110 GAMG: partLevel 5 1.0 7.4519e+00 1.0 2.61e+08 1.6 1.2e+06 4.4e+04 2.3e+02 6 2 2 5 15 6 2 2 5 15 30733 repartition 3 1.0 3.1521e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 1 0 0 0 0 1 0 Invert-Sort 3 1.0 1.2159e-0116.6 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 1 0 0 0 0 1 0 Move A 3 1.0 6.3250e-01 1.0 0.00e+00 0.0 8.8e+04 1.8e+03 5.1e+01 1 0 0 0 3 1 0 0 0 3 0 Move P 3 1.0 1.0553e+00 1.0 0.00e+00 0.0 4.4e+04 6.4e+01 5.1e+01 1 0 0 0 3 1 0 0 0 3 0 PCSetUp 4 1.0 1.1538e+01 1.0 5.84e+08 1.4 5.8e+06 1.4e+04 7.7e+02 10 4 8 8 50 10 4 8 8 50 45060 PCSetUpOnBlocks 135 1.0 4.9708e-01 1.3 1.55e+08 1.3 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 281031 PCApply 143 1.0 3.0010e+01 1.1 1.34e+10 1.2 6.7e+07 1.2e+04 1.0e+02 25 89 91 80 7 25 89 91 80 7 401277 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 369 369 141361368 0. Matrix 158 158 768281328 0. Matrix Coarsen 5 5 3180 0. Distributed Mesh 18 18 94176 0. Index Set 144 144 15496224 0. IS L to G Mapping 18 18 6803624 0. Star Forest Graph 41 41 35424 0. Discrete System 18 18 16632 0. Vec Scatter 61 61 10664952 0. Krylov Solver 20 20 314880 0. Preconditioner 15 15 15212 0. PetscRandom 10 10 6460 0. Viewer 1 0 0 0. ======================================================================================================================== Average time to get PetscTime(): 0. Average time for MPI_Barrier(): 1.81675e-05 Average time for zero size MPI_Send(): 4.80612e-06 #PETSc Option Table entries: -log_view -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-mpi-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64 --with-blaslapack-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 --download-hypre=/home/users/nus/tsltaywb/scratch/source/v2.14.0.tar.gz --download-ml=/home/users/nus/tsltaywb/scratch/source/ml-v6.2-p4.tar.gz --with-debugging=0 --prefix=/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel --with-shared-libraries=0 --known-mpi-shared-libraries=0 --with-fortran-interfaces=1 --CFLAGS="-xHost -g -O2" --CXXFLAGS="-xHost -g -O2" --FFLAGS="-xHost -g -O2" ----------------------------------------- Libraries compiled on 2018-04-17 05:50:35 on nus02 Machine characteristics: Linux-2.6.32-696.18.7.el6.x86_64-x86_64-with-redhat-6.9-Santiago Using PETSc directory: /home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel Using PETSc arch: ----------------------------------------- Using C compiler: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpicc -xHost -g -O2 Using Fortran compiler: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 -xHost -g -O2 ----------------------------------------- Using include paths: -I/home/users/nus/tsltaywb/lib//include -I/home/users/nus/tsltaywb/lib///include -I/home/users/nus/tsltaywb/lib//include -I/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/include ----------------------------------------- Using C linker: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpicc Using Fortran linker: /app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/bin/mpif90 Using libraries: -Wl,-rpath,/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -L/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -lpetsc -Wl,-rpath,/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -L/home/users/nus/tsltaywb/lib/petsc-3.9.0_intel_rel/lib -Wl,-rpath,/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib/debug_mt -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/ippcp/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/ipp/lib/intel64 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64_lin -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64/gcc4.4 -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/daal/lib/intel64_lin -L/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/tbb/lib/intel64_lin/gcc4.4 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64/lib -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib -lHYPRE -lml -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lX11 -lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lstdc++ -ldl ----------------------------------------- ====================================================================================== Resource Usage on 2018-04-21 01:26:55.164851: JobId: 6808435.wlm01 Project: 11000324 Exit Status: 0 NCPUs Requested: 960 NCPUs Used: 960 CPU Time Used: 00:45:58 Memory Requested: 3840gb Memory Used: 38583332kb Vmem Used: 60953164kb Walltime requested: 00:05:00 Walltime Used: 00:02:12 Execution Nodes Used: (std0864:ncpus=24:mem=100663296kb)+(std1142:ncpus=24:mem=100663296kb)+(std1143:ncpus=24:mem=100663296kb)+(std1144:ncpus=24:mem=100663296kb)+(std1146:ncpus=24:mem=100663296kb)+(std1147:ncpus=24:mem=100663296kb)+(std1150:ncpus=24:mem=100663296kb)+(std1151:ncpus=24:mem=100663296kb)+(std1152:ncpus=24:mem=100663296kb)+(std1153:ncpus=24:mem=100663296kb)+(std1204:ncpus=24:mem=100663296kb)+(std1206:ncpus=24:mem=100663296kb)+(std1207:ncpus=24:mem=100663296kb)+(std1211:ncpus=24:mem=100663296kb)+(std1213:ncpus=24:mem=100663296kb)+(std1214:ncpus=24:mem=100663296kb)+(std1216:ncpus=24:mem=100663296kb)+(std1217:ncpus=24:mem=100663296kb)+(std1218:ncpus=24:mem=100663296kb)+(std1220:ncpus=24:mem=100663296kb)+(std1222:ncpus=24:mem=100663296kb)+(std1225:ncpus=24:mem=100663296kb)+(std1226:ncpus=24:mem=100663296kb)+(std1227:ncpus=24:mem=100663296kb)+(std1228:ncpus=24:mem=100663296kb)+(std1229:ncpus=24:mem=100663296kb)+(std1230:ncpus=24:mem=100663296kb)+(std1231:ncpus=24:mem=100663296kb)+(std1232:ncpus=24:mem=100663296kb)+(std1233:ncpus=24:mem=100663296kb)+(std1234:ncpus=24:mem=100663296kb)+(std1235:ncpus=24:mem=100663296kb)+(std1236:ncpus=24:mem=100663296kb)+(std1237:ncpus=24:mem=100663296kb)+(std1238:ncpus=24:mem=100663296kb)+(std1270:ncpus=24:mem=100663296kb)+(std1271:ncpus=24:mem=100663296kb)+(std1272:ncpus=24:mem=100663296kb)+(std1319:ncpus=24:mem=100663296kb)+(std1320:ncpus=24:mem=100663296kb) ====================================================================================== From mfadams at lbl.gov Sat Apr 21 14:19:36 2018 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 21 Apr 2018 15:19:36 -0400 Subject: [petsc-users] Scaling problem when cores > 600 In-Reply-To: References: <26ed65df-90d0-1f27-5d66-b6488bef3b90@gmail.com> <5a76371a-0eb1-f204-4974-549865ce726d@gmail.com> <35ace3a0-1e35-1e66-6482-98fe2483f24e@gmail.com> <16EEAEE0-BF68-4DF9-B0BE-8DF5D25ECF63@mcs.anl.gov> <4043d81a-8bbf-03be-c536-ae28d6ebefb2@gmail.com> <9346FFA5-9385-4EDD-9C4D-2232CCB3F9F9@mcs.anl.gov> Message-ID: looks fine to me. On Sat, Apr 21, 2018 at 11:34 AM, TAY wee-beng wrote: > Hi, > > I have found some time to work on this scaling problem again. I am now > using: > > mpirun ./a.out -log_view -poisson_pc_type gamg > -poisson_pc_gamg_agg_nsmooths 1 > > I have attached the log_view output for 288, 600, 960, 1440 procs for > comparison. > > Please give some comments. > > > Thank you very much > > Yours sincerely, > > ================================================ > TAY Wee-Beng ??? (Zheng Weiming) > Personal research webpage: http://tayweebeng.wixsite.com/website > Youtube research showcase: https://www.youtube.com/channe > l/UC72ZHtvQNMpNs2uRTSToiLA > linkedin: www.linkedin.com/in/tay-weebeng > ================================================ > > On 7/3/2018 11:58 PM, Smith, Barry F. wrote: > >> What are you using for Poisson log. >> >> If it is a Poisson problem then almost for sure you should be using >> Hypre BoomerAMG?. >> >> It sounds like your matrix does not change. You will need to discuss >> the scaling with the hypre people. >> >> Barry >> >> >> On Mar 7, 2018, at 5:38 AM, TAY wee-beng wrote: >>> >>> >>> On 7/3/2018 6:22 AM, Smith, Barry F. wrote: >>> >>>> The speed up for "Poisson log" is 1.6425364214878704 = >>>> 5.0848e+02/3.0957e+02 >>>> >>>> This is lower than I would expect for Hypre BoomerAMG? >>>> >>>> Are you doing multiple solves with the same matrix with hypre or >>>> is each solve a new matrix? If each solve is a new matrix then you may be >>>> getting expected behavior since the multigrid AMG construction process does >>>> not scale as well as the application of AMG once it is constructed. >>>> >>>> I am forwarding to the hypre team since this is their expertise >>>> not ours >>>> >>>> Barry >>>> >>>> Hi, >>> >>> My LHS of the eqn does not change. Only the RHS changes at each time >>> step. So should this be expected? >>> >>> So maybe I should change to BoomerAMG and compare? >>> >>> Will PETSc GAMG give better performance? >>> >>> Also, I must add that I only partition in the x and y direction. Will >>> this be a factor? >>> >>> Thanks. >>> >>> On Mar 5, 2018, at 11:19 PM, TAY wee-beng wrote: >>>>> >>>>> >>>>> On 5/3/2018 11:43 AM, Smith, Barry F. wrote: >>>>> >>>>>> 360 process >>>>>> >>>>>> KSPSolve 99 1.0 2.6403e+02 1.0 6.67e+10 1.1 2.7e+05 >>>>>> 9.9e+05 5.1e+02 15100 17 42 19 15100 17 42 19 87401 >>>>>> >>>>>> 1920 processes >>>>>> >>>>>> KSPSolve 99 1.0 2.3184e+01 1.0 1.32e+10 1.2 1.5e+06 >>>>>> 4.3e+05 5.1e+02 4100 17 42 19 4100 17 42 19 967717 >>>>>> >>>>>> >>>>>> Ratio of number of processes 5.33 ratio of time for KSPSolve 11.388 >>>>>> so the time for the solve is scaling very well (extremely well actually). >>>>>> The problem is >>>>>> due to "other time" that is not in KSP solve. Note that the >>>>>> percentage of the total time in KSPSolve went from 15 percent of the >>>>>> runtime to 4 percent. This means something outside of KSPSolve is scaling >>>>>> very poorly. You will need to profile the rest of the code to determine >>>>>> where the time is being spent. PetscLogEventRegister() and >>>>>> PetscLogEventBegin/End() will be needed in your code. Already with 360 >>>>>> processes the linear solver is only taking 15 percent of the time. >>>>>> >>>>>> Barry >>>>>> >>>>>> Hi, >>>>> >>>>> I have attached the new logging results with the HYPRE Poisson eqn >>>>> solver. However, due to some problems, I am now using Intel 2018. Should be >>>>> quite similar to 2016 in terms of runtime. Using 360 processes can't work >>>>> this time, and I'm not sure why though. >>>>> >>>>>> On Mar 4, 2018, at 9:23 PM, TAY wee-beng wrote: >>>>>>> >>>>>>> >>>>>>> On 1/3/2018 12:14 PM, Smith, Barry F. wrote: >>>>>>> >>>>>>>> On Feb 28, 2018, at 8:01 PM, TAY wee-beng wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 1/3/2018 12:10 AM, Matthew Knepley wrote: >>>>>>>>> >>>>>>>>>> On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng >>>>>>>>>> wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have a CFD code which uses PETSc and HYPRE. I found that for a >>>>>>>>>> certain case with grid size of 192,570,048, I encounter scaling problem >>>>>>>>>> when my cores > 600. At 600 cores, the code took 10min for 100 time steps. >>>>>>>>>> At 960, 1440 and 2880 cores, it still takes around 10min. At 360 cores, it >>>>>>>>>> took 15min. >>>>>>>>>> >>>>>>>>>> So how can I find the bottleneck? Any recommended steps? >>>>>>>>>> >>>>>>>>>> For any performance question, we need to see the output of >>>>>>>>>> -log_view for all test cases. >>>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> To be more specific, I use PETSc KSPBCGS and HYPRE geometric >>>>>>>>> multigrid (entirely based on HYPRE, no PETSc) for the momentum and Poisson >>>>>>>>> eqns in my code. >>>>>>>>> >>>>>>>>> So can log_view be used in this case to give a meaningful? Since >>>>>>>>> part of the code uses HYPRE? >>>>>>>>> >>>>>>>> Yes, just send the logs. >>>>>>>> >>>>>>>> Hi, >>>>>>> >>>>>>> I have attached the logs, with the number indicating the no. of >>>>>>> cores used. Some of the new results are different from the previous runs, >>>>>>> although I'm using the same cluster. >>>>>>> >>>>>>> Thanks for the help. >>>>>>> >>>>>>>> I also program another subroutine in the past which uses PETSc to >>>>>>>>> solve the Poisson eqn. It uses either HYPRE's boomeramg, KSPBCGS or >>>>>>>>> KSPGMRES. >>>>>>>>> >>>>>>>>> If I use boomeramg, can log_view be used in this case? >>>>>>>>> >>>>>>>>> Or do I have to use KSPBCGS or KSPGMRES, which is directly from >>>>>>>>> PETSc? However, I ran KSPGMRES yesterday with the Poisson eqn and my ans >>>>>>>>> didn't converge. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>>> I must also mention that I partition my grid only in the x and >>>>>>>>>> y direction. There is no partitioning in the z direction due to limited >>>>>>>>>> code development. I wonder if there is a strong effect in this case. >>>>>>>>>> >>>>>>>>>> Maybe. Usually what happens is you fill up memory with a z-column >>>>>>>>>> and cannot scale further. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> -- >>>>>>>>>> Thank you very much >>>>>>>>>> >>>>>>>>>> Yours sincerely, >>>>>>>>>> >>>>>>>>>> ================================================ >>>>>>>>>> TAY Wee-Beng ??? (Zheng Weiming) >>>>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>>>>>> Youtube research showcase: https://www.youtube.com/channe >>>>>>>>>> l/UC72ZHtvQNMpNs2uRTSToiLA >>>>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>>>>>> ================================================ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Apr 21 16:22:37 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sat, 21 Apr 2018 21:22:37 +0000 Subject: [petsc-users] Scaling problem when cores > 600 In-Reply-To: References: <26ed65df-90d0-1f27-5d66-b6488bef3b90@gmail.com> <5a76371a-0eb1-f204-4974-549865ce726d@gmail.com> <35ace3a0-1e35-1e66-6482-98fe2483f24e@gmail.com> <16EEAEE0-BF68-4DF9-B0BE-8DF5D25ECF63@mcs.anl.gov> <4043d81a-8bbf-03be-c536-ae28d6ebefb2@gmail.com> <9346FFA5-9385-4EDD-9C4D-2232CCB3F9F9@mcs.anl.gov> Message-ID: Comparing the time in KSPSolve() >>> 1.8116e+02/3.5276e+01 5.135502891484295 to the increase in number of processes one sees >>> 1440/288. 5.0 so very very good speedup. So the linear solver is working very well. The problem is the rest of the code. On 288 processes the code is spending 72 percent of the runtime in KSPSolve while on 1440 processes it is spending 33 percent of the time. This means something OUTSIDE of the linear solver is not scaling well. No way for me to know where the rest of the time is, could be in your code reading in meshes, or in your code partitioning or something else. You might use PetscLogEventRegister() and PetscLogEventBegin/End() to profile your own code and see where it is spending so much time. Barry > On Apr 21, 2018, at 10:34 AM, TAY wee-beng wrote: > > Hi, > > I have found some time to work on this scaling problem again. I am now using: > > mpirun ./a.out -log_view -poisson_pc_type gamg -poisson_pc_gamg_agg_nsmooths 1 > > I have attached the log_view output for 288, 600, 960, 1440 procs for comparison. > > Please give some comments. > > > Thank you very much > > Yours sincerely, > > ================================================ > TAY Wee-Beng ??? (Zheng Weiming) > Personal research webpage: http://tayweebeng.wixsite.com/website > Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA > linkedin: www.linkedin.com/in/tay-weebeng > ================================================ > > On 7/3/2018 11:58 PM, Smith, Barry F. wrote: >> What are you using for Poisson log. >> >> If it is a Poisson problem then almost for sure you should be using Hypre BoomerAMG?. >> >> It sounds like your matrix does not change. You will need to discuss the scaling with the hypre people. >> >> Barry >> >> >>> On Mar 7, 2018, at 5:38 AM, TAY wee-beng wrote: >>> >>> >>> On 7/3/2018 6:22 AM, Smith, Barry F. wrote: >>>> The speed up for "Poisson log" is 1.6425364214878704 = 5.0848e+02/3.0957e+02 >>>> >>>> This is lower than I would expect for Hypre BoomerAMG? >>>> >>>> Are you doing multiple solves with the same matrix with hypre or is each solve a new matrix? If each solve is a new matrix then you may be getting expected behavior since the multigrid AMG construction process does not scale as well as the application of AMG once it is constructed. >>>> >>>> I am forwarding to the hypre team since this is their expertise not ours >>>> >>>> Barry >>>> >>> Hi, >>> >>> My LHS of the eqn does not change. Only the RHS changes at each time step. So should this be expected? >>> >>> So maybe I should change to BoomerAMG and compare? >>> >>> Will PETSc GAMG give better performance? >>> >>> Also, I must add that I only partition in the x and y direction. Will this be a factor? >>> >>> Thanks. >>> >>>>> On Mar 5, 2018, at 11:19 PM, TAY wee-beng wrote: >>>>> >>>>> >>>>> On 5/3/2018 11:43 AM, Smith, Barry F. wrote: >>>>>> 360 process >>>>>> >>>>>> KSPSolve 99 1.0 2.6403e+02 1.0 6.67e+10 1.1 2.7e+05 9.9e+05 5.1e+02 15100 17 42 19 15100 17 42 19 87401 >>>>>> >>>>>> 1920 processes >>>>>> >>>>>> KSPSolve 99 1.0 2.3184e+01 1.0 1.32e+10 1.2 1.5e+06 4.3e+05 5.1e+02 4100 17 42 19 4100 17 42 19 967717 >>>>>> >>>>>> >>>>>> Ratio of number of processes 5.33 ratio of time for KSPSolve 11.388 so the time for the solve is scaling very well (extremely well actually). The problem is >>>>>> due to "other time" that is not in KSP solve. Note that the percentage of the total time in KSPSolve went from 15 percent of the runtime to 4 percent. This means something outside of KSPSolve is scaling very poorly. You will need to profile the rest of the code to determine where the time is being spent. PetscLogEventRegister() and PetscLogEventBegin/End() will be needed in your code. Already with 360 processes the linear solver is only taking 15 percent of the time. >>>>>> >>>>>> Barry >>>>>> >>>>> Hi, >>>>> >>>>> I have attached the new logging results with the HYPRE Poisson eqn solver. However, due to some problems, I am now using Intel 2018. Should be quite similar to 2016 in terms of runtime. Using 360 processes can't work this time, and I'm not sure why though. >>>>>>> On Mar 4, 2018, at 9:23 PM, TAY wee-beng wrote: >>>>>>> >>>>>>> >>>>>>> On 1/3/2018 12:14 PM, Smith, Barry F. wrote: >>>>>>>>> On Feb 28, 2018, at 8:01 PM, TAY wee-beng wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 1/3/2018 12:10 AM, Matthew Knepley wrote: >>>>>>>>>> On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have a CFD code which uses PETSc and HYPRE. I found that for a certain case with grid size of 192,570,048, I encounter scaling problem when my cores > 600. At 600 cores, the code took 10min for 100 time steps. At 960, 1440 and 2880 cores, it still takes around 10min. At 360 cores, it took 15min. >>>>>>>>>> >>>>>>>>>> So how can I find the bottleneck? Any recommended steps? >>>>>>>>>> >>>>>>>>>> For any performance question, we need to see the output of -log_view for all test cases. >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> To be more specific, I use PETSc KSPBCGS and HYPRE geometric multigrid (entirely based on HYPRE, no PETSc) for the momentum and Poisson eqns in my code. >>>>>>>>> >>>>>>>>> So can log_view be used in this case to give a meaningful? Since part of the code uses HYPRE? >>>>>>>> Yes, just send the logs. >>>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have attached the logs, with the number indicating the no. of cores used. Some of the new results are different from the previous runs, although I'm using the same cluster. >>>>>>> >>>>>>> Thanks for the help. >>>>>>>>> I also program another subroutine in the past which uses PETSc to solve the Poisson eqn. It uses either HYPRE's boomeramg, KSPBCGS or KSPGMRES. >>>>>>>>> >>>>>>>>> If I use boomeramg, can log_view be used in this case? >>>>>>>>> >>>>>>>>> Or do I have to use KSPBCGS or KSPGMRES, which is directly from PETSc? However, I ran KSPGMRES yesterday with the Poisson eqn and my ans didn't converge. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>>> I must also mention that I partition my grid only in the x and y direction. There is no partitioning in the z direction due to limited code development. I wonder if there is a strong effect in this case. >>>>>>>>>> >>>>>>>>>> Maybe. Usually what happens is you fill up memory with a z-column and cannot scale further. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> -- >>>>>>>>>> Thank you very much >>>>>>>>>> >>>>>>>>>> Yours sincerely, >>>>>>>>>> >>>>>>>>>> ================================================ >>>>>>>>>> TAY Wee-Beng ??? (Zheng Weiming) >>>>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>>>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>>>>>> ================================================ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>> > > From zonexo at gmail.com Sun Apr 22 11:42:01 2018 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 23 Apr 2018 00:42:01 +0800 Subject: [petsc-users] Scaling problem when cores > 600 In-Reply-To: References: <26ed65df-90d0-1f27-5d66-b6488bef3b90@gmail.com> <5a76371a-0eb1-f204-4974-549865ce726d@gmail.com> <35ace3a0-1e35-1e66-6482-98fe2483f24e@gmail.com> <16EEAEE0-BF68-4DF9-B0BE-8DF5D25ECF63@mcs.anl.gov> <4043d81a-8bbf-03be-c536-ae28d6ebefb2@gmail.com> <9346FFA5-9385-4EDD-9C4D-2232CCB3F9F9@mcs.anl.gov> Message-ID: <2d796db6-07a2-7b04-7813-b1a0f3ad423d@gmail.com> On 22/4/2018 5:22 AM, Smith, Barry F. wrote: > Comparing the time in KSPSolve() > >>>> 1.8116e+02/3.5276e+01 > 5.135502891484295 > > to the increase in number of processes one sees > >>>> 1440/288. > 5.0 > > so very very good speedup. So the linear solver is working very well. > > The problem is the rest of the code. On 288 processes the code is spending > 72 percent of the runtime in KSPSolve while on 1440 processes it is spending 33 percent of the time. This means something OUTSIDE of the linear solver is not scaling well. No way for me to know where the rest of the time is, could be in your code reading in meshes, or in your code partitioning or something else. You might use PetscLogEventRegister() and PetscLogEventBegin/End() to profile your own code and see where it is spending so much time. > > Barry Hi, Just to confirm, does the KSPSolve represents both the momentum and the Poisson eqn? I have 2 kspsolve, with the momentum eqn having: call KSPSetOptionsPrefix(ksp_semi_xyz,"momentum_",ierr) Poisson eqn having: call KSPSetOptionsPrefix(ksp,"poisson_",ierr) Is there anyway to separate their efficiency? Thanks. > > >> On Apr 21, 2018, at 10:34 AM, TAY wee-beng wrote: >> >> Hi, >> >> I have found some time to work on this scaling problem again. I am now using: >> >> mpirun ./a.out -log_view -poisson_pc_type gamg -poisson_pc_gamg_agg_nsmooths 1 >> >> I have attached the log_view output for 288, 600, 960, 1440 procs for comparison. >> >> Please give some comments. >> >> >> Thank you very much >> >> Yours sincerely, >> >> ================================================ >> TAY Wee-Beng ??? (Zheng Weiming) >> Personal research webpage: http://tayweebeng.wixsite.com/website >> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >> linkedin: www.linkedin.com/in/tay-weebeng >> ================================================ >> >> On 7/3/2018 11:58 PM, Smith, Barry F. wrote: >>> What are you using for Poisson log. >>> >>> If it is a Poisson problem then almost for sure you should be using Hypre BoomerAMG?. >>> >>> It sounds like your matrix does not change. You will need to discuss the scaling with the hypre people. >>> >>> Barry >>> >>> >>>> On Mar 7, 2018, at 5:38 AM, TAY wee-beng wrote: >>>> >>>> >>>> On 7/3/2018 6:22 AM, Smith, Barry F. wrote: >>>>> The speed up for "Poisson log" is 1.6425364214878704 = 5.0848e+02/3.0957e+02 >>>>> >>>>> This is lower than I would expect for Hypre BoomerAMG? >>>>> >>>>> Are you doing multiple solves with the same matrix with hypre or is each solve a new matrix? If each solve is a new matrix then you may be getting expected behavior since the multigrid AMG construction process does not scale as well as the application of AMG once it is constructed. >>>>> >>>>> I am forwarding to the hypre team since this is their expertise not ours >>>>> >>>>> Barry >>>>> >>>> Hi, >>>> >>>> My LHS of the eqn does not change. Only the RHS changes at each time step. So should this be expected? >>>> >>>> So maybe I should change to BoomerAMG and compare? >>>> >>>> Will PETSc GAMG give better performance? >>>> >>>> Also, I must add that I only partition in the x and y direction. Will this be a factor? >>>> >>>> Thanks. >>>> >>>>>> On Mar 5, 2018, at 11:19 PM, TAY wee-beng wrote: >>>>>> >>>>>> >>>>>> On 5/3/2018 11:43 AM, Smith, Barry F. wrote: >>>>>>> 360 process >>>>>>> >>>>>>> KSPSolve 99 1.0 2.6403e+02 1.0 6.67e+10 1.1 2.7e+05 9.9e+05 5.1e+02 15100 17 42 19 15100 17 42 19 87401 >>>>>>> >>>>>>> 1920 processes >>>>>>> >>>>>>> KSPSolve 99 1.0 2.3184e+01 1.0 1.32e+10 1.2 1.5e+06 4.3e+05 5.1e+02 4100 17 42 19 4100 17 42 19 967717 >>>>>>> >>>>>>> >>>>>>> Ratio of number of processes 5.33 ratio of time for KSPSolve 11.388 so the time for the solve is scaling very well (extremely well actually). The problem is >>>>>>> due to "other time" that is not in KSP solve. Note that the percentage of the total time in KSPSolve went from 15 percent of the runtime to 4 percent. This means something outside of KSPSolve is scaling very poorly. You will need to profile the rest of the code to determine where the time is being spent. PetscLogEventRegister() and PetscLogEventBegin/End() will be needed in your code. Already with 360 processes the linear solver is only taking 15 percent of the time. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>> Hi, >>>>>> >>>>>> I have attached the new logging results with the HYPRE Poisson eqn solver. However, due to some problems, I am now using Intel 2018. Should be quite similar to 2016 in terms of runtime. Using 360 processes can't work this time, and I'm not sure why though. >>>>>>>> On Mar 4, 2018, at 9:23 PM, TAY wee-beng wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 1/3/2018 12:14 PM, Smith, Barry F. wrote: >>>>>>>>>> On Feb 28, 2018, at 8:01 PM, TAY wee-beng wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 1/3/2018 12:10 AM, Matthew Knepley wrote: >>>>>>>>>>> On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I have a CFD code which uses PETSc and HYPRE. I found that for a certain case with grid size of 192,570,048, I encounter scaling problem when my cores > 600. At 600 cores, the code took 10min for 100 time steps. At 960, 1440 and 2880 cores, it still takes around 10min. At 360 cores, it took 15min. >>>>>>>>>>> >>>>>>>>>>> So how can I find the bottleneck? Any recommended steps? >>>>>>>>>>> >>>>>>>>>>> For any performance question, we need to see the output of -log_view for all test cases. >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> To be more specific, I use PETSc KSPBCGS and HYPRE geometric multigrid (entirely based on HYPRE, no PETSc) for the momentum and Poisson eqns in my code. >>>>>>>>>> >>>>>>>>>> So can log_view be used in this case to give a meaningful? Since part of the code uses HYPRE? >>>>>>>>> Yes, just send the logs. >>>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have attached the logs, with the number indicating the no. of cores used. Some of the new results are different from the previous runs, although I'm using the same cluster. >>>>>>>> >>>>>>>> Thanks for the help. >>>>>>>>>> I also program another subroutine in the past which uses PETSc to solve the Poisson eqn. It uses either HYPRE's boomeramg, KSPBCGS or KSPGMRES. >>>>>>>>>> >>>>>>>>>> If I use boomeramg, can log_view be used in this case? >>>>>>>>>> >>>>>>>>>> Or do I have to use KSPBCGS or KSPGMRES, which is directly from PETSc? However, I ran KSPGMRES yesterday with the Poisson eqn and my ans didn't converge. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>>> I must also mention that I partition my grid only in the x and y direction. There is no partitioning in the z direction due to limited code development. I wonder if there is a strong effect in this case. >>>>>>>>>>> >>>>>>>>>>> Maybe. Usually what happens is you fill up memory with a z-column and cannot scale further. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> -- >>>>>>>>>>> Thank you very much >>>>>>>>>>> >>>>>>>>>>> Yours sincerely, >>>>>>>>>>> >>>>>>>>>>> ================================================ >>>>>>>>>>> TAY Wee-Beng ??? (Zheng Weiming) >>>>>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>>>>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>>>>>>> ================================================ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> >>>>>> >> From bsmith at mcs.anl.gov Sun Apr 22 11:53:22 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 22 Apr 2018 16:53:22 +0000 Subject: [petsc-users] Scaling problem when cores > 600 In-Reply-To: <2d796db6-07a2-7b04-7813-b1a0f3ad423d@gmail.com> References: <26ed65df-90d0-1f27-5d66-b6488bef3b90@gmail.com> <5a76371a-0eb1-f204-4974-549865ce726d@gmail.com> <35ace3a0-1e35-1e66-6482-98fe2483f24e@gmail.com> <16EEAEE0-BF68-4DF9-B0BE-8DF5D25ECF63@mcs.anl.gov> <4043d81a-8bbf-03be-c536-ae28d6ebefb2@gmail.com> <9346FFA5-9385-4EDD-9C4D-2232CCB3F9F9@mcs.anl.gov> <2d796db6-07a2-7b04-7813-b1a0f3ad423d@gmail.com> Message-ID: <9FB81CBE-CEB6-4010-8EEE-FA3A5A7F0622@mcs.anl.gov> KSPSolve would include all the KSP solves You can use PetscLogStageRegister(), PetscLogStagePush(), PetscLogStagePop(), to have the information about each solve displayed separately. Barry > On Apr 22, 2018, at 11:42 AM, TAY wee-beng wrote: > > > On 22/4/2018 5:22 AM, Smith, Barry F. wrote: >> Comparing the time in KSPSolve() >> >>>>> 1.8116e+02/3.5276e+01 >> 5.135502891484295 >> >> to the increase in number of processes one sees >> >>>>> 1440/288. >> 5.0 >> >> so very very good speedup. So the linear solver is working very well. >> >> The problem is the rest of the code. On 288 processes the code is spending >> 72 percent of the runtime in KSPSolve while on 1440 processes it is spending 33 percent of the time. This means something OUTSIDE of the linear solver is not scaling well. No way for me to know where the rest of the time is, could be in your code reading in meshes, or in your code partitioning or something else. You might use PetscLogEventRegister() and PetscLogEventBegin/End() to profile your own code and see where it is spending so much time. >> >> Barry > Hi, > > Just to confirm, does the KSPSolve represents both the momentum and the Poisson eqn? > > I have 2 kspsolve, with the momentum eqn having: > > call KSPSetOptionsPrefix(ksp_semi_xyz,"momentum_",ierr) > > Poisson eqn having: > > call KSPSetOptionsPrefix(ksp,"poisson_",ierr) > > Is there anyway to separate their efficiency? > > Thanks. > > >> >> >>> On Apr 21, 2018, at 10:34 AM, TAY wee-beng wrote: >>> >>> Hi, >>> >>> I have found some time to work on this scaling problem again. I am now using: >>> >>> mpirun ./a.out -log_view -poisson_pc_type gamg -poisson_pc_gamg_agg_nsmooths 1 >>> >>> I have attached the log_view output for 288, 600, 960, 1440 procs for comparison. >>> >>> Please give some comments. >>> >>> >>> Thank you very much >>> >>> Yours sincerely, >>> >>> ================================================ >>> TAY Wee-Beng ??? (Zheng Weiming) >>> Personal research webpage: http://tayweebeng.wixsite.com/website >>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>> linkedin: www.linkedin.com/in/tay-weebeng >>> ================================================ >>> >>> On 7/3/2018 11:58 PM, Smith, Barry F. wrote: >>>> What are you using for Poisson log. >>>> >>>> If it is a Poisson problem then almost for sure you should be using Hypre BoomerAMG?. >>>> >>>> It sounds like your matrix does not change. You will need to discuss the scaling with the hypre people. >>>> >>>> Barry >>>> >>>> >>>>> On Mar 7, 2018, at 5:38 AM, TAY wee-beng wrote: >>>>> >>>>> >>>>> On 7/3/2018 6:22 AM, Smith, Barry F. wrote: >>>>>> The speed up for "Poisson log" is 1.6425364214878704 = 5.0848e+02/3.0957e+02 >>>>>> >>>>>> This is lower than I would expect for Hypre BoomerAMG? >>>>>> >>>>>> Are you doing multiple solves with the same matrix with hypre or is each solve a new matrix? If each solve is a new matrix then you may be getting expected behavior since the multigrid AMG construction process does not scale as well as the application of AMG once it is constructed. >>>>>> >>>>>> I am forwarding to the hypre team since this is their expertise not ours >>>>>> >>>>>> Barry >>>>>> >>>>> Hi, >>>>> >>>>> My LHS of the eqn does not change. Only the RHS changes at each time step. So should this be expected? >>>>> >>>>> So maybe I should change to BoomerAMG and compare? >>>>> >>>>> Will PETSc GAMG give better performance? >>>>> >>>>> Also, I must add that I only partition in the x and y direction. Will this be a factor? >>>>> >>>>> Thanks. >>>>> >>>>>>> On Mar 5, 2018, at 11:19 PM, TAY wee-beng wrote: >>>>>>> >>>>>>> >>>>>>> On 5/3/2018 11:43 AM, Smith, Barry F. wrote: >>>>>>>> 360 process >>>>>>>> >>>>>>>> KSPSolve 99 1.0 2.6403e+02 1.0 6.67e+10 1.1 2.7e+05 9.9e+05 5.1e+02 15100 17 42 19 15100 17 42 19 87401 >>>>>>>> >>>>>>>> 1920 processes >>>>>>>> >>>>>>>> KSPSolve 99 1.0 2.3184e+01 1.0 1.32e+10 1.2 1.5e+06 4.3e+05 5.1e+02 4100 17 42 19 4100 17 42 19 967717 >>>>>>>> >>>>>>>> >>>>>>>> Ratio of number of processes 5.33 ratio of time for KSPSolve 11.388 so the time for the solve is scaling very well (extremely well actually). The problem is >>>>>>>> due to "other time" that is not in KSP solve. Note that the percentage of the total time in KSPSolve went from 15 percent of the runtime to 4 percent. This means something outside of KSPSolve is scaling very poorly. You will need to profile the rest of the code to determine where the time is being spent. PetscLogEventRegister() and PetscLogEventBegin/End() will be needed in your code. Already with 360 processes the linear solver is only taking 15 percent of the time. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have attached the new logging results with the HYPRE Poisson eqn solver. However, due to some problems, I am now using Intel 2018. Should be quite similar to 2016 in terms of runtime. Using 360 processes can't work this time, and I'm not sure why though. >>>>>>>>> On Mar 4, 2018, at 9:23 PM, TAY wee-beng wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 1/3/2018 12:14 PM, Smith, Barry F. wrote: >>>>>>>>>>> On Feb 28, 2018, at 8:01 PM, TAY wee-beng wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 1/3/2018 12:10 AM, Matthew Knepley wrote: >>>>>>>>>>>> On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng wrote: >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I have a CFD code which uses PETSc and HYPRE. I found that for a certain case with grid size of 192,570,048, I encounter scaling problem when my cores > 600. At 600 cores, the code took 10min for 100 time steps. At 960, 1440 and 2880 cores, it still takes around 10min. At 360 cores, it took 15min. >>>>>>>>>>>> >>>>>>>>>>>> So how can I find the bottleneck? Any recommended steps? >>>>>>>>>>>> >>>>>>>>>>>> For any performance question, we need to see the output of -log_view for all test cases. >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> To be more specific, I use PETSc KSPBCGS and HYPRE geometric multigrid (entirely based on HYPRE, no PETSc) for the momentum and Poisson eqns in my code. >>>>>>>>>>> >>>>>>>>>>> So can log_view be used in this case to give a meaningful? Since part of the code uses HYPRE? >>>>>>>>>> Yes, just send the logs. >>>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I have attached the logs, with the number indicating the no. of cores used. Some of the new results are different from the previous runs, although I'm using the same cluster. >>>>>>>>> >>>>>>>>> Thanks for the help. >>>>>>>>>>> I also program another subroutine in the past which uses PETSc to solve the Poisson eqn. It uses either HYPRE's boomeramg, KSPBCGS or KSPGMRES. >>>>>>>>>>> >>>>>>>>>>> If I use boomeramg, can log_view be used in this case? >>>>>>>>>>> >>>>>>>>>>> Or do I have to use KSPBCGS or KSPGMRES, which is directly from PETSc? However, I ran KSPGMRES yesterday with the Poisson eqn and my ans didn't converge. >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>>> I must also mention that I partition my grid only in the x and y direction. There is no partitioning in the z direction due to limited code development. I wonder if there is a strong effect in this case. >>>>>>>>>>>> >>>>>>>>>>>> Maybe. Usually what happens is you fill up memory with a z-column and cannot scale further. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Matt >>>>>>>>>>>> -- >>>>>>>>>>>> Thank you very much >>>>>>>>>>>> >>>>>>>>>>>> Yours sincerely, >>>>>>>>>>>> >>>>>>>>>>>> ================================================ >>>>>>>>>>>> TAY Wee-Beng ??? (Zheng Weiming) >>>>>>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>>>>>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>>>>>>>> ================================================ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>> >>>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> >>>>>>> >>> From zinlin.zinlin at gmail.com Sun Apr 22 13:13:13 2018 From: zinlin.zinlin at gmail.com (Zin Lin) Date: Sun, 22 Apr 2018 14:13:13 -0400 Subject: [petsc-users] Problems with VecGetArray under sub-communicators Message-ID: Hi I am experiencing possible memory issues with VecGetArray when it is used under sub-communicators (when I split the PETSC_COMM_WORLD to multiple subcomms). The following is the minimal code. Basically, you can test that if you parallelize the vector to more than one processor under a subcomm, the array obtained from the VecGetArray call doesn't seem to be correct. Please test it with 1) mpirun -np 1 ./example -ncomms 1 2) mpirun -np 2 ./example -ncomms 2 3) mpirun -np 2 ./example -ncomms 1 4) mpirun -np 4 ./example -ncomms 2 you will 1) and 2) work as expected while in 3) and 4) some entries of the array are assigned erroneous values. Any input will be appreciated. Thanks Zin Minimal Code PetscErrorCode main(int argc, char **argv) { MPI_Init(NULL, NULL); PetscInitialize(&argc,&argv,NULL,NULL); int size; MPI_Comm_size(MPI_COMM_WORLD, &size); PetscPrintf(PETSC_COMM_WORLD,"\tThe total number of processors is %d\n",size); //check if the number of processors is divisible by the number of subcomms int ncomms, np_per_comm; PetscOptionsGetInt(NULL,"-ncomms",&ncomms,NULL); if(!(size%ncomms==0)) SETERRQ(PETSC_COMM_WORLD,1,"The number of processes must be a multiple of ncomms so that it is divisible by the number of subcomms."); np_per_comm=size/ncomms; PetscPrintf(PETSC_COMM_WORLD,"\tThe number of subcomms is %d.\n\tEach subcomm has %d processors.\n",ncomms,np_per_comm); //calculate the colour of each subcomm ( = rank of each processor / number of processors in each subcomm ) //note once calculated, the colour is fixed throughout the entire run int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm subcomm; int colour = rank/np_per_comm; MPI_Comm_split(MPI_COMM_WORLD, colour, rank, &subcomm); Vec u; PetscScalar *_u; int i,ns,ne; PetscScalar tmp; VecCreateMPI(subcomm,PETSC_DECIDE,10,&u); VecSet(u,1.0+PETSC_i*0); VecGetArray(u,&_u); VecGetOwnershipRange(u,&ns,&ne); for(i=ns;i From dave.mayhem23 at gmail.com Sun Apr 22 14:14:36 2018 From: dave.mayhem23 at gmail.com (Dave May) Date: Sun, 22 Apr 2018 19:14:36 +0000 Subject: [petsc-users] Problems with VecGetArray under sub-communicators In-Reply-To: References: Message-ID: On Sun, 22 Apr 2018 at 20:13, Zin Lin wrote: > Hi > I am experiencing possible memory issues with VecGetArray when it is used > under sub-communicators (when I split the PETSC_COMM_WORLD to multiple > subcomms). The following is the minimal code. Basically, you can test that > if you parallelize the vector to more than one processor under a subcomm, > the array obtained from the VecGetArray call doesn't seem to be correct. > Please test it with > > 1) mpirun -np 1 ./example -ncomms 1 > 2) mpirun -np 2 ./example -ncomms 2 > 3) mpirun -np 2 ./example -ncomms 1 > 4) mpirun -np 4 ./example -ncomms 2 > > you will 1) and 2) work as expected while in 3) and 4) some entries of the > array are assigned erroneous values. > First - your call to PetscPrintf contains a mistake. The second instance of i is missing. Second (more important), you cannot access the raw array obtained via VecGetArray() using global indices. You must use local indices. Change the access to _u[i-ns] and the code should work. Also, debugging based on what printf() tells you can be misleading. It's much better to use valgrind - see here https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Thanks, Dave > Any input will be appreciated. > Thanks > Zin > > Minimal Code > > PetscErrorCode main(int argc, char **argv) > { > > MPI_Init(NULL, NULL); > PetscInitialize(&argc,&argv,NULL,NULL); > int size; > MPI_Comm_size(MPI_COMM_WORLD, &size); > PetscPrintf(PETSC_COMM_WORLD,"\tThe total number of processors is > %d\n",size); > > //check if the number of processors is divisible by the number of > subcomms > int ncomms, np_per_comm; > PetscOptionsGetInt(NULL,"-ncomms",&ncomms,NULL); > if(!(size%ncomms==0)) SETERRQ(PETSC_COMM_WORLD,1,"The number of > processes must be a multiple of ncomms so that it is divisible by the > number of subcomms."); > np_per_comm=size/ncomms; > PetscPrintf(PETSC_COMM_WORLD,"\tThe number of subcomms is %d.\n\tEach > subcomm has %d processors.\n",ncomms,np_per_comm); > > //calculate the colour of each subcomm ( = rank of each processor / > number of processors in each subcomm ) > //note once calculated, the colour is fixed throughout the entire run > int rank; > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > MPI_Comm subcomm; > int colour = rank/np_per_comm; > MPI_Comm_split(MPI_COMM_WORLD, colour, rank, &subcomm); > > Vec u; > PetscScalar *_u; > int i,ns,ne; > PetscScalar tmp; > > VecCreateMPI(subcomm,PETSC_DECIDE,10,&u); > VecSet(u,1.0+PETSC_i*0); > > VecGetArray(u,&_u); > VecGetOwnershipRange(u,&ns,&ne); > for(i=ns;i VecGetValues(u,1,&i,&tmp); > PetscPrintf(PETSC_COMM_SELF,"colour %d, u[%d]_array = %g + i * (%g), > u[%d]_vec = %g + i * %g \n",colour,i,creal(_u[i]),cimag(_u[i]), > Insert i here creal(tmp),cimag(tmp)); > } > VecRestoreArray(u,&_u); > > PetscFinalize(); > return 0; > > } > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zinlin.zinlin at gmail.com Sun Apr 22 14:34:30 2018 From: zinlin.zinlin at gmail.com (Zin Lin) Date: Sun, 22 Apr 2018 15:34:30 -0400 Subject: [petsc-users] Problems with VecGetArray under sub-communicators In-Reply-To: References: Message-ID: Thanks a lot! That clears things up. I do have one more question though. The behavior of DMDAVecGetArray is different then? From ex3.c (below), it doesn't seem like the raw array needs the local indices? >From ex3.c, DMDAVecGetArray (cda,global,&coors); ... DMDAGetCorners (cda,&mstart,&nstart,&pstart,&m,&n,&p); ... for (i=mstart; i wrote: > > On Sun, 22 Apr 2018 at 20:13, Zin Lin wrote: > >> Hi >> I am experiencing possible memory issues with VecGetArray when it is used >> under sub-communicators (when I split the PETSC_COMM_WORLD to multiple >> subcomms). The following is the minimal code. Basically, you can test that >> if you parallelize the vector to more than one processor under a subcomm, >> the array obtained from the VecGetArray call doesn't seem to be correct. >> Please test it with >> >> 1) mpirun -np 1 ./example -ncomms 1 >> 2) mpirun -np 2 ./example -ncomms 2 >> 3) mpirun -np 2 ./example -ncomms 1 >> 4) mpirun -np 4 ./example -ncomms 2 >> >> you will 1) and 2) work as expected while in 3) and 4) some entries of >> the array are assigned erroneous values. >> > > First - your call to PetscPrintf contains a mistake. The second instance > of i is missing. > > Second (more important), you cannot access the raw array obtained via > VecGetArray() using global indices. You must use local indices. > Change the access to _u[i-ns] and the code should work. > > Also, debugging based on what printf() tells you can be misleading. It's > much better to use valgrind - see here > > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > Thanks, > Dave > > >> Any input will be appreciated. >> Thanks >> Zin >> >> Minimal Code >> >> PetscErrorCode main(int argc, char **argv) >> { >> >> MPI_Init(NULL, NULL); >> PetscInitialize(&argc,&argv,NULL,NULL); >> int size; >> MPI_Comm_size(MPI_COMM_WORLD, &size); >> PetscPrintf(PETSC_COMM_WORLD,"\tThe total number of processors is >> %d\n",size); >> >> //check if the number of processors is divisible by the number of >> subcomms >> int ncomms, np_per_comm; >> PetscOptionsGetInt(NULL,"-ncomms",&ncomms,NULL); >> if(!(size%ncomms==0)) SETERRQ(PETSC_COMM_WORLD,1,"The number of >> processes must be a multiple of ncomms so that it is divisible by the >> number of subcomms."); >> np_per_comm=size/ncomms; >> PetscPrintf(PETSC_COMM_WORLD,"\tThe number of subcomms is %d.\n\tEach >> subcomm has %d processors.\n",ncomms,np_per_comm); >> >> //calculate the colour of each subcomm ( = rank of each processor / >> number of processors in each subcomm ) >> //note once calculated, the colour is fixed throughout the entire run >> int rank; >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >> MPI_Comm subcomm; >> int colour = rank/np_per_comm; >> MPI_Comm_split(MPI_COMM_WORLD, colour, rank, &subcomm); >> >> Vec u; >> PetscScalar *_u; >> int i,ns,ne; >> PetscScalar tmp; >> >> VecCreateMPI(subcomm,PETSC_DECIDE,10,&u); >> VecSet(u,1.0+PETSC_i*0); >> >> VecGetArray(u,&_u); >> VecGetOwnershipRange(u,&ns,&ne); >> for(i=ns;i> VecGetValues(u,1,&i,&tmp); >> PetscPrintf(PETSC_COMM_SELF,"colour %d, u[%d]_array = %g + i * (%g), >> u[%d]_vec = %g + i * %g \n",colour,i,creal(_u[i]),cimag(_u[i]), >> > > Insert i here > > creal(tmp),cimag(tmp)); >> } >> VecRestoreArray(u,&_u); >> >> PetscFinalize(); >> return 0; >> >> } >> >> -- Zin Lin -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Apr 22 14:39:36 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 22 Apr 2018 19:39:36 +0000 Subject: [petsc-users] Problems with VecGetArray under sub-communicators In-Reply-To: References: Message-ID: DMDAVecGetArray() uses global indices while VecGetArray() uses local indices. Barry > On Apr 22, 2018, at 2:34 PM, Zin Lin wrote: > > Thanks a lot! > That clears things up. > I do have one more question though. > The behavior of DMDAVecGetArray is different then? From ex3.c (below), it doesn't seem like the raw array needs the local indices? > > From ex3.c, > c(cda,global,&coors); > ... > DMDAGetCorners(cda,&mstart,&nstart,&pstart,&m,&n,&p); > ... > for (i=mstart; i for (j=nstart; j for (k=pstart; k ... > coors[k][j][i].x = ... > > > > > > > > > > > > On Sun, Apr 22, 2018 at 3:14 PM, Dave May wrote: > > On Sun, 22 Apr 2018 at 20:13, Zin Lin wrote: > Hi > I am experiencing possible memory issues with VecGetArray when it is used under sub-communicators (when I split the PETSC_COMM_WORLD to multiple subcomms). The following is the minimal code. Basically, you can test that if you parallelize the vector to more than one processor under a subcomm, the array obtained from the VecGetArray call doesn't seem to be correct. > Please test it with > > 1) mpirun -np 1 ./example -ncomms 1 > 2) mpirun -np 2 ./example -ncomms 2 > 3) mpirun -np 2 ./example -ncomms 1 > 4) mpirun -np 4 ./example -ncomms 2 > > you will 1) and 2) work as expected while in 3) and 4) some entries of the array are assigned erroneous values. > > First - your call to PetscPrintf contains a mistake. The second instance of i is missing. > > Second (more important), you cannot access the raw array obtained via VecGetArray() using global indices. You must use local indices. > Change the access to _u[i-ns] and the code should work. > > Also, debugging based on what printf() tells you can be misleading. It's much better to use valgrind - see here > > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > Thanks, > Dave > > > Any input will be appreciated. > Thanks > Zin > > Minimal Code > > PetscErrorCode main(int argc, char **argv) > { > > MPI_Init(NULL, NULL); > PetscInitialize(&argc,&argv,NULL,NULL); > int size; > MPI_Comm_size(MPI_COMM_WORLD, &size); > PetscPrintf(PETSC_COMM_WORLD,"\tThe total number of processors is %d\n",size); > > //check if the number of processors is divisible by the number of subcomms > int ncomms, np_per_comm; > PetscOptionsGetInt(NULL,"-ncomms",&ncomms,NULL); > if(!(size%ncomms==0)) SETERRQ(PETSC_COMM_WORLD,1,"The number of processes must be a multiple of ncomms so that it is divisible by the number of subcomms."); > np_per_comm=size/ncomms; > PetscPrintf(PETSC_COMM_WORLD,"\tThe number of subcomms is %d.\n\tEach subcomm has %d processors.\n",ncomms,np_per_comm); > > //calculate the colour of each subcomm ( = rank of each processor / number of processors in each subcomm ) > //note once calculated, the colour is fixed throughout the entire run > int rank; > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > MPI_Comm subcomm; > int colour = rank/np_per_comm; > MPI_Comm_split(MPI_COMM_WORLD, colour, rank, &subcomm); > > Vec u; > PetscScalar *_u; > int i,ns,ne; > PetscScalar tmp; > > VecCreateMPI(subcomm,PETSC_DECIDE,10,&u); > VecSet(u,1.0+PETSC_i*0); > > VecGetArray(u,&_u); > VecGetOwnershipRange(u,&ns,&ne); > for(i=ns;i VecGetValues(u,1,&i,&tmp); > PetscPrintf(PETSC_COMM_SELF,"colour %d, u[%d]_array = %g + i * (%g), u[%d]_vec = %g + i * %g \n",colour,i,creal(_u[i]),cimag(_u[i]), > > Insert i here > > creal(tmp),cimag(tmp)); > } > VecRestoreArray(u,&_u); > > PetscFinalize(); > return 0; > > } > > > > > -- > Zin Lin > From aliberkkahraman at yahoo.com Mon Apr 23 02:39:31 2018 From: aliberkkahraman at yahoo.com (Ali Berk Kahraman) Date: Mon, 23 Apr 2018 10:39:31 +0300 Subject: [petsc-users] Possible DIVERGED_BREAKDOWN reasons and ways to avoid it Message-ID: Dear All, I am trying to solve a Poisson-Neumann Problem which does not have a solution by itself on an irregular domain with finite differences. This problem arises in the correction step of a veloicty prediction-correction scheme of Navier-Stokes equation. It requires a least squares solution, or a modified solution. I have asked here before, and with substantial help of Dr. Smith I have learned about KSPLSQR. It helped for some small-sized problems, but when the problem size gets larger, for some reason it cannot converge to a good spot. I have been trying another method in the literature, with little success again. The literature suggests that an augmented linear system which has a solution should be solved. If the poisson neumann problem is Ax=b, the augmented system is as follows, |A?? 1|??? |b| |?????? | = |? | |1?? 0|???? |a| where a is an arbitrary number.? This system has a solution since vector 1 is the right null space of A (or so I have read). However, when I try to solve thiss system for larger system sizes than appr. 8000, it causes a diverged breakdown error for BiCGS with jacobi preconditioning. Is there any way to avoid a breakdown in BiCGS? Why might it happen? Best Regards, Ali Berk Kahraman Bogazici Uni., Istanbul,Turkey From loic.gouarin at math.u-psud.fr Mon Apr 23 05:32:42 2018 From: loic.gouarin at math.u-psud.fr (Loic Gouarin) Date: Mon, 23 Apr 2018 12:32:42 +0200 Subject: [petsc-users] MATIS and DMDA in petsc4py Message-ID: <9faeb719-6e56-3f99-b29c-2978f1149c83@math.u-psud.fr> Hi, I try to use MATIS from petsc4py but I am not able to use setValuesLocal (I have a segmentation fault). And I don't understand why. Could you tell me if I did a mistake or if it's a bug ? My example ======= from petsc4py import PETSc import numpy as np n = 10 da = PETSc.DMDA().create([n, n], dof=1, stencil_width=1) da.setMatType(PETSc.Mat.Type.IS) A = da.createMatrix() #ltog = da.getLGMap() #A.setLGMap(ltog, ltog) elem = da.getElements() melem = np.ones((4, 4)) for e in elem: ??? A.setValuesLocal(e, e, melem, PETSc.InsertMode.ADD_VALUES) Thanks, Loic -- Tel: 01 69 15 60 14 http://www.math.u-psud.fr/~gouarin https://github.com/gouarin From lawrence.mitchell at imperial.ac.uk Mon Apr 23 05:36:22 2018 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Mon, 23 Apr 2018 11:36:22 +0100 Subject: [petsc-users] MATIS and DMDA in petsc4py In-Reply-To: <9faeb719-6e56-3f99-b29c-2978f1149c83@math.u-psud.fr> References: <9faeb719-6e56-3f99-b29c-2978f1149c83@math.u-psud.fr> Message-ID: > On 23 Apr 2018, at 11:32, Loic Gouarin wrote: > > Hi, > > I try to use MATIS from petsc4py but I am not able to use setValuesLocal (I have a segmentation fault). And I don't understand why. > > Could you tell me if I did a mistake or if it's a bug ? This code works for me with: PETSc: 7bfdb6492ce petsc4py: 9137bf78e Perhaps you have a mismatching petsc4py/PETSc combo? Lawrence > My example > ======= > from petsc4py import PETSc > import numpy as np > > n = 10 > da = PETSc.DMDA().create([n, n], dof=1, stencil_width=1) > da.setMatType(PETSc.Mat.Type.IS) > > A = da.createMatrix() > #ltog = da.getLGMap() > #A.setLGMap(ltog, ltog) > > elem = da.getElements() > melem = np.ones((4, 4)) > > for e in elem: > A.setValuesLocal(e, e, melem, PETSc.InsertMode.ADD_VALUES) > > Thanks, > Loic > > -- > Tel: 01 69 15 60 14 > http://www.math.u-psud.fr/~gouarin > https://github.com/gouarin > From loic.gouarin at math.u-psud.fr Mon Apr 23 06:25:11 2018 From: loic.gouarin at math.u-psud.fr (Loic Gouarin) Date: Mon, 23 Apr 2018 13:25:11 +0200 Subject: [petsc-users] MATIS and DMDA in petsc4py In-Reply-To: References: <9faeb719-6e56-3f99-b29c-2978f1149c83@math.u-psud.fr> Message-ID: Thanks Lawrence for your reply. Le 23/04/2018 ? 12:36, Lawrence Mitchell a ?crit?: > >> On 23 Apr 2018, at 11:32, Loic Gouarin wrote: >> >> Hi, >> >> I try to use MATIS from petsc4py but I am not able to use setValuesLocal (I have a segmentation fault). And I don't understand why. >> >> Could you tell me if I did a mistake or if it's a bug ? > This code works for me with: > > PETSc: 7bfdb6492ce > petsc4py: 9137bf78e > > Perhaps you have a mismatching petsc4py/PETSc combo? The problem probably comes from the conda package. I reinstalled a new environment and tried again but I still have this issue. Did you compile petsc and petsc4py ? The versions that I use are those given on conda-forge : petsc-3.8.4 and petsc4py-3.8.1. I did the following thing conda create -n petsc_tmp source activate petsc_tmp conda install petsc4py -c conda-forge and execute my script. Loic > > Lawrence > >> My example >> ======= >> from petsc4py import PETSc >> import numpy as np >> >> n = 10 >> da = PETSc.DMDA().create([n, n], dof=1, stencil_width=1) >> da.setMatType(PETSc.Mat.Type.IS) >> >> A = da.createMatrix() >> #ltog = da.getLGMap() >> #A.setLGMap(ltog, ltog) >> >> elem = da.getElements() >> melem = np.ones((4, 4)) >> >> for e in elem: >> A.setValuesLocal(e, e, melem, PETSc.InsertMode.ADD_VALUES) >> >> Thanks, >> Loic >> >> -- >> Tel: 01 69 15 60 14 >> http://www.math.u-psud.fr/~gouarin >> https://github.com/gouarin >> -- Tel: 01 69 15 60 14 http://www.math.u-psud.fr/~gouarin https://github.com/gouarin -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.c.volmer at student.tue.nl Mon Apr 23 10:41:39 2018 From: j.c.volmer at student.tue.nl (Volmer, J.C.) Date: Mon, 23 Apr 2018 15:41:39 +0000 Subject: [petsc-users] snes ngmres default options Message-ID: Dear PETSc users, Could anyone tell me where I can find the default options for the NGMRES solver if none are provided? I have tried searching in the source code http://www.mcs.anl.gov/petsc/petsc-current/src/snes/impls/ngmres/snesngmres.c.html#SNESNGMRES but for example the parameter ?snes_ngmres_m? I could not deduce from there. All help is appreciated. Best regards, Jasper -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Apr 23 11:05:43 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 23 Apr 2018 16:05:43 +0000 Subject: [petsc-users] Possible DIVERGED_BREAKDOWN reasons and ways to avoid it In-Reply-To: References: Message-ID: <34873014-F93A-4262-B638-37B30EE6D7C1@anl.gov> What are the sizes of the entries of A? Instead of a 1 you could try to use a number proportional to the size of the entries in A. Also what more specifically goes wrong with KSPLSQR for larger problems? Does the convergence slow down too much? What preconditioner did you try? Barry > On Apr 23, 2018, at 2:39 AM, Ali Berk Kahraman wrote: > > Dear All, > > I am trying to solve a Poisson-Neumann Problem which does not have a solution by itself on an irregular domain with finite differences. This problem arises in the correction step of a veloicty prediction-correction scheme of Navier-Stokes equation. It requires a least squares solution, or a modified solution. > > I have asked here before, and with substantial help of Dr. Smith I have learned about KSPLSQR. It helped for some small-sized problems, but when the problem size gets larger, for some reason it cannot converge to a good spot. > > I have been trying another method in the literature, with little success again. The literature suggests that an augmented linear system which has a solution should be solved. If the poisson neumann problem is Ax=b, the augmented system is as follows, > > |A 1| |b| > | | = | | > |1 0| |a| > > where a is an arbitrary number. This system has a solution since vector 1 is the right null space of A (or so I have read). However, when I try to solve thiss system for larger system sizes than appr. 8000, it causes a diverged breakdown error for BiCGS with jacobi preconditioning. Is there any way to avoid a breakdown in BiCGS? Why might it happen? > > Best Regards, > > Ali Berk Kahraman > Bogazici Uni., Istanbul,Turkey > From knepley at gmail.com Mon Apr 23 12:22:44 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 23 Apr 2018 13:22:44 -0400 Subject: [petsc-users] snes ngmres default options In-Reply-To: References: Message-ID: On Mon, Apr 23, 2018 at 11:41 AM, Volmer, J.C. wrote: > Dear PETSc users, > > Could anyone tell me where I can find the default options for the NGMRES > solver if none are provided? > I have tried searching in the source code http://www.mcs.anl.gov/ > petsc/petsc-current/src/snes/impls/ngmres/snesngmres.c.html#SNESNGMRES > but for example the parameter ?snes_ngmres_m? I could not deduce from > there. > If you want to know what they are, you can use -snes_view. They are set here https://bitbucket.org/petsc/petsc/src/79b862a042ee9b723f68875412fde954bbcb6cb2/src/snes/impls/ngmres/snesngmres.c?at=master&fileviewer=file-view-default#snesngmres.c-547 Thanks, Matt > All help is appreciated. > > Best regards, > > Jasper > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.lanthaler at gmail.com Tue Apr 24 02:26:51 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Tue, 24 Apr 2018 09:26:51 +0200 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: References: Message-ID: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> Hi there, I was wondering if you could help me with the correct configuration of PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure which information would be useful to you, but basically the problem seems to be in correctly compiling it with intel compiler and the existing mkl library. The pre-installed mkl version is intel/17.0.4.196 I have tried various things and finally, I got it at least to compile with the following options (options chosen in reference to the mkl link advisor...): ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 --with-scalar-type=complex --download-scalapack --download-mumps=yes --download-superlu --with-batch --known-mpi-shared-libraries=1 *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_sequential.a ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl" *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" --known-64-bit-blas-indices=0 After compilation, when trying to compile /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get linking errors (attached). Would anyone be able to help me out here? I really don't have a good understanding of this. I'm attaching the configuration.log file, as well as the linking errors when trying to compile ex5f. Thank you very much in advance! Best regards, Samuel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ftn -c -O -I/users/lanthale/petsc-git/include -I/users/lanthale/petsc-git/arch-complex/include -I/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include -I/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64 -o ex5f.o ex5f.F90 ftn -O -o ex5f ex5f.o -L/users/lanthale/petsc-git/arch-complex/lib -Wl,-rpath,/users/lanthale/petsc-git/arch-complex/lib -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -Wl,--start-group -Wl,--end-group -lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl -lstdc++ -ldl /users/lanthale/petsc-git/arch-complex/lib/libpetsc.a(dlimpl.o): In function `PetscDLOpen': dlimpl.c:(.text+0x38): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /users/lanthale/petsc-git/arch-complex/lib/libpetsc.a(send.o): In function `PetscOpenSocket': send.c:(.text+0xcf): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zgeqrf_offload_thunks.o): In function `mkl_lapack_ao_zgeqrf': zgeqrf_offload.c:(.text+0x9f): undefined reference to `mkl_lapack_ao_zgeqrf_hao' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zpotrf_offload_thunks.o): In function `mkl_lapack_ao_zpotrf': zpotrf_offload.c:(.text+0x76): undefined reference to `mkl_lapack_ao_zpotrf_hao' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zgetrf_offload_thunks.o): In function `mkl_lapack_ao_zgetrf': zgetrf_offload.c:(.text+0x89): undefined reference to `mkl_lapack_ao_zgetrf_hao' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xzgeqrf.o): In function `mkl_lapack_xzgeqrf': xzgeqrf_gen.f:(.text+0x246): undefined reference to `mkl_lapack_zgeqrf_pfnr' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zsytrf.o): In function `mkl_lapack_zsytrf': zsytrf_gen.f:(.text+0x35b): undefined reference to `mkl_lapack_zlasyf' zsytrf_gen.f:(.text+0x5d1): undefined reference to `mkl_lapack_zlasyf' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zsteqr.o): In function `mkl_lapack_zsteqr': zsteqr_gen.f:(.text+0xd4e): undefined reference to `mkl_lapack_dlasr3' zsteqr_gen.f:(.text+0x1300): undefined reference to `mkl_lapack_dlasr3' zsteqr_gen.f:(.text+0x2138): undefined reference to `mkl_lapack_dlasr3' zsteqr_gen.f:(.text+0x3206): undefined reference to `mkl_lapack_dlasr3' zsteqr_gen.f:(.text+0x33a2): undefined reference to `mkl_lapack_dlasr3' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zsteqr.o):zsteqr_gen.f:(.text+0x353d): more undefined references to `mkl_lapack_dlasr3' follow /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zsteqr.o): In function `mkl_lapack_zsteqr': zsteqr_gen.f:(.text+0x3938): undefined reference to `mkl_lapack_zlaset' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zlaqrf.o): In function `mkl_lapack_zlaqrf': zlaqrf_gen.f:(.text+0x19c): undefined reference to `mkl_lapack_zgeqrf_pf' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zhetrf.o): In function `mkl_lapack_zhetrf': zhetrf_gen.f:(.text+0x35b): undefined reference to `mkl_lapack_zlahef' zhetrf_gen.f:(.text+0x5d1): undefined reference to `mkl_lapack_zlahef' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zheevx.o): In function `mkl_lapack_zheevx': zheevx_gen.f:(.text+0x664): undefined reference to `mkl_lapack_zlanhe' zheevx_gen.f:(.text+0x8e9): undefined reference to `mkl_lapack_zhetrd' zheevx_gen.f:(.text+0xa46): undefined reference to `mkl_lapack_zlacpy' zheevx_gen.f:(.text+0xfef): undefined reference to `mkl_blas_dscal' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zheev.o): In function `mkl_lapack_zheev': zheev_gen.f:(.text+0x267): undefined reference to `mkl_lapack_zhetrd' zheev_gen.f:(.text+0x4d2): undefined reference to `mkl_lapack_zlanhe' zheev_gen.f:(.text+0x5f5): undefined reference to `mkl_lapack_zhetrd' zheev_gen.f:(.text+0x78f): undefined reference to `mkl_blas_dscal' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zgetrs.o): In function `mkl_lapack_zgetrs': zgetrs_gen.f:(.text+0x2df): undefined reference to `mkl_lapack_zlaswp' zgetrs_gen.f:(.text+0x31c): undefined reference to `mkl_lapack_zlaswp' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zgesvd.o): In function `mkl_lapack_zgesvd': zgesvd_gen.f:(.text+0xda2): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0xdef): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0xe46): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x1720): undefined reference to `mkl_lapack_zlange' zgesvd_gen.f:(.text+0x1903): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x1afa): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x1c69): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x1cc0): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x2025): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x233e): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x239f): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x2501): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x27d9): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x288d): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x28ee): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x2cf4): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x2d4b): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x3062): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x3133): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x3449): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x34a0): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x3601): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x381c): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x38b0): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x3981): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x3ca6): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x3cfd): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x3e4e): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x40d2): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x416b): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x41cb): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x4543): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x4575): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x45cc): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x4889): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x491f): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x49f0): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x4d17): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x4daf): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x4e06): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x4f06): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x5128): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x5151): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x51e7): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x52b8): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x55df): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x5674): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x56cb): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x57b8): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x59dd): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x5a67): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x5afd): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x5b50): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x5ffc): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x60f2): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x64e0): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x6541): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x6893): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x6a05): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x6a37): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x6a96): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x6afc): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x6e13): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x711c): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x714c): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x71b6): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x721c): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x731d): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x75eb): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x7674): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x76a4): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x770e): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x7770): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x7adc): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x7b0e): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x7b6d): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x7bd3): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x7e5a): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x7e8a): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x7ef1): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x7f6c): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x8263): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x8295): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x82f4): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x8358): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x845e): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x867c): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x86df): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x870f): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x8776): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x87f1): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x8ae5): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x8b17): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x8b76): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x8bda): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x8cd1): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x8f29): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x8f59): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x8fc0): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x8ff0): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x905a): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x93b3): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x93e3): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x9415): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x9474): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x94d8): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x973a): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x979d): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x97cd): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x9834): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x98aa): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x9baf): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0x9bdf): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x9c43): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0x9c75): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x9cd4): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0x9ddb): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0x9ff5): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0xa01f): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0xa082): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0xa0b2): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0xa119): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0xa196): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0xa49c): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0xa4cc): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0xa52d): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0xa55f): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0xa5b8): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0xa6ab): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0xa8d5): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0xa933): undefined reference to `mkl_lapack_zgelqf' zgesvd_gen.f:(.text+0xa963): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0xa9c4): undefined reference to `mkl_lapack_zunglq' zgesvd_gen.f:(.text+0xa9f4): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0xaa5e): undefined reference to `mkl_lapack_zlaset' zgesvd_gen.f:(.text+0xaeff): undefined reference to `mkl_lapack_zlacpy' zgesvd_gen.f:(.text+0xafc2): undefined reference to `mkl_lapack_zlacpy' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zgerdb.o): In function `mkl_lapack_zgerdb': zgerdb_gen.f:(.text+0x449): undefined reference to `mkl_lapack_zgbbrdm' zgerdb_gen.f:(.text+0x666): undefined reference to `mkl_lapack_zlaset' zgerdb_gen.f:(.text+0x6a9): undefined reference to `mkl_lapack_zlaset' zgerdb_gen.f:(.text+0xf9d): undefined reference to `mkl_lapack_zunmlq' zgerdb_gen.f:(.text+0x1030): undefined reference to `mkl_lapack_zlaset' zgerdb_gen.f:(.text+0x1096): undefined reference to `mkl_lapack_zlaset' zgerdb_gen.f:(.text+0x120f): undefined reference to `mkl_lapack_zgbbrdm' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zgelss.o): In function `mkl_lapack_zgelss': zgelss_gen.f:(.text+0x529): undefined reference to `mkl_lapack_zgelqf' zgelss_gen.f:(.text+0x714): undefined reference to `mkl_lapack_zunmlq' zgelss_gen.f:(.text+0xa7b): undefined reference to `mkl_lapack_zlange' zgelss_gen.f:(.text+0xbc7): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0xbf9): undefined reference to `mkl_lapack_dlaset' zgelss_gen.f:(.text+0xca8): undefined reference to `mkl_lapack_zlange' zgelss_gen.f:(.text+0xf7e): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0x1311): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0x137c): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0x144e): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0x1527): undefined reference to `mkl_lapack_zlacpy' zgelss_gen.f:(.text+0x16f8): undefined reference to `mkl_lapack_zlacpy' zgelss_gen.f:(.text+0x17e0): undefined reference to `mkl_lapack_zgelqf' zgelss_gen.f:(.text+0x1807): undefined reference to `mkl_lapack_zlacpy' zgelss_gen.f:(.text+0x1861): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0x1c17): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0x1c7e): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0x1d54): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0x1e81): undefined reference to `mkl_lapack_zlacpy' zgelss_gen.f:(.text+0x207f): undefined reference to `mkl_lapack_zlacpy' zgelss_gen.f:(.text+0x2113): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0x21b7): undefined reference to `mkl_lapack_zunmlq' zgelss_gen.f:(.text+0x252c): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0x2599): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0x2670): undefined reference to `mkl_lapack_zlaset' zgelss_gen.f:(.text+0x2748): undefined reference to `mkl_lapack_zlacpy' zgelss_gen.f:(.text+0x2925): undefined reference to `mkl_lapack_zlacpy' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zgels.o): In function `mkl_lapack_zgels': zgels_gen.f:(.text+0x4a7): undefined reference to `mkl_lapack_zlaset' zgels_gen.f:(.text+0x564): undefined reference to `mkl_lapack_zlange' zgels_gen.f:(.text+0x74b): undefined reference to `mkl_lapack_zlange' zgels_gen.f:(.text+0xbc6): undefined reference to `mkl_lapack_zgelqf' zgels_gen.f:(.text+0xc95): undefined reference to `mkl_lapack_zunmlq' zgels_gen.f:(.text+0xe44): undefined reference to `mkl_lapack_zlaset' zgels_gen.f:(.text+0x142d): undefined reference to `mkl_lapack_zunmlq' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zgeev.o): In function `mkl_lapack_zgeev': zgeev_gen.f:(.text+0x67f): undefined reference to `mkl_lapack_zlange' zgeev_gen.f:(.text+0x82d): undefined reference to `mkl_lapack_zlacpy' zgeev_gen.f:(.text+0x989): undefined reference to `mkl_lapack_zlacpy' zgeev_gen.f:(.text+0x9ce): undefined reference to `mkl_lapack_zlacpy' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zgebrd.o): In function `mkl_lapack_zgebrd': zgebrd_gen.f:(.text+0x4ef): undefined reference to `mkl_lapack_zlabrd' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zgbbrd.o): In function `mkl_lapack_zgbbrd': zgbbrd_gen.f:(.text+0x2fb): undefined reference to `mkl_lapack_zlaset' zgbbrd_gen.f:(.text+0x33b): undefined reference to `mkl_lapack_zlaset' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zbdsqr.o): In function `mkl_lapack_zbdsqr': zbdsqr_gen.f:(.text+0xcde): undefined reference to `mkl_lapack_dlasr3' zbdsqr_gen.f:(.text+0xdd4): undefined reference to `mkl_lapack_dlasr3' zbdsqr_gen.f:(.text+0xee0): undefined reference to `mkl_lapack_dlasr3' zbdsqr_gen.f:(.text+0x1224): undefined reference to `mkl_lapack_dlasr3' zbdsqr_gen.f:(.text+0x133c): undefined reference to `mkl_lapack_dlasr3' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zbdsqr.o):zbdsqr_gen.f:(.text+0x1464): more undefined references to `mkl_lapack_dlasr3' follow /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zgerb2m.o): In function `mkl_lapack_zgerb2m': zgerb2m_gen.f:(.text+0x156): undefined reference to `mkl_lapack_zgeqrf_pf' zgerb2m_gen.f:(.text+0x181): undefined reference to `mkl_lapack_zlacpy' zgerb2m_gen.f:(.text+0x1ac): undefined reference to `mkl_lapack_zlacpy' zgerb2m_gen.f:(.text+0x438): undefined reference to `mkl_lapack_zgelqf_pf' zgerb2m_gen.f:(.text+0x463): undefined reference to `mkl_lapack_zlacpy' zgerb2m_gen.f:(.text+0x498): undefined reference to `mkl_lapack_zlacpy' zgerb2m_gen.f:(.text+0x71a): undefined reference to `mkl_lapack_zlacpy' zgerb2m_gen.f:(.text+0x8ad): undefined reference to `mkl_lapack_zlacpy' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zgetrf_local.o): In function `mkl_lapack_zgetrf_local': zgetrf_local.c:(.text+0x48e): undefined reference to `mkl_lapack_zlaswp' zgetrf_local.c:(.text+0x4f9): undefined reference to `mkl_lapack_zlaswp' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_s_i4_avx512_mic.o): In function `mkl_sparse_s_do_spmm_i4_avx512_mic': xspmm.c:(.text+0x400): undefined reference to `mkl_sparse_s_csr__g_n_spmm_i4' xspmm.c:(.text+0x5aa): undefined reference to `mkl_sparse_s_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(optimize_r_s_i4_avx512_mic.o): In function `mkl_sparse_optimize_i4_avx512_mic': optimize.c:(.text+0xd0): undefined reference to `mkl_sparse_copy_sparse_matrix_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx512_mic.o): In function `mkl_sparse_set_mv_hint_i4_avx512_mic': hints.c:(.text+0xe8): undefined reference to `mkl_sparse_optimize_csr_mv_i4' hints.c:(.text+0xfa): undefined reference to `mkl_sparse_optimize_bsr_mv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx512_mic.o): In function `mkl_sparse_set_mm_hint_i4_avx512_mic': hints.c:(.text+0x372): undefined reference to `mkl_sparse_optimize_csr_mm_i4' hints.c:(.text+0x383): undefined reference to `mkl_sparse_optimize_bsr_mm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx512_mic.o): In function `mkl_sparse_set_sv_hint_i4_avx512_mic': hints.c:(.text+0x5f8): undefined reference to `mkl_sparse_optimize_csr_trsv_i4' hints.c:(.text+0x60a): undefined reference to `mkl_sparse_optimize_bsr_trsv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx512_mic.o): In function `mkl_sparse_set_sm_hint_i4_avx512_mic': hints.c:(.text+0x884): undefined reference to `mkl_sparse_optimize_bsr_trsm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_d_i4_avx512_mic.o): In function `mkl_sparse_d_do_spmm_i4_avx512_mic': xspmm.c:(.text+0x3a0): undefined reference to `mkl_sparse_d_csr__g_n_spmm_i4' xspmm.c:(.text+0x54a): undefined reference to `mkl_sparse_d_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_s_i4_avx512_mic.o): In function `mkl_sparse_c_do_spmm_i4_avx512_mic': xspmm.c:(.text+0x3a0): undefined reference to `mkl_sparse_c_csr__g_n_spmm_i4' xspmm.c:(.text+0x54e): undefined reference to `mkl_sparse_c_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_d_i4_avx512_mic.o): In function `mkl_sparse_z_do_spmm_i4_avx512_mic': xspmm.c:(.text+0x3a0): undefined reference to `mkl_sparse_z_csr__g_n_spmm_i4' xspmm.c:(.text+0x54e): undefined reference to `mkl_sparse_z_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_s_i4_avx512.o): In function `mkl_sparse_s_do_spmm_i4_avx512': xspmm.c:(.text+0x35d): undefined reference to `mkl_sparse_s_csr__g_n_spmm_i4' xspmm.c:(.text+0x4fa): undefined reference to `mkl_sparse_s_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(optimize_r_s_i4_avx512.o): In function `mkl_sparse_optimize_i4_avx512': optimize.c:(.text+0xbf): undefined reference to `mkl_sparse_copy_sparse_matrix_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx512.o): In function `mkl_sparse_set_mv_hint_i4_avx512': hints.c:(.text+0xd0): undefined reference to `mkl_sparse_optimize_csr_mv_i4' hints.c:(.text+0xdf): undefined reference to `mkl_sparse_optimize_bsr_mv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx512.o): In function `mkl_sparse_set_mm_hint_i4_avx512': hints.c:(.text+0x32e): undefined reference to `mkl_sparse_optimize_csr_mm_i4' hints.c:(.text+0x33c): undefined reference to `mkl_sparse_optimize_bsr_mm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx512.o): In function `mkl_sparse_set_sv_hint_i4_avx512': hints.c:(.text+0x580): undefined reference to `mkl_sparse_optimize_csr_trsv_i4' hints.c:(.text+0x58f): undefined reference to `mkl_sparse_optimize_bsr_trsv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx512.o): In function `mkl_sparse_set_sm_hint_i4_avx512': hints.c:(.text+0x7dc): undefined reference to `mkl_sparse_optimize_bsr_trsm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_d_i4_avx512.o): In function `mkl_sparse_d_do_spmm_i4_avx512': xspmm.c:(.text+0x2fd): undefined reference to `mkl_sparse_d_csr__g_n_spmm_i4' xspmm.c:(.text+0x494): undefined reference to `mkl_sparse_d_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_s_i4_avx512.o): In function `mkl_sparse_c_do_spmm_i4_avx512': xspmm.c:(.text+0x2fd): undefined reference to `mkl_sparse_c_csr__g_n_spmm_i4' xspmm.c:(.text+0x49e): undefined reference to `mkl_sparse_c_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_d_i4_avx512.o): In function `mkl_sparse_z_do_spmm_i4_avx512': xspmm.c:(.text+0x2fd): undefined reference to `mkl_sparse_z_csr__g_n_spmm_i4' xspmm.c:(.text+0x49e): undefined reference to `mkl_sparse_z_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_s_i4_avx2.o): In function `mkl_sparse_s_do_spmm_i4_avx2': xspmm.c:(.text+0x357): undefined reference to `mkl_sparse_s_csr__g_n_spmm_i4' xspmm.c:(.text+0x4e8): undefined reference to `mkl_sparse_s_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(optimize_r_s_i4_avx2.o): In function `mkl_sparse_optimize_i4_avx2': optimize.c:(.text+0xbf): undefined reference to `mkl_sparse_copy_sparse_matrix_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx2.o): In function `mkl_sparse_set_mv_hint_i4_avx2': hints.c:(.text+0xcd): undefined reference to `mkl_sparse_optimize_csr_mv_i4' hints.c:(.text+0xdc): undefined reference to `mkl_sparse_optimize_bsr_mv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx2.o): In function `mkl_sparse_set_mm_hint_i4_avx2': hints.c:(.text+0x33b): undefined reference to `mkl_sparse_optimize_csr_mm_i4' hints.c:(.text+0x349): undefined reference to `mkl_sparse_optimize_bsr_mm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx2.o): In function `mkl_sparse_set_sv_hint_i4_avx2': hints.c:(.text+0x58d): undefined reference to `mkl_sparse_optimize_csr_trsv_i4' hints.c:(.text+0x59c): undefined reference to `mkl_sparse_optimize_bsr_trsv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx2.o): In function `mkl_sparse_set_sm_hint_i4_avx2': hints.c:(.text+0x7f9): undefined reference to `mkl_sparse_optimize_bsr_trsm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_d_i4_avx2.o): In function `mkl_sparse_d_do_spmm_i4_avx2': xspmm.c:(.text+0x2f7): undefined reference to `mkl_sparse_d_csr__g_n_spmm_i4' xspmm.c:(.text+0x485): undefined reference to `mkl_sparse_d_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_s_i4_avx2.o): In function `mkl_sparse_c_do_spmm_i4_avx2': xspmm.c:(.text+0x2f7): undefined reference to `mkl_sparse_c_csr__g_n_spmm_i4' xspmm.c:(.text+0x48c): undefined reference to `mkl_sparse_c_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_d_i4_avx2.o): In function `mkl_sparse_z_do_spmm_i4_avx2': xspmm.c:(.text+0x2f7): undefined reference to `mkl_sparse_z_csr__g_n_spmm_i4' xspmm.c:(.text+0x48c): undefined reference to `mkl_sparse_z_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_s_i4_avx.o): In function `mkl_sparse_s_do_spmm_i4_avx': xspmm.c:(.text+0x336): undefined reference to `mkl_sparse_s_csr__g_n_spmm_i4' xspmm.c:(.text+0x4d0): undefined reference to `mkl_sparse_s_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(optimize_r_s_i4_avx.o): In function `mkl_sparse_optimize_i4_avx': optimize.c:(.text+0xbf): undefined reference to `mkl_sparse_copy_sparse_matrix_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx.o): In function `mkl_sparse_set_mv_hint_i4_avx': hints.c:(.text+0xc7): undefined reference to `mkl_sparse_optimize_csr_mv_i4' hints.c:(.text+0xd5): undefined reference to `mkl_sparse_optimize_bsr_mv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx.o): In function `mkl_sparse_set_mm_hint_i4_avx': hints.c:(.text+0x3a7): undefined reference to `mkl_sparse_optimize_csr_mm_i4' hints.c:(.text+0x3b5): undefined reference to `mkl_sparse_optimize_bsr_mm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx.o): In function `mkl_sparse_set_sv_hint_i4_avx': hints.c:(.text+0x667): undefined reference to `mkl_sparse_optimize_csr_trsv_i4' hints.c:(.text+0x675): undefined reference to `mkl_sparse_optimize_bsr_trsv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_avx.o): In function `mkl_sparse_set_sm_hint_i4_avx': hints.c:(.text+0x935): undefined reference to `mkl_sparse_optimize_bsr_trsm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_d_i4_avx.o): In function `mkl_sparse_d_do_spmm_i4_avx': xspmm.c:(.text+0x2d6): undefined reference to `mkl_sparse_d_csr__g_n_spmm_i4' xspmm.c:(.text+0x46b): undefined reference to `mkl_sparse_d_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_s_i4_avx.o): In function `mkl_sparse_c_do_spmm_i4_avx': xspmm.c:(.text+0x2d6): undefined reference to `mkl_sparse_c_csr__g_n_spmm_i4' xspmm.c:(.text+0x46e): undefined reference to `mkl_sparse_c_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_d_i4_avx.o): In function `mkl_sparse_z_do_spmm_i4_avx': xspmm.c:(.text+0x2d6): undefined reference to `mkl_sparse_z_csr__g_n_spmm_i4' xspmm.c:(.text+0x46e): undefined reference to `mkl_sparse_z_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_s_i4_mc3.o): In function `mkl_sparse_s_do_spmm_i4_mc3': xspmm.c:(.text+0x336): undefined reference to `mkl_sparse_s_csr__g_n_spmm_i4' xspmm.c:(.text+0x4d0): undefined reference to `mkl_sparse_s_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(optimize_r_s_i4_mc3.o): In function `mkl_sparse_optimize_i4_mc3': optimize.c:(.text+0xbf): undefined reference to `mkl_sparse_copy_sparse_matrix_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_mc3.o): In function `mkl_sparse_set_mv_hint_i4_mc3': hints.c:(.text+0xc7): undefined reference to `mkl_sparse_optimize_csr_mv_i4' hints.c:(.text+0xd5): undefined reference to `mkl_sparse_optimize_bsr_mv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_mc3.o): In function `mkl_sparse_set_mm_hint_i4_mc3': hints.c:(.text+0x397): undefined reference to `mkl_sparse_optimize_csr_mm_i4' hints.c:(.text+0x3a5): undefined reference to `mkl_sparse_optimize_bsr_mm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_mc3.o): In function `mkl_sparse_set_sv_hint_i4_mc3': hints.c:(.text+0x647): undefined reference to `mkl_sparse_optimize_csr_trsv_i4' hints.c:(.text+0x655): undefined reference to `mkl_sparse_optimize_bsr_trsv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_mc3.o): In function `mkl_sparse_set_sm_hint_i4_mc3': hints.c:(.text+0x915): undefined reference to `mkl_sparse_optimize_bsr_trsm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_d_i4_mc3.o): In function `mkl_sparse_d_do_spmm_i4_mc3': xspmm.c:(.text+0x2d6): undefined reference to `mkl_sparse_d_csr__g_n_spmm_i4' xspmm.c:(.text+0x46b): undefined reference to `mkl_sparse_d_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_s_i4_mc3.o): In function `mkl_sparse_c_do_spmm_i4_mc3': xspmm.c:(.text+0x2d6): undefined reference to `mkl_sparse_c_csr__g_n_spmm_i4' xspmm.c:(.text+0x46e): undefined reference to `mkl_sparse_c_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_d_i4_mc3.o): In function `mkl_sparse_z_do_spmm_i4_mc3': xspmm.c:(.text+0x2d6): undefined reference to `mkl_sparse_z_csr__g_n_spmm_i4' xspmm.c:(.text+0x46e): undefined reference to `mkl_sparse_z_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_s_i4_mc.o): In function `mkl_sparse_s_do_spmm_i4_mc': xspmm.c:(.text+0x336): undefined reference to `mkl_sparse_s_csr__g_n_spmm_i4' xspmm.c:(.text+0x4d0): undefined reference to `mkl_sparse_s_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(optimize_r_s_i4_mc.o): In function `mkl_sparse_optimize_i4_mc': optimize.c:(.text+0xbf): undefined reference to `mkl_sparse_copy_sparse_matrix_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_mc.o): In function `mkl_sparse_set_mv_hint_i4_mc': hints.c:(.text+0xc8): undefined reference to `mkl_sparse_optimize_csr_mv_i4' hints.c:(.text+0xd7): undefined reference to `mkl_sparse_optimize_bsr_mv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_mc.o): In function `mkl_sparse_set_mm_hint_i4_mc': hints.c:(.text+0x358): undefined reference to `mkl_sparse_optimize_csr_mm_i4' hints.c:(.text+0x366): undefined reference to `mkl_sparse_optimize_bsr_mm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_mc.o): In function `mkl_sparse_set_sv_hint_i4_mc': hints.c:(.text+0x5c8): undefined reference to `mkl_sparse_optimize_csr_trsv_i4' hints.c:(.text+0x5d7): undefined reference to `mkl_sparse_optimize_bsr_trsv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_mc.o): In function `mkl_sparse_set_sm_hint_i4_mc': hints.c:(.text+0x856): undefined reference to `mkl_sparse_optimize_bsr_trsm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_d_i4_mc.o): In function `mkl_sparse_d_do_spmm_i4_mc': xspmm.c:(.text+0x2d6): undefined reference to `mkl_sparse_d_csr__g_n_spmm_i4' xspmm.c:(.text+0x46b): undefined reference to `mkl_sparse_d_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_s_i4_mc.o): In function `mkl_sparse_c_do_spmm_i4_mc': xspmm.c:(.text+0x2d6): undefined reference to `mkl_sparse_c_csr__g_n_spmm_i4' xspmm.c:(.text+0x46e): undefined reference to `mkl_sparse_c_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_d_i4_mc.o): In function `mkl_sparse_z_do_spmm_i4_mc': xspmm.c:(.text+0x2d6): undefined reference to `mkl_sparse_z_csr__g_n_spmm_i4' xspmm.c:(.text+0x46e): undefined reference to `mkl_sparse_z_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_s_i4_def.o): In function `mkl_sparse_s_do_spmm_i4_def': xspmm.c:(.text+0x336): undefined reference to `mkl_sparse_s_csr__g_n_spmm_i4' xspmm.c:(.text+0x4d0): undefined reference to `mkl_sparse_s_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(optimize_r_s_i4_def.o): In function `mkl_sparse_optimize_i4_def': optimize.c:(.text+0xbf): undefined reference to `mkl_sparse_copy_sparse_matrix_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_def.o): In function `mkl_sparse_set_mv_hint_i4_def': hints.c:(.text+0xc7): undefined reference to `mkl_sparse_optimize_csr_mv_i4' hints.c:(.text+0xd5): undefined reference to `mkl_sparse_optimize_bsr_mv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_def.o): In function `mkl_sparse_set_mm_hint_i4_def': hints.c:(.text+0x397): undefined reference to `mkl_sparse_optimize_csr_mm_i4' hints.c:(.text+0x3a5): undefined reference to `mkl_sparse_optimize_bsr_mm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_def.o): In function `mkl_sparse_set_sv_hint_i4_def': hints.c:(.text+0x647): undefined reference to `mkl_sparse_optimize_csr_trsv_i4' hints.c:(.text+0x655): undefined reference to `mkl_sparse_optimize_bsr_trsv_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(hints_r_s_i4_def.o): In function `mkl_sparse_set_sm_hint_i4_def': hints.c:(.text+0x915): undefined reference to `mkl_sparse_optimize_bsr_trsm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_r_d_i4_def.o): In function `mkl_sparse_d_do_spmm_i4_def': xspmm.c:(.text+0x2d6): undefined reference to `mkl_sparse_d_csr__g_n_spmm_i4' xspmm.c:(.text+0x46b): undefined reference to `mkl_sparse_d_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_s_i4_def.o): In function `mkl_sparse_c_do_spmm_i4_def': xspmm.c:(.text+0x2d6): undefined reference to `mkl_sparse_c_csr__g_n_spmm_i4' xspmm.c:(.text+0x46e): undefined reference to `mkl_sparse_c_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xspmm_c_d_i4_def.o): In function `mkl_sparse_z_do_spmm_i4_def': xspmm.c:(.text+0x2d6): undefined reference to `mkl_sparse_z_csr__g_n_spmm_i4' xspmm.c:(.text+0x46e): undefined reference to `mkl_sparse_z_bsr__g_n_spmm_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(ztrevc3.o): In function `mkl_lapack_ztrevc3': ztrevc3_gen.f:(.text+0x67e): undefined reference to `mkl_lapack_zlaset' ztrevc3_gen.f:(.text+0x1837): undefined reference to `mkl_lapack_zlacpy' ztrevc3_gen.f:(.text+0x1b8d): undefined reference to `mkl_lapack_zlacpy' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zunmtr.o): In function `mkl_lapack_zunmtr': zunmtr_gen.f:(.text+0x590): undefined reference to `mkl_lapack_zunmql' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zunmbr.o): In function `mkl_lapack_zunmbr': zunmbr_gen.f:(.text+0x793): undefined reference to `mkl_lapack_zunmlq' zunmbr_gen.f:(.text+0x829): undefined reference to `mkl_lapack_zunmlq' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zungtr.o): In function `mkl_lapack_zungtr': zungtr_gen.f:(.text+0x75a): undefined reference to `mkl_lapack_zungql' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zungbr.o): In function `mkl_lapack_zungbr': zungbr_gen.f:(.text+0x35a): undefined reference to `mkl_lapack_zunglq' zungbr_gen.f:(.text+0x3a7): undefined reference to `mkl_lapack_zunglq' zungbr_gen.f:(.text+0x8a8): undefined reference to `mkl_lapack_zunglq' zungbr_gen.f:(.text+0x8f9): undefined reference to `mkl_lapack_zunglq' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zstein.o): In function `mkl_lapack_zstein': zstein_gen.f:(.text+0x83c): undefined reference to `mkl_blas_dscal' zstein_gen.f:(.text+0xce0): undefined reference to `mkl_blas_dscal' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zlatrs.o): In function `mkl_lapack_zlatrs': zlatrs_gen.f:(.text+0x555): undefined reference to `mkl_blas_dscal' zlatrs_gen.f:(.text+0x2085): undefined reference to `mkl_blas_dscal' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zhseqr.o): In function `mkl_lapack_zhseqr': zhseqr_gen.f:(.text+0x3bb): undefined reference to `mkl_lapack_zlaset' zhseqr_gen.f:(.text+0x7b8): undefined reference to `mkl_lapack_zlacpy' zhseqr_gen.f:(.text+0x814): undefined reference to `mkl_lapack_zlaset' zhseqr_gen.f:(.text+0x8d7): undefined reference to `mkl_lapack_zlacpy' zhseqr_gen.f:(.text+0x952): undefined reference to `mkl_lapack_zlaset' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zherdb.o): In function `mkl_lapack_zherdb': zherdb_gen.f:(.text+0x18d9): undefined reference to `mkl_lapack_zlacpy' zherdb_gen.f:(.text+0x1939): undefined reference to `mkl_lapack_zlacpy' zherdb_gen.f:(.text+0x1d0b): undefined reference to `mkl_lapack_zhbrdbm' zherdb_gen.f:(.text+0x1eb6): undefined reference to `mkl_lapack_zungql' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zherb2m.o): In function `mkl_lapack_zherb2m': zherb2m_gen.f:(.text+0x108): undefined reference to `mkl_lapack_zgeqrf_pf' zherb2m_gen.f:(.text+0x133): undefined reference to `mkl_lapack_zlacpy' zherb2m_gen.f:(.text+0x15e): undefined reference to `mkl_lapack_zlacpy' zherb2m_gen.f:(.text+0x30c): undefined reference to `mkl_lapack_zgeqlf_pf' zherb2m_gen.f:(.text+0x337): undefined reference to `mkl_lapack_zlacpy' zherb2m_gen.f:(.text+0x362): undefined reference to `mkl_lapack_zlacpy' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx512_mic.o): In function `mkl_sparse_s_convert_bsr_i4_avx512_mic': xconvert.c:(.text+0x90a): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0x9ba): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0xb50): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0xbf3): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0xc25): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' xconvert.c:(.text+0xc60): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx512_mic.o): In function `mkl_sparse_s_export_csr_data_i4_avx512_mic': xconvert.c:(.text+0xdd9): undefined reference to `mkl_sparse_s_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx512_mic.o): In function `mkl_sparse_s_convert_csc_to_csr_i4_avx512_mic': xconvert.c:(.text+0x18a3): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx512_mic.o): In function `mkl_sparse_s_transpose_csr3a_i4_avx512_mic': xconvert.c:(.text+0x1b01): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx512_mic.o): In function `mkl_sparse_d_convert_bsr_i4_avx512_mic': xconvert.c:(.text+0x29a): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x34a): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x4e0): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x583): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x5b5): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' xconvert.c:(.text+0x5f0): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx512_mic.o): In function `mkl_sparse_d_export_csr_data_i4_avx512_mic': xconvert.c:(.text+0x769): undefined reference to `mkl_sparse_d_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx512_mic.o): In function `mkl_sparse_d_convert_csc_to_csr_i4_avx512_mic': xconvert.c:(.text+0x1248): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx512_mic.o): In function `mkl_sparse_d_transpose_csr3a_i4_avx512_mic': xconvert.c:(.text+0x14a2): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx512_mic.o): In function `mkl_sparse_c_convert_bsr_i4_avx512_mic': xconvert.c:(.text+0x29a): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x34a): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x4e0): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x583): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x5b5): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' xconvert.c:(.text+0x5f0): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx512_mic.o): In function `mkl_sparse_c_export_csr_data_i4_avx512_mic': xconvert.c:(.text+0x769): undefined reference to `mkl_sparse_c_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx512_mic.o): In function `mkl_sparse_c_convert_csc_to_csr_i4_avx512_mic': xconvert.c:(.text+0x1248): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx512_mic.o): In function `mkl_sparse_c_transpose_csr3a_i4_avx512_mic': xconvert.c:(.text+0x14a2): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx512_mic.o): In function `mkl_sparse_z_convert_bsr_i4_avx512_mic': xconvert.c:(.text+0x29a): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x34a): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x4e4): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x587): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x5b9): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' xconvert.c:(.text+0x5f4): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx512_mic.o): In function `mkl_sparse_z_export_csr_data_i4_avx512_mic': xconvert.c:(.text+0x771): undefined reference to `mkl_sparse_z_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx512_mic.o): In function `mkl_sparse_z_convert_csc_to_csr_i4_avx512_mic': xconvert.c:(.text+0x1248): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx512_mic.o): In function `mkl_sparse_z_transpose_csr3a_i4_avx512_mic': xconvert.c:(.text+0x14a2): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx512.o): In function `mkl_sparse_s_convert_bsr_i4_avx512': xconvert.c:(.text+0x836): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0x8d3): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0xa36): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0xaaf): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0xaed): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' xconvert.c:(.text+0xb13): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx512.o): In function `mkl_sparse_s_export_csr_data_i4_avx512': xconvert.c:(.text+0xc52): undefined reference to `mkl_sparse_s_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx512.o): In function `mkl_sparse_s_convert_csc_to_csr_i4_avx512': xconvert.c:(.text+0x1690): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx512.o): In function `mkl_sparse_s_transpose_csr3a_i4_avx512': xconvert.c:(.text+0x18cf): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx512.o): In function `mkl_sparse_d_convert_bsr_i4_avx512': xconvert.c:(.text+0x226): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x2c3): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x426): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x49f): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x4dd): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' xconvert.c:(.text+0x503): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx512.o): In function `mkl_sparse_d_export_csr_data_i4_avx512': xconvert.c:(.text+0x642): undefined reference to `mkl_sparse_d_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx512.o): In function `mkl_sparse_d_convert_csc_to_csr_i4_avx512': xconvert.c:(.text+0x1085): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx512.o): In function `mkl_sparse_d_transpose_csr3a_i4_avx512': xconvert.c:(.text+0x12c0): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx512.o): In function `mkl_sparse_c_convert_bsr_i4_avx512': xconvert.c:(.text+0x226): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x2c3): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x426): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x49f): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x4dd): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' xconvert.c:(.text+0x503): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx512.o): In function `mkl_sparse_c_export_csr_data_i4_avx512': xconvert.c:(.text+0x642): undefined reference to `mkl_sparse_c_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx512.o): In function `mkl_sparse_c_convert_csc_to_csr_i4_avx512': xconvert.c:(.text+0x1085): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx512.o): In function `mkl_sparse_c_transpose_csr3a_i4_avx512': xconvert.c:(.text+0x12c0): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx512.o): In function `mkl_sparse_z_convert_bsr_i4_avx512': xconvert.c:(.text+0x226): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x2c3): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x42a): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x4a3): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x4e1): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' xconvert.c:(.text+0x507): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx512.o): In function `mkl_sparse_z_export_csr_data_i4_avx512': xconvert.c:(.text+0x65a): undefined reference to `mkl_sparse_z_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx512.o): In function `mkl_sparse_z_convert_csc_to_csr_i4_avx512': xconvert.c:(.text+0x1075): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx512.o): In function `mkl_sparse_z_transpose_csr3a_i4_avx512': xconvert.c:(.text+0x12b0): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx2.o): In function `mkl_sparse_s_convert_bsr_i4_avx2': xconvert.c:(.text+0x746): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0x7e3): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0x946): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0x9bf): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0x9fd): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' xconvert.c:(.text+0xa23): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx2.o): In function `mkl_sparse_s_export_csr_data_i4_avx2': xconvert.c:(.text+0xb62): undefined reference to `mkl_sparse_s_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx2.o): In function `mkl_sparse_s_convert_csc_to_csr_i4_avx2': xconvert.c:(.text+0x1480): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx2.o): In function `mkl_sparse_s_transpose_csr3a_i4_avx2': xconvert.c:(.text+0x16bf): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx2.o): In function `mkl_sparse_d_convert_bsr_i4_avx2': xconvert.c:(.text+0x226): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x2c3): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x426): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x49f): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x4dd): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' xconvert.c:(.text+0x503): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx2.o): In function `mkl_sparse_d_export_csr_data_i4_avx2': xconvert.c:(.text+0x642): undefined reference to `mkl_sparse_d_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx2.o): In function `mkl_sparse_d_convert_csc_to_csr_i4_avx2': xconvert.c:(.text+0xf65): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx2.o): In function `mkl_sparse_d_transpose_csr3a_i4_avx2': xconvert.c:(.text+0x11a0): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx2.o): In function `mkl_sparse_c_convert_bsr_i4_avx2': xconvert.c:(.text+0x226): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x2c3): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x426): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x49f): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x4dd): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' xconvert.c:(.text+0x503): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx2.o): In function `mkl_sparse_c_export_csr_data_i4_avx2': xconvert.c:(.text+0x642): undefined reference to `mkl_sparse_c_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx2.o): In function `mkl_sparse_c_convert_csc_to_csr_i4_avx2': xconvert.c:(.text+0xf65): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx2.o): In function `mkl_sparse_c_transpose_csr3a_i4_avx2': xconvert.c:(.text+0x11a0): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx2.o): In function `mkl_sparse_z_convert_bsr_i4_avx2': xconvert.c:(.text+0x226): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x2c3): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x42a): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x4a3): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x4e1): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' xconvert.c:(.text+0x507): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx2.o): In function `mkl_sparse_z_export_csr_data_i4_avx2': xconvert.c:(.text+0x65a): undefined reference to `mkl_sparse_z_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx2.o): In function `mkl_sparse_z_convert_csc_to_csr_i4_avx2': xconvert.c:(.text+0xf85): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx2.o): In function `mkl_sparse_z_transpose_csr3a_i4_avx2': xconvert.c:(.text+0x11c0): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx.o): In function `mkl_sparse_s_convert_bsr_i4_avx': xconvert.c:(.text+0x712): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0x7aa): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0x902): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0x974): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0x9ab): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' xconvert.c:(.text+0x9ca): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx.o): In function `mkl_sparse_s_export_csr_data_i4_avx': xconvert.c:(.text+0xb02): undefined reference to `mkl_sparse_s_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx.o): In function `mkl_sparse_s_convert_csc_to_csr_i4_avx': xconvert.c:(.text+0x139c): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_avx.o): In function `mkl_sparse_s_transpose_csr3a_i4_avx': xconvert.c:(.text+0x15c9): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx.o): In function `mkl_sparse_d_convert_bsr_i4_avx': xconvert.c:(.text+0x212): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x2aa): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x402): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x474): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x4ab): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' xconvert.c:(.text+0x4ca): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx.o): In function `mkl_sparse_d_export_csr_data_i4_avx': xconvert.c:(.text+0x602): undefined reference to `mkl_sparse_d_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx.o): In function `mkl_sparse_d_convert_csc_to_csr_i4_avx': xconvert.c:(.text+0xee1): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_avx.o): In function `mkl_sparse_d_transpose_csr3a_i4_avx': xconvert.c:(.text+0x110a): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx.o): In function `mkl_sparse_c_convert_bsr_i4_avx': xconvert.c:(.text+0x212): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x2aa): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x402): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x474): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x4ab): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' xconvert.c:(.text+0x4ca): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx.o): In function `mkl_sparse_c_export_csr_data_i4_avx': xconvert.c:(.text+0x602): undefined reference to `mkl_sparse_c_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx.o): In function `mkl_sparse_c_convert_csc_to_csr_i4_avx': xconvert.c:(.text+0xea1): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_avx.o): In function `mkl_sparse_c_transpose_csr3a_i4_avx': xconvert.c:(.text+0x10ca): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx.o): In function `mkl_sparse_z_convert_bsr_i4_avx': xconvert.c:(.text+0x212): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x2aa): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x406): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x478): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x4af): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' xconvert.c:(.text+0x4ce): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx.o): In function `mkl_sparse_z_export_csr_data_i4_avx': xconvert.c:(.text+0x60a): undefined reference to `mkl_sparse_z_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx.o): In function `mkl_sparse_z_convert_csc_to_csr_i4_avx': xconvert.c:(.text+0xf51): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_avx.o): In function `mkl_sparse_z_transpose_csr3a_i4_avx': xconvert.c:(.text+0x117a): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_mc3.o): In function `mkl_sparse_s_convert_bsr_i4_mc3': xconvert.c:(.text+0x701): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0x799): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0x8f0): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0x962): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0x999): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' xconvert.c:(.text+0x9b8): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_mc3.o): In function `mkl_sparse_s_export_csr_data_i4_mc3': xconvert.c:(.text+0xaf0): undefined reference to `mkl_sparse_s_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_mc3.o): In function `mkl_sparse_s_convert_csc_to_csr_i4_mc3': xconvert.c:(.text+0x140b): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_mc3.o): In function `mkl_sparse_s_transpose_csr3a_i4_mc3': xconvert.c:(.text+0x1638): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_mc3.o): In function `mkl_sparse_d_convert_bsr_i4_mc3': xconvert.c:(.text+0x211): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x2a9): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x400): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x472): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x4a9): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' xconvert.c:(.text+0x4c8): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_mc3.o): In function `mkl_sparse_d_export_csr_data_i4_mc3': xconvert.c:(.text+0x600): undefined reference to `mkl_sparse_d_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_mc3.o): In function `mkl_sparse_d_convert_csc_to_csr_i4_mc3': xconvert.c:(.text+0xf20): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_mc3.o): In function `mkl_sparse_d_transpose_csr3a_i4_mc3': xconvert.c:(.text+0x1149): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_mc3.o): In function `mkl_sparse_c_convert_bsr_i4_mc3': xconvert.c:(.text+0x211): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x2a9): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x400): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x472): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x4a9): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' xconvert.c:(.text+0x4c8): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_mc3.o): In function `mkl_sparse_c_export_csr_data_i4_mc3': xconvert.c:(.text+0x600): undefined reference to `mkl_sparse_c_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_mc3.o): In function `mkl_sparse_c_convert_csc_to_csr_i4_mc3': xconvert.c:(.text+0xf20): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_mc3.o): In function `mkl_sparse_c_transpose_csr3a_i4_mc3': xconvert.c:(.text+0x1149): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_mc3.o): In function `mkl_sparse_z_convert_bsr_i4_mc3': xconvert.c:(.text+0x211): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x2a9): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x403): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x475): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x4ac): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' xconvert.c:(.text+0x4cb): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_mc3.o): In function `mkl_sparse_z_export_csr_data_i4_mc3': xconvert.c:(.text+0x606): undefined reference to `mkl_sparse_z_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_mc3.o): In function `mkl_sparse_z_convert_csc_to_csr_i4_mc3': xconvert.c:(.text+0xf80): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_mc3.o): In function `mkl_sparse_z_transpose_csr3a_i4_mc3': xconvert.c:(.text+0x11a9): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_mc.o): In function `mkl_sparse_s_convert_bsr_i4_mc': xconvert.c:(.text+0x721): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0x7b9): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0x910): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0x982): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0x9b9): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' xconvert.c:(.text+0x9d8): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_mc.o): In function `mkl_sparse_s_export_csr_data_i4_mc': xconvert.c:(.text+0xb13): undefined reference to `mkl_sparse_s_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_mc.o): In function `mkl_sparse_s_convert_csc_to_csr_i4_mc': xconvert.c:(.text+0x139b): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_mc.o): In function `mkl_sparse_s_transpose_csr3a_i4_mc': xconvert.c:(.text+0x15c8): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_mc.o): In function `mkl_sparse_d_convert_bsr_i4_mc': xconvert.c:(.text+0x211): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x2a9): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x400): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x472): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x4a9): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' xconvert.c:(.text+0x4c8): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_mc.o): In function `mkl_sparse_d_export_csr_data_i4_mc': xconvert.c:(.text+0x603): undefined reference to `mkl_sparse_d_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_mc.o): In function `mkl_sparse_d_convert_csc_to_csr_i4_mc': xconvert.c:(.text+0xf30): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_mc.o): In function `mkl_sparse_d_transpose_csr3a_i4_mc': xconvert.c:(.text+0x1159): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_mc.o): In function `mkl_sparse_c_convert_bsr_i4_mc': xconvert.c:(.text+0x211): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x2a9): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x400): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x472): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x4a9): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' xconvert.c:(.text+0x4c8): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_mc.o): In function `mkl_sparse_c_export_csr_data_i4_mc': xconvert.c:(.text+0x603): undefined reference to `mkl_sparse_c_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_mc.o): In function `mkl_sparse_c_convert_csc_to_csr_i4_mc': xconvert.c:(.text+0xe90): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_mc.o): In function `mkl_sparse_c_transpose_csr3a_i4_mc': xconvert.c:(.text+0x10b9): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_mc.o): In function `mkl_sparse_z_convert_bsr_i4_mc': xconvert.c:(.text+0x211): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x2a9): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x403): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x475): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x4ac): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' xconvert.c:(.text+0x4cb): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_mc.o): In function `mkl_sparse_z_export_csr_data_i4_mc': xconvert.c:(.text+0x609): undefined reference to `mkl_sparse_z_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_mc.o): In function `mkl_sparse_z_convert_csc_to_csr_i4_mc': xconvert.c:(.text+0xfa0): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_mc.o): In function `mkl_sparse_z_transpose_csr3a_i4_mc': xconvert.c:(.text+0x11c9): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_def.o): In function `mkl_sparse_s_convert_bsr_i4_def': xconvert.c:(.text+0x731): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0x7c9): undefined reference to `mkl_sparse_s_csrbsr_i4' xconvert.c:(.text+0x921): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0x993): undefined reference to `mkl_sparse_s_bsrbsr_i4' xconvert.c:(.text+0x9ca): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' xconvert.c:(.text+0x9e9): undefined reference to `mkl_sparse_s_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_def.o): In function `mkl_sparse_s_export_csr_data_i4_def': xconvert.c:(.text+0xb22): undefined reference to `mkl_sparse_s_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_def.o): In function `mkl_sparse_s_convert_csc_to_csr_i4_def': xconvert.c:(.text+0x13ab): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_s_i4_def.o): In function `mkl_sparse_s_transpose_csr3a_i4_def': xconvert.c:(.text+0x15d8): undefined reference to `mkl_sparse_s_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_def.o): In function `mkl_sparse_d_convert_bsr_i4_def': xconvert.c:(.text+0x211): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x2a9): undefined reference to `mkl_sparse_d_csrbsr_i4' xconvert.c:(.text+0x401): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x473): undefined reference to `mkl_sparse_d_bsrbsr_i4' xconvert.c:(.text+0x4aa): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' xconvert.c:(.text+0x4c9): undefined reference to `mkl_sparse_d_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_def.o): In function `mkl_sparse_d_export_csr_data_i4_def': xconvert.c:(.text+0x602): undefined reference to `mkl_sparse_d_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_def.o): In function `mkl_sparse_d_convert_csc_to_csr_i4_def': xconvert.c:(.text+0xf30): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_r_d_i4_def.o): In function `mkl_sparse_d_transpose_csr3a_i4_def': xconvert.c:(.text+0x1159): undefined reference to `mkl_sparse_d_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_def.o): In function `mkl_sparse_c_convert_bsr_i4_def': xconvert.c:(.text+0x211): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x2a9): undefined reference to `mkl_sparse_c_csrbsr_i4' xconvert.c:(.text+0x401): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x473): undefined reference to `mkl_sparse_c_bsrbsr_i4' xconvert.c:(.text+0x4aa): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' xconvert.c:(.text+0x4c9): undefined reference to `mkl_sparse_c_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_def.o): In function `mkl_sparse_c_export_csr_data_i4_def': xconvert.c:(.text+0x602): undefined reference to `mkl_sparse_c_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_def.o): In function `mkl_sparse_c_convert_csc_to_csr_i4_def': xconvert.c:(.text+0xe90): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_s_i4_def.o): In function `mkl_sparse_c_transpose_csr3a_i4_def': xconvert.c:(.text+0x10b9): undefined reference to `mkl_sparse_c_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_def.o): In function `mkl_sparse_z_convert_bsr_i4_def': xconvert.c:(.text+0x211): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x2a9): undefined reference to `mkl_sparse_z_csrbsr_i4' xconvert.c:(.text+0x405): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x477): undefined reference to `mkl_sparse_z_bsrbsr_i4' xconvert.c:(.text+0x4ae): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' xconvert.c:(.text+0x4cd): undefined reference to `mkl_sparse_z_convert_bsr_transpose_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_def.o): In function `mkl_sparse_z_export_csr_data_i4_def': xconvert.c:(.text+0x60a): undefined reference to `mkl_sparse_z_convert_coo_to_csr_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_def.o): In function `mkl_sparse_z_convert_csc_to_csr_i4_def': xconvert.c:(.text+0xf80): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(xconvert_c_d_i4_def.o): In function `mkl_sparse_z_transpose_csr3a_i4_def': xconvert.c:(.text+0x11a9): undefined reference to `mkl_sparse_z_convert_csc_to_csr_omp_i4' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zlaqr0.o): In function `mkl_lapack_zlaqr0': zlaqr0_gen.f:(.text+0xe05): undefined reference to `mkl_lapack_zlacpy' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zlahr2.o): In function `mkl_lapack_zlahr2': zlahr2_gen.f:(.text+0xa21): undefined reference to `mkl_lapack_zlacpy' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zlaqr5.o): In function `mkl_lapack_zlaqr5': zlaqr5_gen.f:(.text+0x87f): undefined reference to `mkl_lapack_zlaset' zlaqr5_gen.f:(.text+0x2c0e): undefined reference to `mkl_lapack_zlacpy' zlaqr5_gen.f:(.text+0x2e12): undefined reference to `mkl_lapack_zlacpy' zlaqr5_gen.f:(.text+0x2ffb): undefined reference to `mkl_lapack_zlacpy' zlaqr5_gen.f:(.text+0x31fa): undefined reference to `mkl_lapack_zlacpy' zlaqr5_gen.f:(.text+0x3229): undefined reference to `mkl_lapack_zlaset' zlaqr5_gen.f:(.text+0x3389): undefined reference to `mkl_lapack_zlacpy' zlaqr5_gen.f:(.text+0x3524): undefined reference to `mkl_lapack_zlacpy' zlaqr5_gen.f:(.text+0x3704): undefined reference to `mkl_lapack_zlacpy' zlaqr5_gen.f:(.text+0x3733): undefined reference to `mkl_lapack_zlaset' zlaqr5_gen.f:(.text+0x38be): undefined reference to `mkl_lapack_zlacpy' zlaqr5_gen.f:(.text+0x3ab6): undefined reference to `mkl_lapack_zlacpy' zlaqr5_gen.f:(.text+0x3c71): undefined reference to `mkl_lapack_zlacpy' zlaqr5_gen.f:(.text+0x3ca0): undefined reference to `mkl_lapack_zlaset' zlaqr5_gen.f:(.text+0x3e25): undefined reference to `mkl_lapack_zlacpy' zlaqr5_gen.f:(.text+0x4007): undefined reference to `mkl_lapack_zlacpy' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zlaqr4.o): In function `mkl_lapack_zlaqr4': zlaqr4_gen.f:(.text+0xe05): undefined reference to `mkl_lapack_zlacpy' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zlaqr3.o): In function `mkl_lapack_zlaqr3': zlaqr3_gen.f:(.text+0x51e): undefined reference to `mkl_lapack_zlacpy' zlaqr3_gen.f:(.text+0x5c6): undefined reference to `mkl_lapack_zlaset' zlaqr3_gen.f:(.text+0xd8b): undefined reference to `mkl_lapack_zlaset' zlaqr3_gen.f:(.text+0xfa8): undefined reference to `mkl_lapack_zlacpy' zlaqr3_gen.f:(.text+0x1229): undefined reference to `mkl_lapack_zlacpy' zlaqr3_gen.f:(.text+0x13c8): undefined reference to `mkl_lapack_zlacpy' zlaqr3_gen.f:(.text+0x156d): undefined reference to `mkl_lapack_zlacpy' /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(zlaqr2.o): In function `mkl_lapack_zlaqr2': zlaqr2_gen.f:(.text+0x465): undefined reference to `mkl_lapack_zlacpy' zlaqr2_gen.f:(.text+0x525): undefined reference to `mkl_lapack_zlaset' zlaqr2_gen.f:(.text+0xc59): undefined reference to `mkl_lapack_zlaset' zlaqr2_gen.f:(.text+0xe8a): undefined reference to `mkl_lapack_zlacpy' zlaqr2_gen.f:(.text+0x1117): undefined reference to `mkl_lapack_zlacpy' zlaqr2_gen.f:(.text+0x12b8): undefined reference to `mkl_lapack_zlacpy' zlaqr2_gen.f:(.text+0x145f): undefined reference to `mkl_lapack_zlacpy' /usr/bin/ld: link errors found, deleting executable `ex5f' /usr/bin/X11/sha1sum: ex5f: No such file or directory makefile:50: recipe for target 'ex5f' failed make: [ex5f] Error 1 (ignored) /usr/bin/rm -f ex5f.o -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 14598584 bytes Desc: not available URL: From stefano.zampini at gmail.com Tue Apr 24 07:53:04 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Tue, 24 Apr 2018 15:53:04 +0300 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> Message-ID: <465A623F-EF8A-4341-A38A-2A8F212BD91E@gmail.com> Petsc configure should be able to detect MKL and add the proper linking by just using the -with-blaslapack-dir= option. Alternatively, you can use --COPTFLAGS=?-mkl-sequential? > On Apr 24, 2018, at 10:26 AM, Samuel Lanthaler wrote: > > From stefano.zampini at gmail.com Tue Apr 24 07:53:55 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Tue, 24 Apr 2018 15:53:55 +0300 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: <465A623F-EF8A-4341-A38A-2A8F212BD91E@gmail.com> References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> <465A623F-EF8A-4341-A38A-2A8F212BD91E@gmail.com> Message-ID: <560EA9E4-DAE7-433C-B4DF-9765D5C5A4ED@gmail.com> sorry, the last option should be ?COPTFLAGS='-mkl=sequential' > On Apr 24, 2018, at 3:53 PM, Stefano Zampini wrote: > > Petsc configure should be able to detect MKL and add the proper linking by just using the -with-blaslapack-dir= option. > Alternatively, you can use --COPTFLAGS=?-mkl-sequential? > >> On Apr 24, 2018, at 10:26 AM, Samuel Lanthaler wrote: >> >> > From balay at mcs.anl.gov Tue Apr 24 07:55:43 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 24 Apr 2018 07:55:43 -0500 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> Message-ID: On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > Hi there, > > I was wondering if you could help me with the correct configuration of > PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure which > information would be useful to you, but basically the problem seems to be in > correctly compiling it with intel compiler and the existing mkl library. > > The pre-installed mkl version is > > intel/17.0.4.196 > > I have tried various things and finally, I got it at least to compile with the > following options (options chosen in reference to the mkl link advisor...): > > ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 > --with-scalar-type=complex --download-scalapack --download-mumps=yes > --download-superlu --with-batch --known-mpi-shared-libraries=1 > *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a > ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group > ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a > ${MKLROOT}/lib/intel64/libmkl_sequential.a > ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl" > *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" Can you remove the above 2 options [--with-blaslapack-lib, --with-blaslapack-include] and use: --with-blas-lapack-dir=${MKLROOT} And see if you still have this problem? Satish > --known-64-bit-blas-indices=0 > > After compilation, when trying to compile > /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get linking > errors (attached). Would anyone be able to help me out here? I really don't > have a good understanding of this. > > I'm attaching the configuration.log file, as well as the linking errors when > trying to compile ex5f. > > Thank you very much in advance! > > Best regards, > Samuel > From yongxiang27 at gmail.com Tue Apr 24 08:29:20 2018 From: yongxiang27 at gmail.com (Yongxiang Wu) Date: Tue, 24 Apr 2018 15:29:20 +0200 Subject: [petsc-users] Error: likely due to memory corruption Message-ID: Hello PETSC users: i am using slepc4py to compute a geenral eigenvalue problem. I have compiled PETSC/SLEPC and petsc4py/slepc4py on a cray cluster (hazelhen). When I run my test case, I get the following error. Would anyone provide me the direction to solve it? regards Yong Slepsc begins ... Parallel read in petsc binary matrix for each processor!!! load petsc binary finished Start solving ... [0]PETSC ERROR: [2]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: [2]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ Caught signal number 4 Illegal instruction: Likely due to memory corruption Caught signal number 4 Illegal instruction: Likely due to memory corruption [5]PETSC ERROR: [0]PETSC ERROR: [2]PETSC ERROR: ------------------------------------------------------------------------ Try option -start_in_debugger or -on_error_attach_debugger Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: [2]PETSC ERROR: [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Caught signal number 4 Illegal instruction: Likely due to memory corruption Caught signal number 4 Illegal instruction: Likely due to memory corruption [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [5]PETSC ERROR: [3]PETSC ERROR: [4]PETSC ERROR: ------------------------------------------------------------------------ Try option -start_in_debugger or -on_error_attach_debugger ------------------------------------------------------------------------ or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [3]PETSC ERROR: [1]PETSC ERROR: Caught signal number 4 Illegal instruction: Likely due to memory corruption or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [3]PETSC ERROR: [4]PETSC ERROR: [2]PETSC ERROR: Caught signal number 4 Illegal instruction: Likely due to memory corruption configure using --with-debugging=yes, recompile, link, and run [2]PETSC ERROR: to get more information on the crash. to get more information on the crash. [5]PETSC ERROR: [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [5]PETSC ERROR: [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run configure using --with-debugging=yes, recompile, link, and run [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [5]PETSC ERROR: to get more information on the crash. [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [4]PETSC ERROR: to get more information on the crash. [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors configure using --with-debugging=yes, recompile, link, and run [4]PETSC ERROR: [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run to get more information on the crash. [4]PETSC ERROR: to get more information on the crash. Rank 2 [Tue Apr 24 14:17:09 2018] [c4-3c2s4n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 Rank 0 [Tue Apr 24 14:17:09 2018] [c4-3c2s4n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 Rank 5 [Tue Apr 24 14:17:09 2018] [c4-3c2s4n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5 Rank 1 [Tue Apr 24 14:17:09 2018] [c4-3c2s4n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 Rank 3 [Tue Apr 24 14:17:09 2018] [c4-3c2s4n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 Rank 4 [Tue Apr 24 14:17:09 2018] [c4-3c2s4n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4 _pmiu_daemon(SIGCHLD): [NID 07826] [c4-3c2s4n2] [Tue Apr 24 14:17:09 2018] PE RANK 0 exit signal Aborted [NID 07826] 2018-04-24 14:17:09 Apid 298658: initiated application termination Application 298658 exit codes: 134 Application 298658 resources: utime ~6s, stime ~2s, Rss ~488036, inblocks ~862, outblocks ~0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 24 08:34:03 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Apr 2018 09:34:03 -0400 Subject: [petsc-users] Error: likely due to memory corruption In-Reply-To: References: Message-ID: On Tue, Apr 24, 2018 at 9:29 AM, Yongxiang Wu wrote: > Hello PETSC users: > i am using slepc4py to compute a geenral eigenvalue problem. I have > compiled PETSC/SLEPC and petsc4py/slepc4py on a cray cluster (hazelhen). > When I run my test case, I get the following error. Would anyone provide me > the direction to solve it? > As Satish said, it looks like you are running on hardware that the binary was not compiled for (maybe KNL?). Thanks, Matt > regards > Yong > > Slepsc begins ... > Parallel read in petsc binary matrix for each processor!!! > load petsc binary finished > Start solving ... > [0]PETSC ERROR: [2]PETSC ERROR: ------------------------------ > ------------------------------------------ > [5]PETSC ERROR: ------------------------------ > ------------------------------------------ > [0]PETSC ERROR: [2]PETSC ERROR: [1]PETSC ERROR: > ------------------------------------------------------------------------ > Caught signal number 4 Illegal instruction: Likely due to memory corruption > Caught signal number 4 Illegal instruction: Likely due to memory corruption > [5]PETSC ERROR: [0]PETSC ERROR: [2]PETSC ERROR: > ------------------------------------------------------------------------ > Try option -start_in_debugger or -on_error_attach_debugger > Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: [2]PETSC ERROR: [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > Caught signal number 4 Illegal instruction: Likely due to memory corruption > Caught signal number 4 Illegal instruction: Likely due to memory corruption > [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: [2]PETSC ERROR: or try http://valgrind.org on GNU/linux > and Apple Mac OS X to find memory corruption errors > [5]PETSC ERROR: [3]PETSC ERROR: [4]PETSC ERROR: > ------------------------------------------------------------------------ > Try option -start_in_debugger or -on_error_attach_debugger > ------------------------------------------------------------------------ > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [3]PETSC ERROR: [1]PETSC ERROR: Caught signal number 4 Illegal > instruction: Likely due to memory corruption > or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [3]PETSC ERROR: [4]PETSC ERROR: [2]PETSC ERROR: Caught signal number 4 > Illegal instruction: Likely due to memory corruption > configure using --with-debugging=yes, recompile, link, and run > [2]PETSC ERROR: to get more information on the crash. > to get more information on the crash. > [5]PETSC ERROR: [1]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [5]PETSC ERROR: [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > configure using --with-debugging=yes, recompile, link, and run > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [5]PETSC ERROR: to get more information on the crash. > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [4]PETSC ERROR: to get more information on the crash. > [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > configure using --with-debugging=yes, recompile, link, and run > [4]PETSC ERROR: [3]PETSC ERROR: configure using --with-debugging=yes, > recompile, link, and run > to get more information on the crash. > [4]PETSC ERROR: to get more information on the crash. > Rank 2 [Tue Apr 24 14:17:09 2018] [c4-3c2s4n2] application called > MPI_Abort(MPI_COMM_WORLD, 59) - process 2 > Rank 0 [Tue Apr 24 14:17:09 2018] [c4-3c2s4n2] application called > MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > Rank 5 [Tue Apr 24 14:17:09 2018] [c4-3c2s4n2] application called > MPI_Abort(MPI_COMM_WORLD, 59) - process 5 > Rank 1 [Tue Apr 24 14:17:09 2018] [c4-3c2s4n2] application called > MPI_Abort(MPI_COMM_WORLD, 59) - process 1 > Rank 3 [Tue Apr 24 14:17:09 2018] [c4-3c2s4n2] application called > MPI_Abort(MPI_COMM_WORLD, 59) - process 3 > Rank 4 [Tue Apr 24 14:17:09 2018] [c4-3c2s4n2] application called > MPI_Abort(MPI_COMM_WORLD, 59) - process 4 > _pmiu_daemon(SIGCHLD): [NID 07826] [c4-3c2s4n2] [Tue Apr 24 14:17:09 2018] > PE RANK 0 exit signal Aborted > [NID 07826] 2018-04-24 14:17:09 Apid 298658: initiated application > termination > Application 298658 exit codes: 134 > Application 298658 resources: utime ~6s, stime ~2s, Rss ~488036, inblocks > ~862, outblocks ~0 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.lanthaler at gmail.com Tue Apr 24 08:42:09 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Tue, 24 Apr 2018 15:42:09 +0200 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> Message-ID: Dear Satish and Stefano, Thank you for your answers. I believe I had initially tried to use the option --with-blaslapack-dir=[...], instead of specifying lib and include directly. But that gives me an error message: ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- You set a value for --with-blaslapack-dir=, but /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be used ******************************************************************************* I attach the new configure.log. Do you think there is something wrong with the mkl version that I'm trying to use? Or is it only related to petsc? Cheers, Samuel On 04/24/2018 02:55 PM, Satish Balay wrote: > On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >> Hi there, >> >> I was wondering if you could help me with the correct configuration of >> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure which >> information would be useful to you, but basically the problem seems to be in >> correctly compiling it with intel compiler and the existing mkl library. >> >> The pre-installed mkl version is >> >> intel/17.0.4.196 >> >> I have tried various things and finally, I got it at least to compile with the >> following options (options chosen in reference to the mkl link advisor...): >> >> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 >> --with-scalar-type=complex --download-scalapack --download-mumps=yes >> --download-superlu --with-batch --known-mpi-shared-libraries=1 >> *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a >> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group >> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a >> ${MKLROOT}/lib/intel64/libmkl_sequential.a >> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl" >> *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" > Can you remove the above 2 options [--with-blaslapack-lib, --with-blaslapack-include] and use: > > --with-blas-lapack-dir=${MKLROOT} > > And see if you still have this problem? > > Satish > >> --known-64-bit-blas-indices=0 >> >> After compilation, when trying to compile >> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get linking >> errors (attached). Would anyone be able to help me out here? I really don't >> have a good understanding of this. >> >> I'm attaching the configuration.log file, as well as the linking errors when >> trying to compile ex5f. >> >> Thank you very much in advance! >> >> Best regards, >> Samuel >> -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 2013953 bytes Desc: not available URL: From knepley at gmail.com Tue Apr 24 08:47:26 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Apr 2018 09:47:26 -0400 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> Message-ID: On Tue, Apr 24, 2018 at 9:42 AM, Samuel Lanthaler wrote: > Dear Satish and Stefano, > > Thank you for your answers. I believe I had initially tried to use the > option --with-blaslapack-dir=[...], instead of specifying lib and include > directly. But that gives me an error message: > Satish, should this be --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib ? Matt > ************************************************************ > ******************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > ------------------------------------------------------------ > ------------------- > You set a value for --with-blaslapack-dir=, but > /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be used > ************************************************************ > ******************* > > I attach the new configure.log. Do you think there is something wrong with > the mkl version that I'm trying to use? Or is it only related to petsc? > > Cheers, > Samuel > > > On 04/24/2018 02:55 PM, Satish Balay wrote: > >> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >> >> Hi there, >>> >>> I was wondering if you could help me with the correct configuration of >>> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure >>> which >>> information would be useful to you, but basically the problem seems to >>> be in >>> correctly compiling it with intel compiler and the existing mkl library. >>> >>> The pre-installed mkl version is >>> >>> intel/17.0.4.196 >>> >>> I have tried various things and finally, I got it at least to compile >>> with the >>> following options (options chosen in reference to the mkl link >>> advisor...): >>> >>> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 >>> --with-scalar-type=complex --download-scalapack --download-mumps=yes >>> --download-superlu --with-batch --known-mpi-shared-libraries=1 >>> *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a >>> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group >>> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a >>> ${MKLROOT}/lib/intel64/libmkl_sequential.a >>> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl" >>> *--with-blaslapack-include*="[/opt/intel/compilers_and_libra >>> ries_2017.4.196/linux/mkl/include,/opt/intel/compilers_ >>> and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" >>> >> Can you remove the above 2 options [--with-blaslapack-lib, >> --with-blaslapack-include] and use: >> >> --with-blas-lapack-dir=${MKLROOT} >> >> And see if you still have this problem? >> >> Satish >> >> --known-64-bit-blas-indices=0 >>> >>> After compilation, when trying to compile >>> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get linking >>> errors (attached). Would anyone be able to help me out here? I really >>> don't >>> have a good understanding of this. >>> >>> I'm attaching the configuration.log file, as well as the linking errors >>> when >>> trying to compile ex5f. >>> >>> Thank you very much in advance! >>> >>> Best regards, >>> Samuel >>> >>> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Apr 24 08:50:37 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 24 Apr 2018 08:50:37 -0500 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> Message-ID: >>>>>> Executing: cc -o /tmp/petsc-tyd_Hm/config.libraries/conftest -O /tmp/petsc-tyd_Hm/config.libraries/conftest.o -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl Possible ERROR while running linker: exit code 256 stderr: /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(mkl_semaphore.o): In function `mkl_serv_load_inspector': mkl_semaphore.c:(.text+0x123): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(dgetrs.o): In function `mkl_lapack_dgetrs': dgetrs_gen.f:(.text+0x224): undefined reference to `mkl_blas_dtrsm' dgetrs_gen.f:(.text+0x2b1): undefined reference to `mkl_blas_dtrsm' dgetrs_gen.f:(.text+0x2e7): undefined reference to `mkl_lapack_dlaswp' <<<<<<<< For some reason the compiler is picking up static version of MKL - instead of shared libraries. What do you have for: ls /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ Satish On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > Dear Satish and Stefano, > > Thank you for your answers. I believe I had initially tried to use the option > --with-blaslapack-dir=[...], instead of specifying lib and include directly. > But that gives me an error message: > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > ------------------------------------------------------------------------------- > You set a value for --with-blaslapack-dir=, but > /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be used > ******************************************************************************* > > I attach the new configure.log. Do you think there is something wrong with the > mkl version that I'm trying to use? Or is it only related to petsc? > > Cheers, > Samuel > > > On 04/24/2018 02:55 PM, Satish Balay wrote: > > On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > > > >> Hi there, > >> > >> I was wondering if you could help me with the correct configuration of > >> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure which > >> information would be useful to you, but basically the problem seems to be > >> in > >> correctly compiling it with intel compiler and the existing mkl library. > >> > >> The pre-installed mkl version is > >> > >> intel/17.0.4.196 > >> > >> I have tried various things and finally, I got it at least to compile with > >> the > >> following options (options chosen in reference to the mkl link advisor...): > >> > >> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 > >> --with-scalar-type=complex --download-scalapack --download-mumps=yes > >> --download-superlu --with-batch --known-mpi-shared-libraries=1 > >> *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a > >> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group > >> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a > >> ${MKLROOT}/lib/intel64/libmkl_sequential.a > >> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl" > >> *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" > > Can you remove the above 2 options [--with-blaslapack-lib, > > --with-blaslapack-include] and use: > > > > --with-blas-lapack-dir=${MKLROOT} > > > > And see if you still have this problem? > > > > Satish > > > >> --known-64-bit-blas-indices=0 > >> > >> After compilation, when trying to compile > >> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get linking > >> errors (attached). Would anyone be able to help me out here? I really don't > >> have a good understanding of this. > >> > >> I'm attaching the configuration.log file, as well as the linking errors > >> when > >> trying to compile ex5f. > >> > >> Thank you very much in advance! > >> > >> Best regards, > >> Samuel > >> > > From s.lanthaler at gmail.com Tue Apr 24 08:55:09 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Tue, 24 Apr 2018 15:55:09 +0200 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> Message-ID: <7ad332fc-290d-c8f0-4a60-b0f4b7b02415@gmail.com> Dear Satish, I get the following when doing the ls: [lanthale at daint103 mkl]$ ls /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ libmkl_ao_worker.so libmkl_blacs_sgimpt_lp64.so libmkl_intel_lp64.a libmkl_sequential.so libmkl_avx2.so libmkl_blas95_ilp64.a libmkl_intel_lp64.so libmkl_tbb_thread.a libmkl_avx512_mic.so libmkl_blas95_lp64.a libmkl_intel_thread.a libmkl_tbb_thread.so libmkl_avx512.so libmkl_cdft_core.a libmkl_intel_thread.so libmkl_vml_avx2.so libmkl_avx.so libmkl_cdft_core.so libmkl_lapack95_ilp64.a libmkl_vml_avx512_mic.so libmkl_blacs_intelmpi_ilp64.a libmkl_core.a libmkl_lapack95_lp64.a libmkl_vml_avx512.so libmkl_blacs_intelmpi_ilp64.so libmkl_core.so libmkl_mc3.so libmkl_vml_avx.so libmkl_blacs_intelmpi_lp64.a libmkl_def.so libmkl_mc.so libmkl_vml_cmpt.so libmkl_blacs_intelmpi_lp64.so libmkl_gf_ilp64.a libmkl_pgi_thread.a libmkl_vml_def.so libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.so libmkl_pgi_thread.so libmkl_vml_mc2.so libmkl_blacs_openmpi_ilp64.so libmkl_gf_lp64.a libmkl_rt.so libmkl_vml_mc3.so libmkl_blacs_openmpi_lp64.a libmkl_gf_lp64.so libmkl_scalapack_ilp64.a libmkl_vml_mc.so libmkl_blacs_openmpi_lp64.so libmkl_gnu_thread.a libmkl_scalapack_ilp64.so locale libmkl_blacs_sgimpt_ilp64.a libmkl_gnu_thread.so libmkl_scalapack_lp64.a libmkl_blacs_sgimpt_ilp64.so libmkl_intel_ilp64.a libmkl_scalapack_lp64.so libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_sequential.a Cheers, Samuel On 04/24/2018 03:50 PM, Satish Balay wrote: > Executing: cc -o /tmp/petsc-tyd_Hm/config.libraries/conftest -O /tmp/petsc-tyd_Hm/config.libraries/conftest.o -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl > Possible ERROR while running linker: exit code 256 > stderr: > /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(mkl_semaphore.o): In function `mkl_serv_load_inspector': > mkl_semaphore.c:(.text+0x123): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking > /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(dgetrs.o): In function `mkl_lapack_dgetrs': > dgetrs_gen.f:(.text+0x224): undefined reference to `mkl_blas_dtrsm' > dgetrs_gen.f:(.text+0x2b1): undefined reference to `mkl_blas_dtrsm' > dgetrs_gen.f:(.text+0x2e7): undefined reference to `mkl_lapack_dlaswp' > <<<<<<<< > > For some reason the compiler is picking up static version of MKL - instead of shared libraries. > > What do you have for: > > ls /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ > > Satish > > On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >> Dear Satish and Stefano, >> >> Thank you for your answers. I believe I had initially tried to use the option >> --with-blaslapack-dir=[...], instead of specifying lib and include directly. >> But that gives me an error message: >> >> ******************************************************************************* >> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for >> details): >> ------------------------------------------------------------------------------- >> You set a value for --with-blaslapack-dir=, but >> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be used >> ******************************************************************************* >> >> I attach the new configure.log. Do you think there is something wrong with the >> mkl version that I'm trying to use? Or is it only related to petsc? >> >> Cheers, >> Samuel >> >> >> On 04/24/2018 02:55 PM, Satish Balay wrote: >>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>> >>>> Hi there, >>>> >>>> I was wondering if you could help me with the correct configuration of >>>> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure which >>>> information would be useful to you, but basically the problem seems to be >>>> in >>>> correctly compiling it with intel compiler and the existing mkl library. >>>> >>>> The pre-installed mkl version is >>>> >>>> intel/17.0.4.196 >>>> >>>> I have tried various things and finally, I got it at least to compile with >>>> the >>>> following options (options chosen in reference to the mkl link advisor...): >>>> >>>> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 >>>> --with-scalar-type=complex --download-scalapack --download-mumps=yes >>>> --download-superlu --with-batch --known-mpi-shared-libraries=1 >>>> *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a >>>> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group >>>> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a >>>> ${MKLROOT}/lib/intel64/libmkl_sequential.a >>>> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl" >>>> *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" >>> Can you remove the above 2 options [--with-blaslapack-lib, >>> --with-blaslapack-include] and use: >>> >>> --with-blas-lapack-dir=${MKLROOT} >>> >>> And see if you still have this problem? >>> >>> Satish >>> >>>> --known-64-bit-blas-indices=0 >>>> >>>> After compilation, when trying to compile >>>> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get linking >>>> errors (attached). Would anyone be able to help me out here? I really don't >>>> have a good understanding of this. >>>> >>>> I'm attaching the configuration.log file, as well as the linking errors >>>> when >>>> trying to compile ex5f. >>>> >>>> Thank you very much in advance! >>>> >>>> Best regards, >>>> Samuel >>>> >> From balay at mcs.anl.gov Tue Apr 24 09:03:04 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 24 Apr 2018 09:03:04 -0500 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: <7ad332fc-290d-c8f0-4a60-b0f4b7b02415@gmail.com> References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> <7ad332fc-290d-c8f0-4a60-b0f4b7b02415@gmail.com> Message-ID: Ok - I do see: libmkl_intel_lp64.so libmkl_sequential.so libmkl_core.so - these should be the ones that get picked up. I do not know why 'cc' is picking up static libraries instead of the shared ones. Perhaps it needs an extra option 'LDFLAGS=-dynamic' Satish On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > Dear Satish, > > I get the following when doing the ls: > > [lanthale at daint103 mkl]$ ls > /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ > libmkl_ao_worker.so libmkl_blacs_sgimpt_lp64.so > libmkl_intel_lp64.a libmkl_sequential.so > libmkl_avx2.so libmkl_blas95_ilp64.a libmkl_intel_lp64.so > libmkl_tbb_thread.a > libmkl_avx512_mic.so libmkl_blas95_lp64.a libmkl_intel_thread.a > libmkl_tbb_thread.so > libmkl_avx512.so libmkl_cdft_core.a libmkl_intel_thread.so > libmkl_vml_avx2.so > libmkl_avx.so libmkl_cdft_core.so libmkl_lapack95_ilp64.a > libmkl_vml_avx512_mic.so > libmkl_blacs_intelmpi_ilp64.a libmkl_core.a libmkl_lapack95_lp64.a > libmkl_vml_avx512.so > libmkl_blacs_intelmpi_ilp64.so libmkl_core.so libmkl_mc3.so > libmkl_vml_avx.so > libmkl_blacs_intelmpi_lp64.a libmkl_def.so libmkl_mc.so > libmkl_vml_cmpt.so > libmkl_blacs_intelmpi_lp64.so libmkl_gf_ilp64.a libmkl_pgi_thread.a > libmkl_vml_def.so > libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.so libmkl_pgi_thread.so > libmkl_vml_mc2.so > libmkl_blacs_openmpi_ilp64.so libmkl_gf_lp64.a libmkl_rt.so > libmkl_vml_mc3.so > libmkl_blacs_openmpi_lp64.a libmkl_gf_lp64.so libmkl_scalapack_ilp64.a > libmkl_vml_mc.so > libmkl_blacs_openmpi_lp64.so libmkl_gnu_thread.a libmkl_scalapack_ilp64.so > locale > libmkl_blacs_sgimpt_ilp64.a libmkl_gnu_thread.so libmkl_scalapack_lp64.a > libmkl_blacs_sgimpt_ilp64.so libmkl_intel_ilp64.a libmkl_scalapack_lp64.so > libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_sequential.a > > > Cheers, > Samuel > > On 04/24/2018 03:50 PM, Satish Balay wrote: > > Executing: cc -o /tmp/petsc-tyd_Hm/config.libraries/conftest -O > > /tmp/petsc-tyd_Hm/config.libraries/conftest.o > > -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 > > -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 > > -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl > > Possible ERROR while running linker: exit code 256 > > stderr: > > /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(mkl_semaphore.o): > > In function `mkl_serv_load_inspector': > > mkl_semaphore.c:(.text+0x123): warning: Using 'dlopen' in statically linked > > applications requires at runtime the shared libraries from the glibc version > > used for linking > > /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(dgetrs.o): > > In function `mkl_lapack_dgetrs': > > dgetrs_gen.f:(.text+0x224): undefined reference to `mkl_blas_dtrsm' > > dgetrs_gen.f:(.text+0x2b1): undefined reference to `mkl_blas_dtrsm' > > dgetrs_gen.f:(.text+0x2e7): undefined reference to `mkl_lapack_dlaswp' > > <<<<<<<< > > > > For some reason the compiler is picking up static version of MKL - instead > > of shared libraries. > > > > What do you have for: > > > > ls /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ > > > > Satish > > > > On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > > > >> Dear Satish and Stefano, > >> > >> Thank you for your answers. I believe I had initially tried to use the > >> option > >> --with-blaslapack-dir=[...], instead of specifying lib and include > >> directly. > >> But that gives me an error message: > >> > >> ******************************************************************************* > >> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > >> details): > >> ------------------------------------------------------------------------------- > >> You set a value for --with-blaslapack-dir=, but > >> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be used > >> ******************************************************************************* > >> > >> I attach the new configure.log. Do you think there is something wrong with > >> the > >> mkl version that I'm trying to use? Or is it only related to petsc? > >> > >> Cheers, > >> Samuel > >> > >> > >> On 04/24/2018 02:55 PM, Satish Balay wrote: > >>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >>> > >>>> Hi there, > >>>> > >>>> I was wondering if you could help me with the correct configuration of > >>>> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure which > >>>> information would be useful to you, but basically the problem seems to be > >>>> in > >>>> correctly compiling it with intel compiler and the existing mkl library. > >>>> > >>>> The pre-installed mkl version is > >>>> > >>>> intel/17.0.4.196 > >>>> > >>>> I have tried various things and finally, I got it at least to compile > >>>> with > >>>> the > >>>> following options (options chosen in reference to the mkl link > >>>> advisor...): > >>>> > >>>> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 > >>>> --with-scalar-type=complex --download-scalapack --download-mumps=yes > >>>> --download-superlu --with-batch --known-mpi-shared-libraries=1 > >>>> *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a > >>>> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group > >>>> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a > >>>> ${MKLROOT}/lib/intel64/libmkl_sequential.a > >>>> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl" > >>>> *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" > >>> Can you remove the above 2 options [--with-blaslapack-lib, > >>> --with-blaslapack-include] and use: > >>> > >>> --with-blas-lapack-dir=${MKLROOT} > >>> > >>> And see if you still have this problem? > >>> > >>> Satish > >>> > >>>> --known-64-bit-blas-indices=0 > >>>> > >>>> After compilation, when trying to compile > >>>> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get linking > >>>> errors (attached). Would anyone be able to help me out here? I really > >>>> don't > >>>> have a good understanding of this. > >>>> > >>>> I'm attaching the configuration.log file, as well as the linking errors > >>>> when > >>>> trying to compile ex5f. > >>>> > >>>> Thank you very much in advance! > >>>> > >>>> Best regards, > >>>> Samuel > >>>> > >> > > From s.lanthaler at gmail.com Tue Apr 24 09:08:31 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Tue, 24 Apr 2018 16:08:31 +0200 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> <7ad332fc-290d-c8f0-4a60-b0f4b7b02415@gmail.com> Message-ID: Sorry, I'm confused: Can I add this somehow when doing the ./configure? Or do you mean to add it when compiling ex5f with the configuration I had initially? Cheers, Samuel On 04/24/2018 04:03 PM, Satish Balay wrote: > Ok - I do see: libmkl_intel_lp64.so libmkl_sequential.so libmkl_core.so - these should > be the ones that get picked up. > > I do not know why 'cc' is picking up static libraries instead of the > shared ones. Perhaps it needs an extra option > > 'LDFLAGS=-dynamic' > > Satish > > On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >> Dear Satish, >> >> I get the following when doing the ls: >> >> [lanthale at daint103 mkl]$ ls >> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ >> libmkl_ao_worker.so libmkl_blacs_sgimpt_lp64.so >> libmkl_intel_lp64.a libmkl_sequential.so >> libmkl_avx2.so libmkl_blas95_ilp64.a libmkl_intel_lp64.so >> libmkl_tbb_thread.a >> libmkl_avx512_mic.so libmkl_blas95_lp64.a libmkl_intel_thread.a >> libmkl_tbb_thread.so >> libmkl_avx512.so libmkl_cdft_core.a libmkl_intel_thread.so >> libmkl_vml_avx2.so >> libmkl_avx.so libmkl_cdft_core.so libmkl_lapack95_ilp64.a >> libmkl_vml_avx512_mic.so >> libmkl_blacs_intelmpi_ilp64.a libmkl_core.a libmkl_lapack95_lp64.a >> libmkl_vml_avx512.so >> libmkl_blacs_intelmpi_ilp64.so libmkl_core.so libmkl_mc3.so >> libmkl_vml_avx.so >> libmkl_blacs_intelmpi_lp64.a libmkl_def.so libmkl_mc.so >> libmkl_vml_cmpt.so >> libmkl_blacs_intelmpi_lp64.so libmkl_gf_ilp64.a libmkl_pgi_thread.a >> libmkl_vml_def.so >> libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.so libmkl_pgi_thread.so >> libmkl_vml_mc2.so >> libmkl_blacs_openmpi_ilp64.so libmkl_gf_lp64.a libmkl_rt.so >> libmkl_vml_mc3.so >> libmkl_blacs_openmpi_lp64.a libmkl_gf_lp64.so libmkl_scalapack_ilp64.a >> libmkl_vml_mc.so >> libmkl_blacs_openmpi_lp64.so libmkl_gnu_thread.a libmkl_scalapack_ilp64.so >> locale >> libmkl_blacs_sgimpt_ilp64.a libmkl_gnu_thread.so libmkl_scalapack_lp64.a >> libmkl_blacs_sgimpt_ilp64.so libmkl_intel_ilp64.a libmkl_scalapack_lp64.so >> libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_sequential.a >> >> >> Cheers, >> Samuel >> >> On 04/24/2018 03:50 PM, Satish Balay wrote: >>> Executing: cc -o /tmp/petsc-tyd_Hm/config.libraries/conftest -O >>> /tmp/petsc-tyd_Hm/config.libraries/conftest.o >>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 >>> -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 >>> -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl >>> Possible ERROR while running linker: exit code 256 >>> stderr: >>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(mkl_semaphore.o): >>> In function `mkl_serv_load_inspector': >>> mkl_semaphore.c:(.text+0x123): warning: Using 'dlopen' in statically linked >>> applications requires at runtime the shared libraries from the glibc version >>> used for linking >>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(dgetrs.o): >>> In function `mkl_lapack_dgetrs': >>> dgetrs_gen.f:(.text+0x224): undefined reference to `mkl_blas_dtrsm' >>> dgetrs_gen.f:(.text+0x2b1): undefined reference to `mkl_blas_dtrsm' >>> dgetrs_gen.f:(.text+0x2e7): undefined reference to `mkl_lapack_dlaswp' >>> <<<<<<<< >>> >>> For some reason the compiler is picking up static version of MKL - instead >>> of shared libraries. >>> >>> What do you have for: >>> >>> ls /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ >>> >>> Satish >>> >>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>> >>>> Dear Satish and Stefano, >>>> >>>> Thank you for your answers. I believe I had initially tried to use the >>>> option >>>> --with-blaslapack-dir=[...], instead of specifying lib and include >>>> directly. >>>> But that gives me an error message: >>>> >>>> ******************************************************************************* >>>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for >>>> details): >>>> ------------------------------------------------------------------------------- >>>> You set a value for --with-blaslapack-dir=, but >>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be used >>>> ******************************************************************************* >>>> >>>> I attach the new configure.log. Do you think there is something wrong with >>>> the >>>> mkl version that I'm trying to use? Or is it only related to petsc? >>>> >>>> Cheers, >>>> Samuel >>>> >>>> >>>> On 04/24/2018 02:55 PM, Satish Balay wrote: >>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>>>> >>>>>> Hi there, >>>>>> >>>>>> I was wondering if you could help me with the correct configuration of >>>>>> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure which >>>>>> information would be useful to you, but basically the problem seems to be >>>>>> in >>>>>> correctly compiling it with intel compiler and the existing mkl library. >>>>>> >>>>>> The pre-installed mkl version is >>>>>> >>>>>> intel/17.0.4.196 >>>>>> >>>>>> I have tried various things and finally, I got it at least to compile >>>>>> with >>>>>> the >>>>>> following options (options chosen in reference to the mkl link >>>>>> advisor...): >>>>>> >>>>>> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 >>>>>> --with-scalar-type=complex --download-scalapack --download-mumps=yes >>>>>> --download-superlu --with-batch --known-mpi-shared-libraries=1 >>>>>> *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a >>>>>> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group >>>>>> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a >>>>>> ${MKLROOT}/lib/intel64/libmkl_sequential.a >>>>>> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl" >>>>>> *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" >>>>> Can you remove the above 2 options [--with-blaslapack-lib, >>>>> --with-blaslapack-include] and use: >>>>> >>>>> --with-blas-lapack-dir=${MKLROOT} >>>>> >>>>> And see if you still have this problem? >>>>> >>>>> Satish >>>>> >>>>>> --known-64-bit-blas-indices=0 >>>>>> >>>>>> After compilation, when trying to compile >>>>>> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get linking >>>>>> errors (attached). Would anyone be able to help me out here? I really >>>>>> don't >>>>>> have a good understanding of this. >>>>>> >>>>>> I'm attaching the configuration.log file, as well as the linking errors >>>>>> when >>>>>> trying to compile ex5f. >>>>>> >>>>>> Thank you very much in advance! >>>>>> >>>>>> Best regards, >>>>>> Samuel >>>>>> >> From balay at mcs.anl.gov Tue Apr 24 09:12:58 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 24 Apr 2018 09:12:58 -0500 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> Message-ID: BTW: In this case configure succeed - but your example compile command failed. Configure is using: BLAS/LAPACK: -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_blas95_lp64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_lapack95_lp64 -Wl,--start-group -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_intel_lp64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_sequential -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_core -Wl,--end-group -lpthread -lm -ldl however your link command has: ftn -O -o ex5f ex5f.o -L/users/lanthale/petsc-git/arch-complex/lib -Wl,-rpath,/users/lanthale/petsc-git/arch-complex/lib -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -Wl,--start-group -Wl,--end-group -lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl -lstdc++ -ldl This does not match configure detected list. PETSC_WITH_EXTERNAL_LIB = -L/users/lanthale/petsc-git/arch-complex/lib -Wl,-rpath,/users/lanthale/petsc-git/arch-complex/lib -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -Wl,--start-group -Wl,--end-group -lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl -lstdc++ -ldl Ok - this formatting got changed [a potential bug in petsc configure] - and perhaps the reason for this failure. If you still have this build - you could manually edit arch-complex/lib/petsc/conf/petscvariables and change the above to the following and see if it makes a difference PETSC_WITH_EXTERNAL_LIB = -L/users/lanthale/petsc-git/arch-complex/lib -Wl,-rpath,/users/lanthale/petsc-git/arch-complex/lib -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -lpthread -lm -ldl -lstdc++ -ldl Satish On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > Hi there, > > I was wondering if you could help me with the correct configuration of > PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure which > information would be useful to you, but basically the problem seems to be in > correctly compiling it with intel compiler and the existing mkl library. > > The pre-installed mkl version is > > intel/17.0.4.196 > > I have tried various things and finally, I got it at least to compile with the > following options (options chosen in reference to the mkl link advisor...): > > ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 > --with-scalar-type=complex --download-scalapack --download-mumps=yes > --download-superlu --with-batch --known-mpi-shared-libraries=1 > *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a > ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group > ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a > ${MKLROOT}/lib/intel64/libmkl_sequential.a > ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl" > *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" > --known-64-bit-blas-indices=0 > > After compilation, when trying to compile > /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get linking > errors (attached). Would anyone be able to help me out here? I really don't > have a good understanding of this. > > I'm attaching the configuration.log file, as well as the linking errors when > trying to compile ex5f. > > Thank you very much in advance! > > Best regards, > Samuel > From balay at mcs.anl.gov Tue Apr 24 09:14:21 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 24 Apr 2018 09:14:21 -0500 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> <7ad332fc-290d-c8f0-4a60-b0f4b7b02415@gmail.com> Message-ID: Sorry I should have clarified - its a configure option. Satish On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > Sorry, I'm confused: Can I add this somehow when doing the ./configure? Or do > you mean to add it when compiling ex5f with the configuration I had initially? > > Cheers, > Samuel > > > On 04/24/2018 04:03 PM, Satish Balay wrote: > > Ok - I do see: libmkl_intel_lp64.so libmkl_sequential.so libmkl_core.so - > > these should > > be the ones that get picked up. > > > > I do not know why 'cc' is picking up static libraries instead of the > > shared ones. Perhaps it needs an extra option > > > > 'LDFLAGS=-dynamic' > > > > Satish > > > > On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > > > >> Dear Satish, > >> > >> I get the following when doing the ls: > >> > >> [lanthale at daint103 mkl]$ ls > >> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ > >> libmkl_ao_worker.so libmkl_blacs_sgimpt_lp64.so > >> libmkl_intel_lp64.a libmkl_sequential.so > >> libmkl_avx2.so libmkl_blas95_ilp64.a libmkl_intel_lp64.so > >> libmkl_tbb_thread.a > >> libmkl_avx512_mic.so libmkl_blas95_lp64.a libmkl_intel_thread.a > >> libmkl_tbb_thread.so > >> libmkl_avx512.so libmkl_cdft_core.a libmkl_intel_thread.so > >> libmkl_vml_avx2.so > >> libmkl_avx.so libmkl_cdft_core.so libmkl_lapack95_ilp64.a > >> libmkl_vml_avx512_mic.so > >> libmkl_blacs_intelmpi_ilp64.a libmkl_core.a libmkl_lapack95_lp64.a > >> libmkl_vml_avx512.so > >> libmkl_blacs_intelmpi_ilp64.so libmkl_core.so libmkl_mc3.so > >> libmkl_vml_avx.so > >> libmkl_blacs_intelmpi_lp64.a libmkl_def.so libmkl_mc.so > >> libmkl_vml_cmpt.so > >> libmkl_blacs_intelmpi_lp64.so libmkl_gf_ilp64.a libmkl_pgi_thread.a > >> libmkl_vml_def.so > >> libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.so libmkl_pgi_thread.so > >> libmkl_vml_mc2.so > >> libmkl_blacs_openmpi_ilp64.so libmkl_gf_lp64.a libmkl_rt.so > >> libmkl_vml_mc3.so > >> libmkl_blacs_openmpi_lp64.a libmkl_gf_lp64.so libmkl_scalapack_ilp64.a > >> libmkl_vml_mc.so > >> libmkl_blacs_openmpi_lp64.so libmkl_gnu_thread.a > >> libmkl_scalapack_ilp64.so > >> locale > >> libmkl_blacs_sgimpt_ilp64.a libmkl_gnu_thread.so > >> libmkl_scalapack_lp64.a > >> libmkl_blacs_sgimpt_ilp64.so libmkl_intel_ilp64.a > >> libmkl_scalapack_lp64.so > >> libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_sequential.a > >> > >> > >> Cheers, > >> Samuel > >> > >> On 04/24/2018 03:50 PM, Satish Balay wrote: > >>> Executing: cc -o /tmp/petsc-tyd_Hm/config.libraries/conftest -O > >>> /tmp/petsc-tyd_Hm/config.libraries/conftest.o > >>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 > >>> -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 > >>> -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl > >>> Possible ERROR while running linker: exit code 256 > >>> stderr: > >>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(mkl_semaphore.o): > >>> In function `mkl_serv_load_inspector': > >>> mkl_semaphore.c:(.text+0x123): warning: Using 'dlopen' in statically > >>> linked > >>> applications requires at runtime the shared libraries from the glibc > >>> version > >>> used for linking > >>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(dgetrs.o): > >>> In function `mkl_lapack_dgetrs': > >>> dgetrs_gen.f:(.text+0x224): undefined reference to `mkl_blas_dtrsm' > >>> dgetrs_gen.f:(.text+0x2b1): undefined reference to `mkl_blas_dtrsm' > >>> dgetrs_gen.f:(.text+0x2e7): undefined reference to `mkl_lapack_dlaswp' > >>> <<<<<<<< > >>> > >>> For some reason the compiler is picking up static version of MKL - instead > >>> of shared libraries. > >>> > >>> What do you have for: > >>> > >>> ls /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ > >>> > >>> Satish > >>> > >>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >>> > >>>> Dear Satish and Stefano, > >>>> > >>>> Thank you for your answers. I believe I had initially tried to use the > >>>> option > >>>> --with-blaslapack-dir=[...], instead of specifying lib and include > >>>> directly. > >>>> But that gives me an error message: > >>>> > >>>> ******************************************************************************* > >>>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > >>>> for > >>>> details): > >>>> ------------------------------------------------------------------------------- > >>>> You set a value for --with-blaslapack-dir=, but > >>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be used > >>>> ******************************************************************************* > >>>> > >>>> I attach the new configure.log. Do you think there is something wrong > >>>> with > >>>> the > >>>> mkl version that I'm trying to use? Or is it only related to petsc? > >>>> > >>>> Cheers, > >>>> Samuel > >>>> > >>>> > >>>> On 04/24/2018 02:55 PM, Satish Balay wrote: > >>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >>>>> > >>>>>> Hi there, > >>>>>> > >>>>>> I was wondering if you could help me with the correct configuration of > >>>>>> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure > >>>>>> which > >>>>>> information would be useful to you, but basically the problem seems to > >>>>>> be > >>>>>> in > >>>>>> correctly compiling it with intel compiler and the existing mkl > >>>>>> library. > >>>>>> > >>>>>> The pre-installed mkl version is > >>>>>> > >>>>>> intel/17.0.4.196 > >>>>>> > >>>>>> I have tried various things and finally, I got it at least to compile > >>>>>> with > >>>>>> the > >>>>>> following options (options chosen in reference to the mkl link > >>>>>> advisor...): > >>>>>> > >>>>>> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 > >>>>>> --with-scalar-type=complex --download-scalapack --download-mumps=yes > >>>>>> --download-superlu --with-batch --known-mpi-shared-libraries=1 > >>>>>> *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a > >>>>>> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group > >>>>>> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a > >>>>>> ${MKLROOT}/lib/intel64/libmkl_sequential.a > >>>>>> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm > >>>>>> -ldl" > >>>>>> *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" > >>>>> Can you remove the above 2 options [--with-blaslapack-lib, > >>>>> --with-blaslapack-include] and use: > >>>>> > >>>>> --with-blas-lapack-dir=${MKLROOT} > >>>>> > >>>>> And see if you still have this problem? > >>>>> > >>>>> Satish > >>>>> > >>>>>> --known-64-bit-blas-indices=0 > >>>>>> > >>>>>> After compilation, when trying to compile > >>>>>> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get > >>>>>> linking > >>>>>> errors (attached). Would anyone be able to help me out here? I really > >>>>>> don't > >>>>>> have a good understanding of this. > >>>>>> > >>>>>> I'm attaching the configuration.log file, as well as the linking errors > >>>>>> when > >>>>>> trying to compile ex5f. > >>>>>> > >>>>>> Thank you very much in advance! > >>>>>> > >>>>>> Best regards, > >>>>>> Samuel > >>>>>> > >> > > > From s.lanthaler at gmail.com Tue Apr 24 09:15:01 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Tue, 24 Apr 2018 16:15:01 +0200 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> <7ad332fc-290d-c8f0-4a60-b0f4b7b02415@gmail.com> Message-ID: <336e0555-fb21-76d3-ee45-ce224ba1d540@gmail.com> ... wait a minute, I found this on the cluster website Regardless of which programming environment (compiler suite) you are using, you should always compile code with the Cray wrapper commands (|cc| for C code, |CC| for C++ code and |ftn| for Fortran code). _The compiler wrappers produce statically-linked executables by default; for dynamic libraries use the __|-dynamic|__flag or __|export CRAYPE_LINK_TYPE=dynamic|__before building._ Note that dynamic linking is the default when the cudatoolkit module is loaded. You should include the compiler flags appropriate to the underlying compiler suite (Intel, GNU, PGI, Cray). There are also specific options for the wrappers themselves: see the man pages of |ftn|, |cc|, and |CC| for details || That sounds like what you were saying Satish. I'll try again with "|export CRAYPE_LINK_TYPE=dynamic"| and come back to you! Thanks. On 04/24/2018 04:03 PM, Satish Balay wrote: > Ok - I do see: libmkl_intel_lp64.so libmkl_sequential.so libmkl_core.so - these should > be the ones that get picked up. > > I do not know why 'cc' is picking up static libraries instead of the > shared ones. Perhaps it needs an extra option > > 'LDFLAGS=-dynamic' > > Satish > > On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >> Dear Satish, >> >> I get the following when doing the ls: >> >> [lanthale at daint103 mkl]$ ls >> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ >> libmkl_ao_worker.so libmkl_blacs_sgimpt_lp64.so >> libmkl_intel_lp64.a libmkl_sequential.so >> libmkl_avx2.so libmkl_blas95_ilp64.a libmkl_intel_lp64.so >> libmkl_tbb_thread.a >> libmkl_avx512_mic.so libmkl_blas95_lp64.a libmkl_intel_thread.a >> libmkl_tbb_thread.so >> libmkl_avx512.so libmkl_cdft_core.a libmkl_intel_thread.so >> libmkl_vml_avx2.so >> libmkl_avx.so libmkl_cdft_core.so libmkl_lapack95_ilp64.a >> libmkl_vml_avx512_mic.so >> libmkl_blacs_intelmpi_ilp64.a libmkl_core.a libmkl_lapack95_lp64.a >> libmkl_vml_avx512.so >> libmkl_blacs_intelmpi_ilp64.so libmkl_core.so libmkl_mc3.so >> libmkl_vml_avx.so >> libmkl_blacs_intelmpi_lp64.a libmkl_def.so libmkl_mc.so >> libmkl_vml_cmpt.so >> libmkl_blacs_intelmpi_lp64.so libmkl_gf_ilp64.a libmkl_pgi_thread.a >> libmkl_vml_def.so >> libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.so libmkl_pgi_thread.so >> libmkl_vml_mc2.so >> libmkl_blacs_openmpi_ilp64.so libmkl_gf_lp64.a libmkl_rt.so >> libmkl_vml_mc3.so >> libmkl_blacs_openmpi_lp64.a libmkl_gf_lp64.so libmkl_scalapack_ilp64.a >> libmkl_vml_mc.so >> libmkl_blacs_openmpi_lp64.so libmkl_gnu_thread.a libmkl_scalapack_ilp64.so >> locale >> libmkl_blacs_sgimpt_ilp64.a libmkl_gnu_thread.so libmkl_scalapack_lp64.a >> libmkl_blacs_sgimpt_ilp64.so libmkl_intel_ilp64.a libmkl_scalapack_lp64.so >> libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_sequential.a >> >> >> Cheers, >> Samuel >> >> On 04/24/2018 03:50 PM, Satish Balay wrote: >>> Executing: cc -o /tmp/petsc-tyd_Hm/config.libraries/conftest -O >>> /tmp/petsc-tyd_Hm/config.libraries/conftest.o >>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 >>> -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 >>> -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl >>> Possible ERROR while running linker: exit code 256 >>> stderr: >>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(mkl_semaphore.o): >>> In function `mkl_serv_load_inspector': >>> mkl_semaphore.c:(.text+0x123): warning: Using 'dlopen' in statically linked >>> applications requires at runtime the shared libraries from the glibc version >>> used for linking >>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(dgetrs.o): >>> In function `mkl_lapack_dgetrs': >>> dgetrs_gen.f:(.text+0x224): undefined reference to `mkl_blas_dtrsm' >>> dgetrs_gen.f:(.text+0x2b1): undefined reference to `mkl_blas_dtrsm' >>> dgetrs_gen.f:(.text+0x2e7): undefined reference to `mkl_lapack_dlaswp' >>> <<<<<<<< >>> >>> For some reason the compiler is picking up static version of MKL - instead >>> of shared libraries. >>> >>> What do you have for: >>> >>> ls /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ >>> >>> Satish >>> >>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>> >>>> Dear Satish and Stefano, >>>> >>>> Thank you for your answers. I believe I had initially tried to use the >>>> option >>>> --with-blaslapack-dir=[...], instead of specifying lib and include >>>> directly. >>>> But that gives me an error message: >>>> >>>> ******************************************************************************* >>>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for >>>> details): >>>> ------------------------------------------------------------------------------- >>>> You set a value for --with-blaslapack-dir=, but >>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be used >>>> ******************************************************************************* >>>> >>>> I attach the new configure.log. Do you think there is something wrong with >>>> the >>>> mkl version that I'm trying to use? Or is it only related to petsc? >>>> >>>> Cheers, >>>> Samuel >>>> >>>> >>>> On 04/24/2018 02:55 PM, Satish Balay wrote: >>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>>>> >>>>>> Hi there, >>>>>> >>>>>> I was wondering if you could help me with the correct configuration of >>>>>> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure which >>>>>> information would be useful to you, but basically the problem seems to be >>>>>> in >>>>>> correctly compiling it with intel compiler and the existing mkl library. >>>>>> >>>>>> The pre-installed mkl version is >>>>>> >>>>>> intel/17.0.4.196 >>>>>> >>>>>> I have tried various things and finally, I got it at least to compile >>>>>> with >>>>>> the >>>>>> following options (options chosen in reference to the mkl link >>>>>> advisor...): >>>>>> >>>>>> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 >>>>>> --with-scalar-type=complex --download-scalapack --download-mumps=yes >>>>>> --download-superlu --with-batch --known-mpi-shared-libraries=1 >>>>>> *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a >>>>>> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group >>>>>> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a >>>>>> ${MKLROOT}/lib/intel64/libmkl_sequential.a >>>>>> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl" >>>>>> *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" >>>>> Can you remove the above 2 options [--with-blaslapack-lib, >>>>> --with-blaslapack-include] and use: >>>>> >>>>> --with-blas-lapack-dir=${MKLROOT} >>>>> >>>>> And see if you still have this problem? >>>>> >>>>> Satish >>>>> >>>>>> --known-64-bit-blas-indices=0 >>>>>> >>>>>> After compilation, when trying to compile >>>>>> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get linking >>>>>> errors (attached). Would anyone be able to help me out here? I really >>>>>> don't >>>>>> have a good understanding of this. >>>>>> >>>>>> I'm attaching the configuration.log file, as well as the linking errors >>>>>> when >>>>>> trying to compile ex5f. >>>>>> >>>>>> Thank you very much in advance! >>>>>> >>>>>> Best regards, >>>>>> Samuel >>>>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Apr 24 09:15:38 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 24 Apr 2018 09:15:38 -0500 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: <560EA9E4-DAE7-433C-B4DF-9765D5C5A4ED@gmail.com> References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> <465A623F-EF8A-4341-A38A-2A8F212BD91E@gmail.com> <560EA9E4-DAE7-433C-B4DF-9765D5C5A4ED@gmail.com> Message-ID: If this were to work - you might also need FOPTFLAGS='-mkl=sequential' CXXOPTFLAGS='-mkl=sequential' [perhaps CFLAGS/FFLAGS/CXXFLAGS is more suited for this option] Satish On Tue, 24 Apr 2018, Stefano Zampini wrote: > sorry, the last option should be ?COPTFLAGS='-mkl=sequential' > > > On Apr 24, 2018, at 3:53 PM, Stefano Zampini wrote: > > > > Petsc configure should be able to detect MKL and add the proper linking by just using the -with-blaslapack-dir= option. > > Alternatively, you can use --COPTFLAGS=?-mkl-sequential? > > > >> On Apr 24, 2018, at 10:26 AM, Samuel Lanthaler wrote: > >> > >> > > > > From s.lanthaler at gmail.com Tue Apr 24 11:23:12 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Tue, 24 Apr 2018 18:23:12 +0200 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> <7ad332fc-290d-c8f0-4a60-b0f4b7b02415@gmail.com> Message-ID: <5094d7bb-f74c-8fee-b1c4-3b3ded0aaea1@gmail.com> Dear Satish, As you suggested, the configure appears to work with ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 --with-scalar-type=complex --download-scalapack --download-mumps=yes --download-superlu --with-batch --known-mpi-shared-libraries=1 --with-blaslapack-dir=${MKLROOT} --known-64-bit-blas-indices=0 --LDFLAGS="-dynamic" However, after doing the reconfigure, I tried to compile using make, and ran into trouble with CLINKER: [...] CC arch-complex/obj/sys/classes/draw/impls/x/xops.o CLINKER arch-complex/lib/libpetsc.so.3.09.0 Warning: -dynamic was already seen on command line, overriding with -shared. /usr/bin/ld: arch-complex/obj/sys/fsrc/somefort.o: relocation R_X86_64_32 against `petscfortran5_' can not be used when making a shared object; recompile with -fPIC arch-complex/obj/sys/fsrc/somefort.o: error adding symbols: Bad value [...] **************************ERROR************************************* Error during compile, check arch-complex/lib/petsc/conf/make.log Send it and arch-complex/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov ******************************************************************** makefile:30: recipe for target 'all' failed make: *** [all] Error 1 Any ideas? Again, I'm attaching the .log files. On 04/24/2018 04:14 PM, Satish Balay wrote: > Sorry I should have clarified - its a configure option. > > Satish > > On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >> Sorry, I'm confused: Can I add this somehow when doing the ./configure? Or do >> you mean to add it when compiling ex5f with the configuration I had initially? >> >> Cheers, >> Samuel >> >> >> On 04/24/2018 04:03 PM, Satish Balay wrote: >>> Ok - I do see: libmkl_intel_lp64.so libmkl_sequential.so libmkl_core.so - >>> these should >>> be the ones that get picked up. >>> >>> I do not know why 'cc' is picking up static libraries instead of the >>> shared ones. Perhaps it needs an extra option >>> >>> 'LDFLAGS=-dynamic' >>> >>> Satish >>> >>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>> >>>> Dear Satish, >>>> >>>> I get the following when doing the ls: >>>> >>>> [lanthale at daint103 mkl]$ ls >>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ >>>> libmkl_ao_worker.so libmkl_blacs_sgimpt_lp64.so >>>> libmkl_intel_lp64.a libmkl_sequential.so >>>> libmkl_avx2.so libmkl_blas95_ilp64.a libmkl_intel_lp64.so >>>> libmkl_tbb_thread.a >>>> libmkl_avx512_mic.so libmkl_blas95_lp64.a libmkl_intel_thread.a >>>> libmkl_tbb_thread.so >>>> libmkl_avx512.so libmkl_cdft_core.a libmkl_intel_thread.so >>>> libmkl_vml_avx2.so >>>> libmkl_avx.so libmkl_cdft_core.so libmkl_lapack95_ilp64.a >>>> libmkl_vml_avx512_mic.so >>>> libmkl_blacs_intelmpi_ilp64.a libmkl_core.a libmkl_lapack95_lp64.a >>>> libmkl_vml_avx512.so >>>> libmkl_blacs_intelmpi_ilp64.so libmkl_core.so libmkl_mc3.so >>>> libmkl_vml_avx.so >>>> libmkl_blacs_intelmpi_lp64.a libmkl_def.so libmkl_mc.so >>>> libmkl_vml_cmpt.so >>>> libmkl_blacs_intelmpi_lp64.so libmkl_gf_ilp64.a libmkl_pgi_thread.a >>>> libmkl_vml_def.so >>>> libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.so libmkl_pgi_thread.so >>>> libmkl_vml_mc2.so >>>> libmkl_blacs_openmpi_ilp64.so libmkl_gf_lp64.a libmkl_rt.so >>>> libmkl_vml_mc3.so >>>> libmkl_blacs_openmpi_lp64.a libmkl_gf_lp64.so libmkl_scalapack_ilp64.a >>>> libmkl_vml_mc.so >>>> libmkl_blacs_openmpi_lp64.so libmkl_gnu_thread.a >>>> libmkl_scalapack_ilp64.so >>>> locale >>>> libmkl_blacs_sgimpt_ilp64.a libmkl_gnu_thread.so >>>> libmkl_scalapack_lp64.a >>>> libmkl_blacs_sgimpt_ilp64.so libmkl_intel_ilp64.a >>>> libmkl_scalapack_lp64.so >>>> libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_sequential.a >>>> >>>> >>>> Cheers, >>>> Samuel >>>> >>>> On 04/24/2018 03:50 PM, Satish Balay wrote: >>>>> Executing: cc -o /tmp/petsc-tyd_Hm/config.libraries/conftest -O >>>>> /tmp/petsc-tyd_Hm/config.libraries/conftest.o >>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 >>>>> -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 >>>>> -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl >>>>> Possible ERROR while running linker: exit code 256 >>>>> stderr: >>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(mkl_semaphore.o): >>>>> In function `mkl_serv_load_inspector': >>>>> mkl_semaphore.c:(.text+0x123): warning: Using 'dlopen' in statically >>>>> linked >>>>> applications requires at runtime the shared libraries from the glibc >>>>> version >>>>> used for linking >>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(dgetrs.o): >>>>> In function `mkl_lapack_dgetrs': >>>>> dgetrs_gen.f:(.text+0x224): undefined reference to `mkl_blas_dtrsm' >>>>> dgetrs_gen.f:(.text+0x2b1): undefined reference to `mkl_blas_dtrsm' >>>>> dgetrs_gen.f:(.text+0x2e7): undefined reference to `mkl_lapack_dlaswp' >>>>> <<<<<<<< >>>>> >>>>> For some reason the compiler is picking up static version of MKL - instead >>>>> of shared libraries. >>>>> >>>>> What do you have for: >>>>> >>>>> ls /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ >>>>> >>>>> Satish >>>>> >>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>>>> >>>>>> Dear Satish and Stefano, >>>>>> >>>>>> Thank you for your answers. I believe I had initially tried to use the >>>>>> option >>>>>> --with-blaslapack-dir=[...], instead of specifying lib and include >>>>>> directly. >>>>>> But that gives me an error message: >>>>>> >>>>>> ******************************************************************************* >>>>>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >>>>>> for >>>>>> details): >>>>>> ------------------------------------------------------------------------------- >>>>>> You set a value for --with-blaslapack-dir=, but >>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be used >>>>>> ******************************************************************************* >>>>>> >>>>>> I attach the new configure.log. Do you think there is something wrong >>>>>> with >>>>>> the >>>>>> mkl version that I'm trying to use? Or is it only related to petsc? >>>>>> >>>>>> Cheers, >>>>>> Samuel >>>>>> >>>>>> >>>>>> On 04/24/2018 02:55 PM, Satish Balay wrote: >>>>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>>>>>> >>>>>>>> Hi there, >>>>>>>> >>>>>>>> I was wondering if you could help me with the correct configuration of >>>>>>>> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure >>>>>>>> which >>>>>>>> information would be useful to you, but basically the problem seems to >>>>>>>> be >>>>>>>> in >>>>>>>> correctly compiling it with intel compiler and the existing mkl >>>>>>>> library. >>>>>>>> >>>>>>>> The pre-installed mkl version is >>>>>>>> >>>>>>>> intel/17.0.4.196 >>>>>>>> >>>>>>>> I have tried various things and finally, I got it at least to compile >>>>>>>> with >>>>>>>> the >>>>>>>> following options (options chosen in reference to the mkl link >>>>>>>> advisor...): >>>>>>>> >>>>>>>> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 >>>>>>>> --with-scalar-type=complex --download-scalapack --download-mumps=yes >>>>>>>> --download-superlu --with-batch --known-mpi-shared-libraries=1 >>>>>>>> *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a >>>>>>>> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group >>>>>>>> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a >>>>>>>> ${MKLROOT}/lib/intel64/libmkl_sequential.a >>>>>>>> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm >>>>>>>> -ldl" >>>>>>>> *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" >>>>>>> Can you remove the above 2 options [--with-blaslapack-lib, >>>>>>> --with-blaslapack-include] and use: >>>>>>> >>>>>>> --with-blas-lapack-dir=${MKLROOT} >>>>>>> >>>>>>> And see if you still have this problem? >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>>> --known-64-bit-blas-indices=0 >>>>>>>> >>>>>>>> After compilation, when trying to compile >>>>>>>> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get >>>>>>>> linking >>>>>>>> errors (attached). Would anyone be able to help me out here? I really >>>>>>>> don't >>>>>>>> have a good understanding of this. >>>>>>>> >>>>>>>> I'm attaching the configuration.log file, as well as the linking errors >>>>>>>> when >>>>>>>> trying to compile ex5f. >>>>>>>> >>>>>>>> Thank you very much in advance! >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Samuel >>>>>>>> >> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: text/x-log Size: 86371 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 14667608 bytes Desc: not available URL: From balay at mcs.anl.gov Tue Apr 24 11:33:57 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 24 Apr 2018 11:33:57 -0500 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: <5094d7bb-f74c-8fee-b1c4-3b3ded0aaea1@gmail.com> References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> <7ad332fc-290d-c8f0-4a60-b0f4b7b02415@gmail.com> <5094d7bb-f74c-8fee-b1c4-3b3ded0aaea1@gmail.com> Message-ID: Hm - I'm not sure why configure didn't try and set -fPIC compiler options. Workarround is: CFLAGS=-fPIC FFLAGS=-fPICC CXXFLAGS=-fPIC or alternatively - you can disable sharedlibraries --with-shared-libraries=0 Satish On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > Dear Satish, > > As you suggested, the configure appears to work with > > ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 > --with-scalar-type=complex --download-scalapack --download-mumps=yes > --download-superlu --with-batch --known-mpi-shared-libraries=1 > --with-blaslapack-dir=${MKLROOT} --known-64-bit-blas-indices=0 > --LDFLAGS="-dynamic" > > However, after doing the reconfigure, I tried to compile using make, and ran > into trouble with CLINKER: > > [...] > CC arch-complex/obj/sys/classes/draw/impls/x/xops.o > CLINKER arch-complex/lib/libpetsc.so.3.09.0 > Warning: > -dynamic was already seen on command line, overriding with -shared. > /usr/bin/ld: arch-complex/obj/sys/fsrc/somefort.o: relocation R_X86_64_32 > against `petscfortran5_' can not be used when making a shared object; > recompile with -fPIC > arch-complex/obj/sys/fsrc/somefort.o: error adding symbols: Bad value > [...] > **************************ERROR************************************* > Error during compile, check arch-complex/lib/petsc/conf/make.log > Send it and arch-complex/lib/petsc/conf/configure.log to > petsc-maint at mcs.anl.gov > ******************************************************************** > makefile:30: recipe for target 'all' failed > make: *** [all] Error 1 > > Any ideas? Again, I'm attaching the .log files. > > > > On 04/24/2018 04:14 PM, Satish Balay wrote: > > Sorry I should have clarified - its a configure option. > > > > Satish > > > > On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > > > >> Sorry, I'm confused: Can I add this somehow when doing the ./configure? Or > >> do > >> you mean to add it when compiling ex5f with the configuration I had > >> initially? > >> > >> Cheers, > >> Samuel > >> > >> > >> On 04/24/2018 04:03 PM, Satish Balay wrote: > >>> Ok - I do see: libmkl_intel_lp64.so libmkl_sequential.so libmkl_core.so - > >>> these should > >>> be the ones that get picked up. > >>> > >>> I do not know why 'cc' is picking up static libraries instead of the > >>> shared ones. Perhaps it needs an extra option > >>> > >>> 'LDFLAGS=-dynamic' > >>> > >>> Satish > >>> > >>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >>> > >>>> Dear Satish, > >>>> > >>>> I get the following when doing the ls: > >>>> > >>>> [lanthale at daint103 mkl]$ ls > >>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ > >>>> libmkl_ao_worker.so libmkl_blacs_sgimpt_lp64.so > >>>> libmkl_intel_lp64.a libmkl_sequential.so > >>>> libmkl_avx2.so libmkl_blas95_ilp64.a > >>>> libmkl_intel_lp64.so > >>>> libmkl_tbb_thread.a > >>>> libmkl_avx512_mic.so libmkl_blas95_lp64.a > >>>> libmkl_intel_thread.a > >>>> libmkl_tbb_thread.so > >>>> libmkl_avx512.so libmkl_cdft_core.a libmkl_intel_thread.so > >>>> libmkl_vml_avx2.so > >>>> libmkl_avx.so libmkl_cdft_core.so > >>>> libmkl_lapack95_ilp64.a > >>>> libmkl_vml_avx512_mic.so > >>>> libmkl_blacs_intelmpi_ilp64.a libmkl_core.a libmkl_lapack95_lp64.a > >>>> libmkl_vml_avx512.so > >>>> libmkl_blacs_intelmpi_ilp64.so libmkl_core.so libmkl_mc3.so > >>>> libmkl_vml_avx.so > >>>> libmkl_blacs_intelmpi_lp64.a libmkl_def.so libmkl_mc.so > >>>> libmkl_vml_cmpt.so > >>>> libmkl_blacs_intelmpi_lp64.so libmkl_gf_ilp64.a libmkl_pgi_thread.a > >>>> libmkl_vml_def.so > >>>> libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.so libmkl_pgi_thread.so > >>>> libmkl_vml_mc2.so > >>>> libmkl_blacs_openmpi_ilp64.so libmkl_gf_lp64.a libmkl_rt.so > >>>> libmkl_vml_mc3.so > >>>> libmkl_blacs_openmpi_lp64.a libmkl_gf_lp64.so > >>>> libmkl_scalapack_ilp64.a > >>>> libmkl_vml_mc.so > >>>> libmkl_blacs_openmpi_lp64.so libmkl_gnu_thread.a > >>>> libmkl_scalapack_ilp64.so > >>>> locale > >>>> libmkl_blacs_sgimpt_ilp64.a libmkl_gnu_thread.so > >>>> libmkl_scalapack_lp64.a > >>>> libmkl_blacs_sgimpt_ilp64.so libmkl_intel_ilp64.a > >>>> libmkl_scalapack_lp64.so > >>>> libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_sequential.a > >>>> > >>>> > >>>> Cheers, > >>>> Samuel > >>>> > >>>> On 04/24/2018 03:50 PM, Satish Balay wrote: > >>>>> Executing: cc -o /tmp/petsc-tyd_Hm/config.libraries/conftest -O > >>>>> /tmp/petsc-tyd_Hm/config.libraries/conftest.o > >>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 > >>>>> -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 > >>>>> -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl > >>>>> Possible ERROR while running linker: exit code 256 > >>>>> stderr: > >>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(mkl_semaphore.o): > >>>>> In function `mkl_serv_load_inspector': > >>>>> mkl_semaphore.c:(.text+0x123): warning: Using 'dlopen' in statically > >>>>> linked > >>>>> applications requires at runtime the shared libraries from the glibc > >>>>> version > >>>>> used for linking > >>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(dgetrs.o): > >>>>> In function `mkl_lapack_dgetrs': > >>>>> dgetrs_gen.f:(.text+0x224): undefined reference to `mkl_blas_dtrsm' > >>>>> dgetrs_gen.f:(.text+0x2b1): undefined reference to `mkl_blas_dtrsm' > >>>>> dgetrs_gen.f:(.text+0x2e7): undefined reference to `mkl_lapack_dlaswp' > >>>>> <<<<<<<< > >>>>> > >>>>> For some reason the compiler is picking up static version of MKL - > >>>>> instead > >>>>> of shared libraries. > >>>>> > >>>>> What do you have for: > >>>>> > >>>>> ls /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ > >>>>> > >>>>> Satish > >>>>> > >>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >>>>> > >>>>>> Dear Satish and Stefano, > >>>>>> > >>>>>> Thank you for your answers. I believe I had initially tried to use the > >>>>>> option > >>>>>> --with-blaslapack-dir=[...], instead of specifying lib and include > >>>>>> directly. > >>>>>> But that gives me an error message: > >>>>>> > >>>>>> ******************************************************************************* > >>>>>> UNABLE to CONFIGURE with GIVEN OPTIONS (see > >>>>>> configure.log > >>>>>> for > >>>>>> details): > >>>>>> ------------------------------------------------------------------------------- > >>>>>> You set a value for --with-blaslapack-dir=, but > >>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be used > >>>>>> ******************************************************************************* > >>>>>> > >>>>>> I attach the new configure.log. Do you think there is something wrong > >>>>>> with > >>>>>> the > >>>>>> mkl version that I'm trying to use? Or is it only related to petsc? > >>>>>> > >>>>>> Cheers, > >>>>>> Samuel > >>>>>> > >>>>>> > >>>>>> On 04/24/2018 02:55 PM, Satish Balay wrote: > >>>>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >>>>>>> > >>>>>>>> Hi there, > >>>>>>>> > >>>>>>>> I was wondering if you could help me with the correct configuration > >>>>>>>> of > >>>>>>>> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure > >>>>>>>> which > >>>>>>>> information would be useful to you, but basically the problem seems > >>>>>>>> to > >>>>>>>> be > >>>>>>>> in > >>>>>>>> correctly compiling it with intel compiler and the existing mkl > >>>>>>>> library. > >>>>>>>> > >>>>>>>> The pre-installed mkl version is > >>>>>>>> > >>>>>>>> intel/17.0.4.196 > >>>>>>>> > >>>>>>>> I have tried various things and finally, I got it at least to compile > >>>>>>>> with > >>>>>>>> the > >>>>>>>> following options (options chosen in reference to the mkl link > >>>>>>>> advisor...): > >>>>>>>> > >>>>>>>> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC > >>>>>>>> --with-debugging=0 > >>>>>>>> --with-scalar-type=complex --download-scalapack --download-mumps=yes > >>>>>>>> --download-superlu --with-batch --known-mpi-shared-libraries=1 > >>>>>>>> *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a > >>>>>>>> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group > >>>>>>>> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a > >>>>>>>> ${MKLROOT}/lib/intel64/libmkl_sequential.a > >>>>>>>> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm > >>>>>>>> -ldl" > >>>>>>>> *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" > >>>>>>> Can you remove the above 2 options [--with-blaslapack-lib, > >>>>>>> --with-blaslapack-include] and use: > >>>>>>> > >>>>>>> --with-blas-lapack-dir=${MKLROOT} > >>>>>>> > >>>>>>> And see if you still have this problem? > >>>>>>> > >>>>>>> Satish > >>>>>>> > >>>>>>>> --known-64-bit-blas-indices=0 > >>>>>>>> > >>>>>>>> After compilation, when trying to compile > >>>>>>>> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get > >>>>>>>> linking > >>>>>>>> errors (attached). Would anyone be able to help me out here? I really > >>>>>>>> don't > >>>>>>>> have a good understanding of this. > >>>>>>>> > >>>>>>>> I'm attaching the configuration.log file, as well as the linking > >>>>>>>> errors > >>>>>>>> when > >>>>>>>> trying to compile ex5f. > >>>>>>>> > >>>>>>>> Thank you very much in advance! > >>>>>>>> > >>>>>>>> Best regards, > >>>>>>>> Samuel > >>>>>>>> > >> > >> > > From rlmackie862 at gmail.com Tue Apr 24 12:19:34 2018 From: rlmackie862 at gmail.com (Randall Mackie) Date: Tue, 24 Apr 2018 10:19:34 -0700 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: <5094d7bb-f74c-8fee-b1c4-3b3ded0aaea1@gmail.com> References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> <7ad332fc-290d-c8f0-4a60-b0f4b7b02415@gmail.com> <5094d7bb-f74c-8fee-b1c4-3b3ded0aaea1@gmail.com> Message-ID: I haven?t followed this thread completely, but have you sourced the Intel mklvars.sh file? I do the following, for example, source ?/intel_2018/mkl/bin/mklvars.sh intel64 Then in our configure it?s simply: ?with-blas-lapack-dir=?/intel_2018/mkl and this works fine and picks up all the right Intel libraries. It could be set up differently on your system though. Randy M. > On Apr 24, 2018, at 9:23 AM, Samuel Lanthaler wrote: > > Dear Satish, > > As you suggested, the configure appears to work with > > ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 --with-scalar-type=complex --download-scalapack --download-mumps=yes --download-superlu --with-batch --known-mpi-shared-libraries=1 --with-blaslapack-dir=${MKLROOT} --known-64-bit-blas-indices=0 --LDFLAGS="-dynamic" > > However, after doing the reconfigure, I tried to compile using make, and ran into trouble with CLINKER: > > [...] > CC arch-complex/obj/sys/classes/draw/impls/x/xops.o > CLINKER arch-complex/lib/libpetsc.so.3.09.0 > Warning: > -dynamic was already seen on command line, overriding with -shared. > /usr/bin/ld: arch-complex/obj/sys/fsrc/somefort.o: relocation R_X86_64_32 against `petscfortran5_' can not be used when making a shared object; recompile with -fPIC > arch-complex/obj/sys/fsrc/somefort.o: error adding symbols: Bad value > [...] > **************************ERROR************************************* > Error during compile, check arch-complex/lib/petsc/conf/make.log > Send it and arch-complex/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov > ******************************************************************** > makefile:30: recipe for target 'all' failed > make: *** [all] Error 1 > > Any ideas? Again, I'm attaching the .log files. > > > > On 04/24/2018 04:14 PM, Satish Balay wrote: >> Sorry I should have clarified - its a configure option. >> >> Satish >> >> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >> >>> Sorry, I'm confused: Can I add this somehow when doing the ./configure? Or do >>> you mean to add it when compiling ex5f with the configuration I had initially? >>> >>> Cheers, >>> Samuel >>> >>> >>> On 04/24/2018 04:03 PM, Satish Balay wrote: >>>> Ok - I do see: libmkl_intel_lp64.so libmkl_sequential.so libmkl_core.so - >>>> these should >>>> be the ones that get picked up. >>>> >>>> I do not know why 'cc' is picking up static libraries instead of the >>>> shared ones. Perhaps it needs an extra option >>>> >>>> 'LDFLAGS=-dynamic' >>>> >>>> Satish >>>> >>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>>> >>>>> Dear Satish, >>>>> >>>>> I get the following when doing the ls: >>>>> >>>>> [lanthale at daint103 mkl]$ ls >>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ >>>>> libmkl_ao_worker.so libmkl_blacs_sgimpt_lp64.so >>>>> libmkl_intel_lp64.a libmkl_sequential.so >>>>> libmkl_avx2.so libmkl_blas95_ilp64.a libmkl_intel_lp64.so >>>>> libmkl_tbb_thread.a >>>>> libmkl_avx512_mic.so libmkl_blas95_lp64.a libmkl_intel_thread.a >>>>> libmkl_tbb_thread.so >>>>> libmkl_avx512.so libmkl_cdft_core.a libmkl_intel_thread.so >>>>> libmkl_vml_avx2.so >>>>> libmkl_avx.so libmkl_cdft_core.so libmkl_lapack95_ilp64.a >>>>> libmkl_vml_avx512_mic.so >>>>> libmkl_blacs_intelmpi_ilp64.a libmkl_core.a libmkl_lapack95_lp64.a >>>>> libmkl_vml_avx512.so >>>>> libmkl_blacs_intelmpi_ilp64.so libmkl_core.so libmkl_mc3.so >>>>> libmkl_vml_avx.so >>>>> libmkl_blacs_intelmpi_lp64.a libmkl_def.so libmkl_mc.so >>>>> libmkl_vml_cmpt.so >>>>> libmkl_blacs_intelmpi_lp64.so libmkl_gf_ilp64.a libmkl_pgi_thread.a >>>>> libmkl_vml_def.so >>>>> libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.so libmkl_pgi_thread.so >>>>> libmkl_vml_mc2.so >>>>> libmkl_blacs_openmpi_ilp64.so libmkl_gf_lp64.a libmkl_rt.so >>>>> libmkl_vml_mc3.so >>>>> libmkl_blacs_openmpi_lp64.a libmkl_gf_lp64.so libmkl_scalapack_ilp64.a >>>>> libmkl_vml_mc.so >>>>> libmkl_blacs_openmpi_lp64.so libmkl_gnu_thread.a >>>>> libmkl_scalapack_ilp64.so >>>>> locale >>>>> libmkl_blacs_sgimpt_ilp64.a libmkl_gnu_thread.so >>>>> libmkl_scalapack_lp64.a >>>>> libmkl_blacs_sgimpt_ilp64.so libmkl_intel_ilp64.a >>>>> libmkl_scalapack_lp64.so >>>>> libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_sequential.a >>>>> >>>>> >>>>> Cheers, >>>>> Samuel >>>>> >>>>> On 04/24/2018 03:50 PM, Satish Balay wrote: >>>>>> Executing: cc -o /tmp/petsc-tyd_Hm/config.libraries/conftest -O >>>>>> /tmp/petsc-tyd_Hm/config.libraries/conftest.o >>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 >>>>>> -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 >>>>>> -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl >>>>>> Possible ERROR while running linker: exit code 256 >>>>>> stderr: >>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(mkl_semaphore.o): >>>>>> In function `mkl_serv_load_inspector': >>>>>> mkl_semaphore.c:(.text+0x123): warning: Using 'dlopen' in statically >>>>>> linked >>>>>> applications requires at runtime the shared libraries from the glibc >>>>>> version >>>>>> used for linking >>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(dgetrs.o): >>>>>> In function `mkl_lapack_dgetrs': >>>>>> dgetrs_gen.f:(.text+0x224): undefined reference to `mkl_blas_dtrsm' >>>>>> dgetrs_gen.f:(.text+0x2b1): undefined reference to `mkl_blas_dtrsm' >>>>>> dgetrs_gen.f:(.text+0x2e7): undefined reference to `mkl_lapack_dlaswp' >>>>>> <<<<<<<< >>>>>> >>>>>> For some reason the compiler is picking up static version of MKL - instead >>>>>> of shared libraries. >>>>>> >>>>>> What do you have for: >>>>>> >>>>>> ls /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ >>>>>> >>>>>> Satish >>>>>> >>>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>>>>> >>>>>>> Dear Satish and Stefano, >>>>>>> >>>>>>> Thank you for your answers. I believe I had initially tried to use the >>>>>>> option >>>>>>> --with-blaslapack-dir=[...], instead of specifying lib and include >>>>>>> directly. >>>>>>> But that gives me an error message: >>>>>>> >>>>>>> ******************************************************************************* >>>>>>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >>>>>>> for >>>>>>> details): >>>>>>> ------------------------------------------------------------------------------- >>>>>>> You set a value for --with-blaslapack-dir=, but >>>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be used >>>>>>> ******************************************************************************* >>>>>>> >>>>>>> I attach the new configure.log. Do you think there is something wrong >>>>>>> with >>>>>>> the >>>>>>> mkl version that I'm trying to use? Or is it only related to petsc? >>>>>>> >>>>>>> Cheers, >>>>>>> Samuel >>>>>>> >>>>>>> >>>>>>> On 04/24/2018 02:55 PM, Satish Balay wrote: >>>>>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>>>>>>> >>>>>>>>> Hi there, >>>>>>>>> >>>>>>>>> I was wondering if you could help me with the correct configuration of >>>>>>>>> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure >>>>>>>>> which >>>>>>>>> information would be useful to you, but basically the problem seems to >>>>>>>>> be >>>>>>>>> in >>>>>>>>> correctly compiling it with intel compiler and the existing mkl >>>>>>>>> library. >>>>>>>>> >>>>>>>>> The pre-installed mkl version is >>>>>>>>> >>>>>>>>> intel/17.0.4.196 >>>>>>>>> >>>>>>>>> I have tried various things and finally, I got it at least to compile >>>>>>>>> with >>>>>>>>> the >>>>>>>>> following options (options chosen in reference to the mkl link >>>>>>>>> advisor...): >>>>>>>>> >>>>>>>>> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 >>>>>>>>> --with-scalar-type=complex --download-scalapack --download-mumps=yes >>>>>>>>> --download-superlu --with-batch --known-mpi-shared-libraries=1 >>>>>>>>> *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a >>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group >>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a >>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_sequential.a >>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm >>>>>>>>> -ldl" >>>>>>>>> *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" >>>>>>>> Can you remove the above 2 options [--with-blaslapack-lib, >>>>>>>> --with-blaslapack-include] and use: >>>>>>>> >>>>>>>> --with-blas-lapack-dir=${MKLROOT} >>>>>>>> >>>>>>>> And see if you still have this problem? >>>>>>>> >>>>>>>> Satish >>>>>>>> >>>>>>>>> --known-64-bit-blas-indices=0 >>>>>>>>> >>>>>>>>> After compilation, when trying to compile >>>>>>>>> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get >>>>>>>>> linking >>>>>>>>> errors (attached). Would anyone be able to help me out here? I really >>>>>>>>> don't >>>>>>>>> have a good understanding of this. >>>>>>>>> >>>>>>>>> I'm attaching the configuration.log file, as well as the linking errors >>>>>>>>> when >>>>>>>>> trying to compile ex5f. >>>>>>>>> >>>>>>>>> Thank you very much in advance! >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> Samuel >>>>>>>>> >>> >>> > > From danyang.su at gmail.com Tue Apr 24 22:57:26 2018 From: danyang.su at gmail.com (Danyang Su) Date: Tue, 24 Apr 2018 20:57:26 -0700 Subject: [petsc-users] DMSetLabelValue takes a lot of time for large domain Message-ID: <3ee3963e-3fbe-9440-6754-861ef9271ae8@gmail.com> Hi All, I use DMPlex in unstructured grid code and recently found DMSetLabelValue takes a lot of time for large problem, e.g., num. of cells > 1 million. In my code, I use DMPlexCreateFromCellList () Loop over all cells/nodes{ DMSetLabelValue } DMPlexDistribute The code works fine except DMSetLabelValue takes a lot of time for large problem. I use DMSetLabelValue to set material id for all the nodes or cells so that each subdomain has a copy of material id. Is there any other functions that can be used more efficient, e.g. set labels by array, not 1 by 1? Thanks, Danyang From Savneet.KAUR at cea.fr Wed Apr 25 04:10:40 2018 From: Savneet.KAUR at cea.fr (Savneet Kaur) Date: Wed, 25 Apr 2018 11:10:40 +0200 Subject: [petsc-users] SLEPC matrix Query Message-ID: <2aca64f1-9c74-cf44-2960-d8d35ced3882@cea.fr> Hello, Warm Regards I am Savneet Kaur, a master student at University Paris Saclay and currently pursuing an internship at CEA Saclay (France). I have recently started to understand the slepc and petsc solvers, by taking up the tutorials for eigenvalue problems. In my internship work I have to develop a laplacian matrix from a given transition rate matrix and solve it using SLEPC and PETSC and to evaluate the lowest eigenvalue. I was wondering if I could get some information. I need to diagonalize a 2D Laplacian matrix. And writing a code in C and diagonalizing it is easy. But I am not getting it how to accommodate with the SLEPC Program. Am i suppose to use any packages or it will is done by a loop? Or I need to read some other manual to understand how does it work. Please kindly let me know. The diagonalizing a matrix will be preliminary step of my work. I will be highly obliged to the the team, if I could get help. Hoping for a favorable response from your side. Thank you for the time and consideration. Sincerely, *Savneet Kaur* *Intern at DEN/DANS/DMN/SRMP* *CEA - Centre de Saclay ??B?timent 520 * *91191 Gif-sur-Yvette Cedex* *France* *Tel: +33 (0) 666 749 000* *Email: savneet.kaur at cea.fr * -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 25 04:30:03 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 25 Apr 2018 05:30:03 -0400 Subject: [petsc-users] DMSetLabelValue takes a lot of time for large domain In-Reply-To: <3ee3963e-3fbe-9440-6754-861ef9271ae8@gmail.com> References: <3ee3963e-3fbe-9440-6754-861ef9271ae8@gmail.com> Message-ID: On Tue, Apr 24, 2018 at 11:57 PM, Danyang Su wrote: > Hi All, > > I use DMPlex in unstructured grid code and recently found DMSetLabelValue > takes a lot of time for large problem, e.g., num. of cells > 1 million. In > my code, I use > > DMPlexCreateFromCellList () > > Loop over all cells/nodes{ > > DMSetLabelValue > > } > > DMPlexDistribute > > The code works fine except DMSetLabelValue takes a lot of time for large > problem. I use DMSetLabelValue to set material id for all the nodes or > cells so that each subdomain has a copy of material id. Is there any other > functions that can be used more efficient, e.g. set labels by array, not 1 > by 1? > 1) I just want to make sure that you are running an optimized build, configured using --with-debugging=0 2) How long does it take? 3) Do you have intervening queries to the label between insertions? It should be fast, and I therefore suspect that logging code might be responsible for the performance. I will setup a test. Can you give me an idea roughly a) How many SetValue() calls you make b) How many different label values you use Thanks, Matt > Thanks, > > Danyang > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 25 04:33:03 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 25 Apr 2018 05:33:03 -0400 Subject: [petsc-users] SLEPC matrix Query In-Reply-To: <2aca64f1-9c74-cf44-2960-d8d35ced3882@cea.fr> References: <2aca64f1-9c74-cf44-2960-d8d35ced3882@cea.fr> Message-ID: On Wed, Apr 25, 2018 at 5:10 AM, Savneet Kaur wrote: > Hello, > > Warm Regards > > I am Savneet Kaur, a master student at University Paris Saclay and > currently pursuing an internship at CEA Saclay (France). > > I have recently started to understand the slepc and petsc solvers, by > taking up the tutorials for eigenvalue problems. In my internship work I > have to develop a laplacian matrix from a given transition rate matrix and > solve it using SLEPC and PETSC and to evaluate the lowest eigenvalue. > > I was wondering if I could get some information. I need to diagonalize a > 2D Laplacian matrix. And writing a code in C and diagonalizing it is easy. > But I am not getting it how to accommodate with the SLEPC Program. Am i > suppose to use any packages or it will is done by a loop? Or I need to read > some other manual to understand how does it work. Please kindly let me know. > I would just like to clarify some terminology. "Diagonalize" would usually mean find all eigenvalues and eigenvectors. Is this what you mean? Often, SLEPc users want only a portion of the spectrum since the matrices are enormous. Thanks, Matt > The diagonalizing a matrix will be preliminary step of my work. > > I will be highly obliged to the the team, if I could get help. > > Hoping for a favorable response from your side. > > Thank you for the time and consideration. > > > Sincerely, > > *Savneet Kaur* > > *Intern at DEN/DANS/DMN/SRMP* > > > *CEA - Centre de Saclay ? B?timent 520 * > > *91191 Gif-sur-Yvette Cedex* > > *France* > > *Tel: +33 (0) 666 749 000* > > > *Email: savneet.kaur at cea.fr * > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 25 05:16:16 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 25 Apr 2018 06:16:16 -0400 Subject: [petsc-users] DMSetLabelValue takes a lot of time for large domain In-Reply-To: <3ee3963e-3fbe-9440-6754-861ef9271ae8@gmail.com> References: <3ee3963e-3fbe-9440-6754-861ef9271ae8@gmail.com> Message-ID: On Tue, Apr 24, 2018 at 11:57 PM, Danyang Su wrote: > Hi All, > > I use DMPlex in unstructured grid code and recently found DMSetLabelValue > takes a lot of time for large problem, e.g., num. of cells > 1 million. In > my code, I use > I read your code wrong. For large loop, you should not use the convenience function. You should use > DMPlexCreateFromCellList () DMGetLabel(dm, name, &label) > > Loop over all cells/nodes{ > > DMSetLabelValue > Replace this by DMLabelSetValue(label, point, val) > } > > DMPlexDistribute > > The code works fine except DMSetLabelValue takes a lot of time for large > problem. I use DMSetLabelValue to set material id for all the nodes or > cells so that each subdomain has a copy of material id. Is there any other > functions that can be used more efficient, e.g. set labels by array, not 1 > by 1? > That should take much less time. Thanks, Matt > Thanks, > > Danyang > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Savneet.KAUR at cea.fr Wed Apr 25 07:10:57 2018 From: Savneet.KAUR at cea.fr (Savneet Kaur) Date: Wed, 25 Apr 2018 14:10:57 +0200 Subject: [petsc-users] SLEPC matrix Query In-Reply-To: References: <2aca64f1-9c74-cf44-2960-d8d35ced3882@cea.fr> Message-ID: Hello Again, Yes I would like to find for the initial 10% and 20% of the eigenpairs. But in addition to this,I also want to check for the full spectrum. So, how shall I proceed with it? Thank You. Regards, savneet Le 25/04/2018 ? 11:33, Matthew Knepley a ?crit?: > On Wed, Apr 25, 2018 at 5:10 AM, Savneet Kaur > wrote: > > Hello, > > Warm Regards > > I am Savneet Kaur, a master student at University Paris Saclay and > currently pursuing an internship at CEA Saclay (France). > > I have recently started to understand the slepc and petsc solvers, > by taking up the tutorials for eigenvalue problems. In my > internship work I have to develop a laplacian matrix from a given > transition rate matrix and solve it using SLEPC and PETSC and to > evaluate the lowest eigenvalue. > > I was wondering if I could get some information. I need to > diagonalize a 2D Laplacian matrix. And writing a code in C and > diagonalizing it is easy. But I am not getting it how to > accommodate with the SLEPC Program. Am i suppose to use any > packages or it will is done by a loop? Or I need to read some > other manual to understand how does it work. Please kindly let me > know. > > I would just like to clarify some terminology. "Diagonalize" would > usually mean find all eigenvalues and eigenvectors. > Is this what you mean? Often, SLEPc users want only a portion of the > spectrum since the matrices are enormous. > > ? Thanks, > > ? ? ?Matt > > The diagonalizing a matrix will be preliminary step of my work. > > I will be highly obliged to the the team, if I could get help. > > Hoping for a favorable response from your side. > > Thank you for the time and consideration. > > > Sincerely, > > *Savneet Kaur* > > *Intern at DEN/DANS/DMN/SRMP* > > *CEA - Centre de Saclay ??B?timent 520 > * > > *91191 Gif-sur-Yvette Cedex* > > *France* > > *Tel: +33 (0) 666 749 000* > > *Email: savneet.kaur at cea.fr > * > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 25 07:13:23 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 25 Apr 2018 08:13:23 -0400 Subject: [petsc-users] SLEPC matrix Query In-Reply-To: References: <2aca64f1-9c74-cf44-2960-d8d35ced3882@cea.fr> Message-ID: On Wed, Apr 25, 2018 at 8:10 AM, Savneet Kaur wrote: > Hello Again, > > Yes I would like to find for the initial 10% and 20% of the eigenpairs. > But in addition to this,I also want to check for the full spectrum. > > So, how shall I proceed with it? > First just setup the eigensolve. I don't think anything you are asking for is missing from the standard interface. Checking the full spectrum is only possible with small matrices. SLEPc would just interface to LAPACK for that. Thanks, Matt Thank You. > > Regards, > savneet > > Le 25/04/2018 ? 11:33, Matthew Knepley a ?crit : > > On Wed, Apr 25, 2018 at 5:10 AM, Savneet Kaur wrote: > >> Hello, >> >> Warm Regards >> >> I am Savneet Kaur, a master student at University Paris Saclay and >> currently pursuing an internship at CEA Saclay (France). >> >> I have recently started to understand the slepc and petsc solvers, by >> taking up the tutorials for eigenvalue problems. In my internship work I >> have to develop a laplacian matrix from a given transition rate matrix and >> solve it using SLEPC and PETSC and to evaluate the lowest eigenvalue. >> >> I was wondering if I could get some information. I need to diagonalize a >> 2D Laplacian matrix. And writing a code in C and diagonalizing it is easy. >> But I am not getting it how to accommodate with the SLEPC Program. Am i >> suppose to use any packages or it will is done by a loop? Or I need to read >> some other manual to understand how does it work. Please kindly let me know. >> > I would just like to clarify some terminology. "Diagonalize" would usually > mean find all eigenvalues and eigenvectors. > Is this what you mean? Often, SLEPc users want only a portion of the > spectrum since the matrices are enormous. > > Thanks, > > Matt > >> The diagonalizing a matrix will be preliminary step of my work. >> >> I will be highly obliged to the the team, if I could get help. >> >> Hoping for a favorable response from your side. >> >> Thank you for the time and consideration. >> >> >> Sincerely, >> >> *Savneet Kaur* >> >> *Intern at DEN/DANS/DMN/SRMP* >> >> >> *CEA - Centre de Saclay ? B?timent 520 * >> >> *91191 Gif-sur-Yvette Cedex* >> >> *France* >> >> *Tel: +33 (0) 666 749 000* >> >> >> *Email: savneet.kaur at cea.fr * >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.lanthaler at gmail.com Wed Apr 25 11:17:14 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Wed, 25 Apr 2018 18:17:14 +0200 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> <7ad332fc-290d-c8f0-4a60-b0f4b7b02415@gmail.com> <5094d7bb-f74c-8fee-b1c4-3b3ded0aaea1@gmail.com> Message-ID: Dear Satish, Your workaround worked! Great! Thanks a lot for all your help, everyone. =) Cheers, Samuel On 04/24/2018 06:33 PM, Satish Balay wrote: > Hm - I'm not sure why configure didn't try and set -fPIC compiler options. > > Workarround is: > > CFLAGS=-fPIC FFLAGS=-fPICC CXXFLAGS=-fPIC > > or alternatively - you can disable sharedlibraries > > --with-shared-libraries=0 > > Satish > > On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >> Dear Satish, >> >> As you suggested, the configure appears to work with >> >> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 >> --with-scalar-type=complex --download-scalapack --download-mumps=yes >> --download-superlu --with-batch --known-mpi-shared-libraries=1 >> --with-blaslapack-dir=${MKLROOT} --known-64-bit-blas-indices=0 >> --LDFLAGS="-dynamic" >> >> However, after doing the reconfigure, I tried to compile using make, and ran >> into trouble with CLINKER: >> >> [...] >> CC arch-complex/obj/sys/classes/draw/impls/x/xops.o >> CLINKER arch-complex/lib/libpetsc.so.3.09.0 >> Warning: >> -dynamic was already seen on command line, overriding with -shared. >> /usr/bin/ld: arch-complex/obj/sys/fsrc/somefort.o: relocation R_X86_64_32 >> against `petscfortran5_' can not be used when making a shared object; >> recompile with -fPIC >> arch-complex/obj/sys/fsrc/somefort.o: error adding symbols: Bad value >> [...] >> **************************ERROR************************************* >> Error during compile, check arch-complex/lib/petsc/conf/make.log >> Send it and arch-complex/lib/petsc/conf/configure.log to >> petsc-maint at mcs.anl.gov >> ******************************************************************** >> makefile:30: recipe for target 'all' failed >> make: *** [all] Error 1 >> >> Any ideas? Again, I'm attaching the .log files. >> >> >> >> On 04/24/2018 04:14 PM, Satish Balay wrote: >>> Sorry I should have clarified - its a configure option. >>> >>> Satish >>> >>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>> >>>> Sorry, I'm confused: Can I add this somehow when doing the ./configure? Or >>>> do >>>> you mean to add it when compiling ex5f with the configuration I had >>>> initially? >>>> >>>> Cheers, >>>> Samuel >>>> >>>> >>>> On 04/24/2018 04:03 PM, Satish Balay wrote: >>>>> Ok - I do see: libmkl_intel_lp64.so libmkl_sequential.so libmkl_core.so - >>>>> these should >>>>> be the ones that get picked up. >>>>> >>>>> I do not know why 'cc' is picking up static libraries instead of the >>>>> shared ones. Perhaps it needs an extra option >>>>> >>>>> 'LDFLAGS=-dynamic' >>>>> >>>>> Satish >>>>> >>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>>>> >>>>>> Dear Satish, >>>>>> >>>>>> I get the following when doing the ls: >>>>>> >>>>>> [lanthale at daint103 mkl]$ ls >>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ >>>>>> libmkl_ao_worker.so libmkl_blacs_sgimpt_lp64.so >>>>>> libmkl_intel_lp64.a libmkl_sequential.so >>>>>> libmkl_avx2.so libmkl_blas95_ilp64.a >>>>>> libmkl_intel_lp64.so >>>>>> libmkl_tbb_thread.a >>>>>> libmkl_avx512_mic.so libmkl_blas95_lp64.a >>>>>> libmkl_intel_thread.a >>>>>> libmkl_tbb_thread.so >>>>>> libmkl_avx512.so libmkl_cdft_core.a libmkl_intel_thread.so >>>>>> libmkl_vml_avx2.so >>>>>> libmkl_avx.so libmkl_cdft_core.so >>>>>> libmkl_lapack95_ilp64.a >>>>>> libmkl_vml_avx512_mic.so >>>>>> libmkl_blacs_intelmpi_ilp64.a libmkl_core.a libmkl_lapack95_lp64.a >>>>>> libmkl_vml_avx512.so >>>>>> libmkl_blacs_intelmpi_ilp64.so libmkl_core.so libmkl_mc3.so >>>>>> libmkl_vml_avx.so >>>>>> libmkl_blacs_intelmpi_lp64.a libmkl_def.so libmkl_mc.so >>>>>> libmkl_vml_cmpt.so >>>>>> libmkl_blacs_intelmpi_lp64.so libmkl_gf_ilp64.a libmkl_pgi_thread.a >>>>>> libmkl_vml_def.so >>>>>> libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.so libmkl_pgi_thread.so >>>>>> libmkl_vml_mc2.so >>>>>> libmkl_blacs_openmpi_ilp64.so libmkl_gf_lp64.a libmkl_rt.so >>>>>> libmkl_vml_mc3.so >>>>>> libmkl_blacs_openmpi_lp64.a libmkl_gf_lp64.so >>>>>> libmkl_scalapack_ilp64.a >>>>>> libmkl_vml_mc.so >>>>>> libmkl_blacs_openmpi_lp64.so libmkl_gnu_thread.a >>>>>> libmkl_scalapack_ilp64.so >>>>>> locale >>>>>> libmkl_blacs_sgimpt_ilp64.a libmkl_gnu_thread.so >>>>>> libmkl_scalapack_lp64.a >>>>>> libmkl_blacs_sgimpt_ilp64.so libmkl_intel_ilp64.a >>>>>> libmkl_scalapack_lp64.so >>>>>> libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_sequential.a >>>>>> >>>>>> >>>>>> Cheers, >>>>>> Samuel >>>>>> >>>>>> On 04/24/2018 03:50 PM, Satish Balay wrote: >>>>>>> Executing: cc -o /tmp/petsc-tyd_Hm/config.libraries/conftest -O >>>>>>> /tmp/petsc-tyd_Hm/config.libraries/conftest.o >>>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 >>>>>>> -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 >>>>>>> -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl >>>>>>> Possible ERROR while running linker: exit code 256 >>>>>>> stderr: >>>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(mkl_semaphore.o): >>>>>>> In function `mkl_serv_load_inspector': >>>>>>> mkl_semaphore.c:(.text+0x123): warning: Using 'dlopen' in statically >>>>>>> linked >>>>>>> applications requires at runtime the shared libraries from the glibc >>>>>>> version >>>>>>> used for linking >>>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(dgetrs.o): >>>>>>> In function `mkl_lapack_dgetrs': >>>>>>> dgetrs_gen.f:(.text+0x224): undefined reference to `mkl_blas_dtrsm' >>>>>>> dgetrs_gen.f:(.text+0x2b1): undefined reference to `mkl_blas_dtrsm' >>>>>>> dgetrs_gen.f:(.text+0x2e7): undefined reference to `mkl_lapack_dlaswp' >>>>>>> <<<<<<<< >>>>>>> >>>>>>> For some reason the compiler is picking up static version of MKL - >>>>>>> instead >>>>>>> of shared libraries. >>>>>>> >>>>>>> What do you have for: >>>>>>> >>>>>>> ls /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>>>>>> >>>>>>>> Dear Satish and Stefano, >>>>>>>> >>>>>>>> Thank you for your answers. I believe I had initially tried to use the >>>>>>>> option >>>>>>>> --with-blaslapack-dir=[...], instead of specifying lib and include >>>>>>>> directly. >>>>>>>> But that gives me an error message: >>>>>>>> >>>>>>>> ******************************************************************************* >>>>>>>> UNABLE to CONFIGURE with GIVEN OPTIONS (see >>>>>>>> configure.log >>>>>>>> for >>>>>>>> details): >>>>>>>> ------------------------------------------------------------------------------- >>>>>>>> You set a value for --with-blaslapack-dir=, but >>>>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be used >>>>>>>> ******************************************************************************* >>>>>>>> >>>>>>>> I attach the new configure.log. Do you think there is something wrong >>>>>>>> with >>>>>>>> the >>>>>>>> mkl version that I'm trying to use? Or is it only related to petsc? >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Samuel >>>>>>>> >>>>>>>> >>>>>>>> On 04/24/2018 02:55 PM, Satish Balay wrote: >>>>>>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: >>>>>>>>> >>>>>>>>>> Hi there, >>>>>>>>>> >>>>>>>>>> I was wondering if you could help me with the correct configuration >>>>>>>>>> of >>>>>>>>>> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure >>>>>>>>>> which >>>>>>>>>> information would be useful to you, but basically the problem seems >>>>>>>>>> to >>>>>>>>>> be >>>>>>>>>> in >>>>>>>>>> correctly compiling it with intel compiler and the existing mkl >>>>>>>>>> library. >>>>>>>>>> >>>>>>>>>> The pre-installed mkl version is >>>>>>>>>> >>>>>>>>>> intel/17.0.4.196 >>>>>>>>>> >>>>>>>>>> I have tried various things and finally, I got it at least to compile >>>>>>>>>> with >>>>>>>>>> the >>>>>>>>>> following options (options chosen in reference to the mkl link >>>>>>>>>> advisor...): >>>>>>>>>> >>>>>>>>>> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC >>>>>>>>>> --with-debugging=0 >>>>>>>>>> --with-scalar-type=complex --download-scalapack --download-mumps=yes >>>>>>>>>> --download-superlu --with-batch --known-mpi-shared-libraries=1 >>>>>>>>>> *--with-blaslapack-lib=*" ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a >>>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group >>>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a >>>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_sequential.a >>>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm >>>>>>>>>> -ldl" >>>>>>>>>> *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" >>>>>>>>> Can you remove the above 2 options [--with-blaslapack-lib, >>>>>>>>> --with-blaslapack-include] and use: >>>>>>>>> >>>>>>>>> --with-blas-lapack-dir=${MKLROOT} >>>>>>>>> >>>>>>>>> And see if you still have this problem? >>>>>>>>> >>>>>>>>> Satish >>>>>>>>> >>>>>>>>>> --known-64-bit-blas-indices=0 >>>>>>>>>> >>>>>>>>>> After compilation, when trying to compile >>>>>>>>>> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get >>>>>>>>>> linking >>>>>>>>>> errors (attached). Would anyone be able to help me out here? I really >>>>>>>>>> don't >>>>>>>>>> have a good understanding of this. >>>>>>>>>> >>>>>>>>>> I'm attaching the configuration.log file, as well as the linking >>>>>>>>>> errors >>>>>>>>>> when >>>>>>>>>> trying to compile ex5f. >>>>>>>>>> >>>>>>>>>> Thank you very much in advance! >>>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> Samuel >>>>>>>>>> >>>> >> From danyang.su at gmail.com Wed Apr 25 11:40:28 2018 From: danyang.su at gmail.com (Danyang Su) Date: Wed, 25 Apr 2018 09:40:28 -0700 Subject: [petsc-users] DMSetLabelValue takes a lot of time for large domain In-Reply-To: References: <3ee3963e-3fbe-9440-6754-861ef9271ae8@gmail.com> Message-ID: Hi Matthew, In the worst case, every node/cell may have different label. Below is one of the worst scenario with 102299 nodes and 102299 different labels for test. I found the time cost increase during the loop. The first 9300 loop takes least time (<0.5) while the last 9300 loops takes much more time (>7.7), as shown below. If I use larger mesh with >1 million nodes, it runs very very slowly in this part. The PETSc is configured with optimization on. Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-scalapack --download-parmetis --download-metis --download-ptscotch --download-fblaslapack --download-hypre --download-superlu_dist --download-hdf5=yes --download-ctetgen --with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" istart iend progress CPU_Time time cost - old (sec) time cost - new (sec) 0 9299 0 1524670045.51166 9300 18599 0.100010753 1524670045.99605 0.4843890667 0.497246027 18600 27899 0.200010747 1524670047.32635 1.330302 1.3820912838 27900 37199 0.300010741 1524670049.3066 1.9802515507 2.2439446449 37200 46499 0.400010765 1524670052.1594 2.852804184 3.0739262104 46500 55799 0.500010729 1524670055.90961 3.7502081394 3.9270553589 55800 65099 0.600010753 1524670060.47654 4.5669286251 4.7571902275 65100 74399 0.700010777 1524670066.0941 5.6175630093 5.7428796291 74400 83699 0.800010741 1524670072.53886 6.44475317 6.5761549473 83700 92998 0.900010765 1524670079.99072 7.4518604279 7.4606924057 92999 102298 1 1524670087.71066 7.7199423313 8.2424075603 old code ??????? do ipoint = 0, istart-1 ????????? !c output time cost, use 1 processor to test ????????? if (b_enable_output .and. rank == 0) then ??????????? if (mod(ipoint,iprogress) == 0 .or. ipoint == istart-1) then ????????????? !write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart ????????????? write(*,*) ipoint, (ipoint+1.0)/istart,"time",MPI_Wtime() ??????????? end if ????????? end if ????????? call DMSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint,???????? & ?????????????????????????????? ipoint+1,ierr) ????????? CHKERRQ(ierr) ??????? end do new code ??????? call DMCreateLabel(dmda_flow%da,'cid_lg2g',ierr) ??????? CHKERRQ(ierr) ??????? call DMGetLabel(dmda_flow%da,'cid_lg2g',label, ierr) ??????? CHKERRQ(ierr) ??????? do ipoint = 0, istart-1 ????????? !c output time cost, use 1 processor to test ????????? if (b_enable_output .and. rank == 0) then ??????????? if (mod(ipoint,iprogress) == 0 .or. ipoint == istart-1) then ????????????? !write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart ????????????? write(*,*) ipoint, (ipoint+1.0)/istart,"time",MPI_Wtime() ??????????? end if ????????? end if ????????? call DMLabelSetValue(label,ipoint,ipoint+1,ierr) ????????? CHKERRQ(ierr) ??????? end do Thanks, Danyang On 2018-04-25 03:16 AM, Matthew Knepley wrote: > On Tue, Apr 24, 2018 at 11:57 PM, Danyang Su > wrote: > > Hi All, > > I use DMPlex in unstructured grid code and recently found > DMSetLabelValue takes a lot of time for large problem, e.g., num. > of cells > 1 million. In my code, I use > > > I read your code wrong. For large loop, you should not use the > convenience function. You should use > > DMPlexCreateFromCellList () > > > DMGetLabel(dm, name, &label) > > > Loop over all cells/nodes{ > > DMSetLabelValue > > > Replace this by DMLabelSetValue(label, point, val) > > } > > DMPlexDistribute > > The code works fine except DMSetLabelValue takes a lot of time for > large problem. I use DMSetLabelValue to set material id for all > the nodes or cells so that each subdomain has a copy of material > id. Is there any other functions that can be used more efficient, > e.g. set labels by array, not 1 by 1? > > > That should take much less time. > > ? Thanks, > > ? ? ?Matt > > Thanks, > > Danyang > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 25 11:47:40 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 25 Apr 2018 12:47:40 -0400 Subject: [petsc-users] DMSetLabelValue takes a lot of time for large domain In-Reply-To: References: <3ee3963e-3fbe-9440-6754-861ef9271ae8@gmail.com> Message-ID: On Wed, Apr 25, 2018 at 12:40 PM, Danyang Su wrote: > Hi Matthew, > > In the worst case, every node/cell may have different label. > > Do not use Label for this. Its not an appropriate thing. If every cell is different, just use the cell number. Labels are for mapping a relatively small number of keys (like material IDs) to sets of points (cells, vertices, etc.) Its not a great data structure for a permutation. However, I still do not believe these numbers. The old code does a string comparison every time. I will setup a test. Matt > Below is one of the worst scenario with 102299 nodes and 102299 different > labels for test. I found the time cost increase during the loop. The first > 9300 loop takes least time (<0.5) while the last 9300 loops takes much more > time (>7.7), as shown below. If I use larger mesh with >1 million nodes, it > runs very very slowly in this part. The PETSc is configured with > optimization on. > > Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > --download-mpich --download-scalapack --download-parmetis --download-metis > --download-ptscotch --download-fblaslapack --download-hypre > --download-superlu_dist --download-hdf5=yes --download-ctetgen > --with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" > CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native > -mtune=native" > > istart iend progress CPU_Time time cost - old (sec) time cost - new (sec) > 0 9299 0 1524670045.51166 > > 9300 18599 0.100010753 1524670045.99605 0.4843890667 0.497246027 > 18600 27899 0.200010747 1524670047.32635 1.330302 1.3820912838 > 27900 37199 0.300010741 1524670049.3066 1.9802515507 2.2439446449 > 37200 46499 0.400010765 1524670052.1594 2.852804184 3.0739262104 > 46500 55799 0.500010729 1524670055.90961 3.7502081394 3.9270553589 > 55800 65099 0.600010753 1524670060.47654 4.5669286251 4.7571902275 > 65100 74399 0.700010777 1524670066.0941 5.6175630093 5.7428796291 > 74400 83699 0.800010741 1524670072.53886 6.44475317 6.5761549473 > 83700 92998 0.900010765 1524670079.99072 7.4518604279 7.4606924057 > 92999 102298 1 1524670087.71066 7.7199423313 8.2424075603 > > old code > > do ipoint = 0, istart-1 > !c output time cost, use 1 processor to test > if (b_enable_output .and. rank == 0) then > if (mod(ipoint,iprogress) == 0 .or. ipoint == istart-1) then > !write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart > write(*,*) ipoint, (ipoint+1.0)/istart,"time",MPI_Wtime() > end if > end if > > call DMSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, & > ipoint+1,ierr) > CHKERRQ(ierr) > end do > > > new code > > call DMCreateLabel(dmda_flow%da,'cid_lg2g',ierr) > CHKERRQ(ierr) > > call DMGetLabel(dmda_flow%da,'cid_lg2g',label, ierr) > CHKERRQ(ierr) > > do ipoint = 0, istart-1 > !c output time cost, use 1 processor to test > if (b_enable_output .and. rank == 0) then > if (mod(ipoint,iprogress) == 0 .or. ipoint == istart-1) then > !write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart > write(*,*) ipoint, (ipoint+1.0)/istart,"time",MPI_Wtime() > end if > end if > > call DMLabelSetValue(label,ipoint,ipoint+1,ierr) > CHKERRQ(ierr) > end do > > Thanks, > > Danyang > > On 2018-04-25 03:16 AM, Matthew Knepley wrote: > > On Tue, Apr 24, 2018 at 11:57 PM, Danyang Su wrote: > >> Hi All, >> >> I use DMPlex in unstructured grid code and recently found DMSetLabelValue >> takes a lot of time for large problem, e.g., num. of cells > 1 million. In >> my code, I use >> > > I read your code wrong. For large loop, you should not use the convenience > function. You should use > > >> DMPlexCreateFromCellList () > > > DMGetLabel(dm, name, &label) > > >> > > >> Loop over all cells/nodes{ >> >> DMSetLabelValue >> > > Replace this by DMLabelSetValue(label, point, val) > > >> } >> >> DMPlexDistribute >> >> The code works fine except DMSetLabelValue takes a lot of time for large >> problem. I use DMSetLabelValue to set material id for all the nodes or >> cells so that each subdomain has a copy of material id. Is there any other >> functions that can be used more efficient, e.g. set labels by array, not 1 >> by 1? >> > > That should take much less time. > > Thanks, > > Matt > > >> Thanks, >> >> Danyang >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera-w at sdsu.edu Wed Apr 25 13:26:33 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Wed, 25 Apr 2018 11:26:33 -0700 Subject: [petsc-users] Streams scaling.log Message-ID: Hi, I'm running scaling tests on my system to check why my scaling is so poor, and after following the MPIVersion guidelines my scaling.log output looks like this: Number of MPI processes 1 Processor names node37 Triad: 12856.9252 Rate (MB/s) Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Triad: 9138.3320 Rate (MB/s) Triad: 9945.0006 Rate (MB/s) Triad: 10480.8471 Rate (MB/s) Triad: 12055.4846 Rate (MB/s) Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Triad: 7394.1014 Rate (MB/s) Triad: 5528.9757 Rate (MB/s) Triad: 6052.7506 Rate (MB/s) Triad: 6188.5710 Rate (MB/s) Triad: 6944.4515 Rate (MB/s) Triad: 7407.1594 Rate (MB/s) Triad: 9508.1984 Rate (MB/s) Triad: 10699.7551 Rate (MB/s) Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Number of MPI processes 1 Processor names node37 Triad: 6682.3749 Rate (MB/s) Triad: 6825.3243 Rate (MB/s) Triad: 7217.8178 Rate (MB/s) Triad: 7525.1025 Rate (MB/s) Triad: 7882.1781 Rate (MB/s) Triad: 8071.1430 Rate (MB/s) Triad: 10341.9424 Rate (MB/s) Triad: 10418.4740 Rate (MB/s) Is this normal? i feel is different from what i get from an usual streams test, how can i get it to work properly? Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 25 13:49:22 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 25 Apr 2018 18:49:22 +0000 Subject: [petsc-users] Streams scaling.log In-Reply-To: References: Message-ID: <069E5333-15F4-44D6-B444-9101991E099C@anl.gov> It should not have Number of MPI processes 1 it should have an increasing number of MPI processes. Do you have multiple MPI's installed on your machine and have the "wrong" mpiexec in your path for the MPI you built PETSc with? What command line options did you use for ./configure ? Barry > On Apr 25, 2018, at 1:26 PM, Manuel Valera wrote: > > Hi, > > I'm running scaling tests on my system to check why my scaling is so poor, and after following the MPIVersion guidelines my scaling.log output looks like this: > > Number of MPI processes 1 Processor names node37 > Triad: 12856.9252 Rate (MB/s) > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Triad: 9138.3320 Rate (MB/s) > Triad: 9945.0006 Rate (MB/s) > Triad: 10480.8471 Rate (MB/s) > Triad: 12055.4846 Rate (MB/s) > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Triad: 7394.1014 Rate (MB/s) > Triad: 5528.9757 Rate (MB/s) > Triad: 6052.7506 Rate (MB/s) > Triad: 6188.5710 Rate (MB/s) > Triad: 6944.4515 Rate (MB/s) > Triad: 7407.1594 Rate (MB/s) > Triad: 9508.1984 Rate (MB/s) > Triad: 10699.7551 Rate (MB/s) > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Triad: 6682.3749 Rate (MB/s) > Triad: 6825.3243 Rate (MB/s) > Triad: 7217.8178 Rate (MB/s) > Triad: 7525.1025 Rate (MB/s) > Triad: 7882.1781 Rate (MB/s) > Triad: 8071.1430 Rate (MB/s) > Triad: 10341.9424 Rate (MB/s) > Triad: 10418.4740 Rate (MB/s) > > > Is this normal? i feel is different from what i get from an usual streams test, how can i get it to work properly? > > Thanks, > > > > From rupp at iue.tuwien.ac.at Wed Apr 25 13:51:10 2018 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Wed, 25 Apr 2018 20:51:10 +0200 Subject: [petsc-users] Streams scaling.log In-Reply-To: References: Message-ID: <5bdf1358-5a90-264a-f8e8-1d24deacc0b7@iue.tuwien.ac.at> Hi Manuel, this looks like the wrong MPI gets used. You should see an increasing number of processes, e.g. Number of MPI processes 1 Processor names node37 Triad: 6052.3571 Rate (MB/s) Number of MPI processes 2 Processor names node37 node37 Triad: 9138.9376 Rate (MB/s) Number of MPI processes 3 Processor names node37 node37 node37 Triad: 11077.5905 Rate (MB/s) Number of MPI processes 4 Processor names node37 node37 node37 node37 Triad: 12055.9123 Rate (MB/s) Best regards, Karli On 04/25/2018 08:26 PM, Manuel Valera wrote: > Hi, > > I'm running scaling tests on my system to check why my scaling is so > poor, and after following the MPIVersion guidelines my scaling.log > output looks like this: > > Number of MPI processes 1 Processor names? node37 > Triad:? ? ? ? 12856.9252? ?Rate (MB/s) > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Triad:? ? ? ? ?9138.3320? ?Rate (MB/s) > Triad:? ? ? ? ?9945.0006? ?Rate (MB/s) > Triad:? ? ? ? 10480.8471? ?Rate (MB/s) > Triad:? ? ? ? 12055.4846? ?Rate (MB/s) > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Triad:? ? ? ? ?7394.1014? ?Rate (MB/s) > Triad:? ? ? ? ?5528.9757? ?Rate (MB/s) > Triad:? ? ? ? ?6052.7506? ?Rate (MB/s) > Triad:? ? ? ? ?6188.5710? ?Rate (MB/s) > Triad:? ? ? ? ?6944.4515? ?Rate (MB/s) > Triad:? ? ? ? ?7407.1594? ?Rate (MB/s) > Triad:? ? ? ? ?9508.1984? ?Rate (MB/s) > Triad:? ? ? ? 10699.7551? ?Rate (MB/s) > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Number of MPI processes 1 Processor names? node37 > Triad:? ? ? ? ?6682.3749? ?Rate (MB/s) > Triad:? ? ? ? ?6825.3243? ?Rate (MB/s) > Triad:? ? ? ? ?7217.8178? ?Rate (MB/s) > Triad:? ? ? ? ?7525.1025? ?Rate (MB/s) > Triad:? ? ? ? ?7882.1781? ?Rate (MB/s) > Triad:? ? ? ? ?8071.1430? ?Rate (MB/s) > Triad:? ? ? ? 10341.9424? ?Rate (MB/s) > Triad:? ? ? ? 10418.4740? ?Rate (MB/s) > > > Is this normal? i feel is different from what i get from an usual > streams test, how can i get it to work properly? > > Thanks, > > > > From mvalera-w at sdsu.edu Wed Apr 25 14:12:56 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Wed, 25 Apr 2018 12:12:56 -0700 Subject: [petsc-users] Streams scaling.log In-Reply-To: <5bdf1358-5a90-264a-f8e8-1d24deacc0b7@iue.tuwien.ac.at> References: <5bdf1358-5a90-264a-f8e8-1d24deacc0b7@iue.tuwien.ac.at> Message-ID: Hi and thanks for the quick answer, Yes it looks i am using MPICH for my configure instead of using the system installation of OpenMPI, in the past i had better experience using MPICH but maybe this will be a conflict, should i reconfigure using the system MPI installation? I solved the problem in a different way by login into the nodes i wanted to use and doing the make streams tests there, but i get the following: np speedup 1 1.0 2 1.51 3 2.17 4 2.66 5 2.87 6 3.06 7 3.44 8 3.84 9 3.81 10 3.17 11 3.69 12 3.81 13 3.26 14 3.51 15 3.61 16 3.81 17 3.8 18 3.64 19 3.48 20 4.01 So very modest scaling, this is about the same i get with my application, how can i make it work faster? i am already using --map-by and --machinefile arguments for mpirun, maybe this is also a conflict with the different MPI installations? Thanks, On Wed, Apr 25, 2018 at 11:51 AM, Karl Rupp wrote: > Hi Manuel, > > this looks like the wrong MPI gets used. You should see an increasing > number of processes, e.g. > > Number of MPI processes 1 Processor names node37 > Triad: 6052.3571 Rate (MB/s) > Number of MPI processes 2 Processor names node37 node37 > Triad: 9138.9376 Rate (MB/s) > Number of MPI processes 3 Processor names node37 node37 node37 > Triad: 11077.5905 Rate (MB/s) > Number of MPI processes 4 Processor names node37 node37 node37 node37 > Triad: 12055.9123 Rate (MB/s) > > Best regards, > Karli > > > > > On 04/25/2018 08:26 PM, Manuel Valera wrote: > >> Hi, >> >> I'm running scaling tests on my system to check why my scaling is so >> poor, and after following the MPIVersion guidelines my scaling.log output >> looks like this: >> >> Number of MPI processes 1 Processor names node37 >> Triad: 12856.9252 Rate (MB/s) >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Triad: 9138.3320 Rate (MB/s) >> Triad: 9945.0006 Rate (MB/s) >> Triad: 10480.8471 Rate (MB/s) >> Triad: 12055.4846 Rate (MB/s) >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Triad: 7394.1014 Rate (MB/s) >> Triad: 5528.9757 Rate (MB/s) >> Triad: 6052.7506 Rate (MB/s) >> Triad: 6188.5710 Rate (MB/s) >> Triad: 6944.4515 Rate (MB/s) >> Triad: 7407.1594 Rate (MB/s) >> Triad: 9508.1984 Rate (MB/s) >> Triad: 10699.7551 Rate (MB/s) >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Number of MPI processes 1 Processor names node37 >> Triad: 6682.3749 Rate (MB/s) >> Triad: 6825.3243 Rate (MB/s) >> Triad: 7217.8178 Rate (MB/s) >> Triad: 7525.1025 Rate (MB/s) >> Triad: 7882.1781 Rate (MB/s) >> Triad: 8071.1430 Rate (MB/s) >> Triad: 10341.9424 Rate (MB/s) >> Triad: 10418.4740 Rate (MB/s) >> >> >> Is this normal? i feel is different from what i get from an usual streams >> test, how can i get it to work properly? >> >> Thanks, >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 25 14:30:44 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 25 Apr 2018 19:30:44 +0000 Subject: [petsc-users] Streams scaling.log In-Reply-To: References: <5bdf1358-5a90-264a-f8e8-1d24deacc0b7@iue.tuwien.ac.at> Message-ID: <19F30C3D-C392-4C7F-80FB-41F6913341C4@anl.gov> > On Apr 25, 2018, at 2:12 PM, Manuel Valera wrote: > > Hi and thanks for the quick answer, > > Yes it looks i am using MPICH for my configure instead of using the system installation of OpenMPI, in the past i had better experience using MPICH but maybe this will be a conflict, should i reconfigure using the system MPI installation? > > I solved the problem in a different way by login into the nodes i wanted to use and doing the make streams tests there, but i get the following: > > np speedup > 1 1.0 > 2 1.51 > 3 2.17 > 4 2.66 > 5 2.87 > 6 3.06 > 7 3.44 > 8 3.84 > 9 3.81 > 10 3.17 > 11 3.69 > 12 3.81 > 13 3.26 > 14 3.51 > 15 3.61 > 16 3.81 > 17 3.8 > 18 3.64 > 19 3.48 > 20 4.01 > > So very modest scaling, this is about the same i get with my application, how can i make it work faster? You can't, the memory bandwidth is the limiting factor on this machine (not the number of cores) and there is nothing to be done about it. When buying machines make sure that the memory bandwidth is an important factor in the decision. Barry > i am already using --map-by and --machinefile arguments for mpirun, maybe this is also a conflict with the different MPI installations? > > Thanks, > > > > On Wed, Apr 25, 2018 at 11:51 AM, Karl Rupp wrote: > Hi Manuel, > > this looks like the wrong MPI gets used. You should see an increasing number of processes, e.g. > > Number of MPI processes 1 Processor names node37 > Triad: 6052.3571 Rate (MB/s) > Number of MPI processes 2 Processor names node37 node37 > Triad: 9138.9376 Rate (MB/s) > Number of MPI processes 3 Processor names node37 node37 node37 > Triad: 11077.5905 Rate (MB/s) > Number of MPI processes 4 Processor names node37 node37 node37 node37 > Triad: 12055.9123 Rate (MB/s) > > Best regards, > Karli > > > > > On 04/25/2018 08:26 PM, Manuel Valera wrote: > Hi, > > I'm running scaling tests on my system to check why my scaling is so poor, and after following the MPIVersion guidelines my scaling.log output looks like this: > > Number of MPI processes 1 Processor names node37 > Triad: 12856.9252 Rate (MB/s) > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Triad: 9138.3320 Rate (MB/s) > Triad: 9945.0006 Rate (MB/s) > Triad: 10480.8471 Rate (MB/s) > Triad: 12055.4846 Rate (MB/s) > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Triad: 7394.1014 Rate (MB/s) > Triad: 5528.9757 Rate (MB/s) > Triad: 6052.7506 Rate (MB/s) > Triad: 6188.5710 Rate (MB/s) > Triad: 6944.4515 Rate (MB/s) > Triad: 7407.1594 Rate (MB/s) > Triad: 9508.1984 Rate (MB/s) > Triad: 10699.7551 Rate (MB/s) > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Number of MPI processes 1 Processor names node37 > Triad: 6682.3749 Rate (MB/s) > Triad: 6825.3243 Rate (MB/s) > Triad: 7217.8178 Rate (MB/s) > Triad: 7525.1025 Rate (MB/s) > Triad: 7882.1781 Rate (MB/s) > Triad: 8071.1430 Rate (MB/s) > Triad: 10341.9424 Rate (MB/s) > Triad: 10418.4740 Rate (MB/s) > > > Is this normal? i feel is different from what i get from an usual streams test, how can i get it to work properly? > > Thanks, > > > > > From mvalera-w at sdsu.edu Wed Apr 25 14:34:24 2018 From: mvalera-w at sdsu.edu (Manuel Valera) Date: Wed, 25 Apr 2018 12:34:24 -0700 Subject: [petsc-users] Streams scaling.log In-Reply-To: <19F30C3D-C392-4C7F-80FB-41F6913341C4@anl.gov> References: <5bdf1358-5a90-264a-f8e8-1d24deacc0b7@iue.tuwien.ac.at> <19F30C3D-C392-4C7F-80FB-41F6913341C4@anl.gov> Message-ID: I get it, thanks, that's a strong argument i will tell my advisor about Have a great day, On Wed, Apr 25, 2018 at 12:30 PM, Smith, Barry F. wrote: > > > > On Apr 25, 2018, at 2:12 PM, Manuel Valera wrote: > > > > Hi and thanks for the quick answer, > > > > Yes it looks i am using MPICH for my configure instead of using the > system installation of OpenMPI, in the past i had better experience using > MPICH but maybe this will be a conflict, should i reconfigure using the > system MPI installation? > > > > I solved the problem in a different way by login into the nodes i wanted > to use and doing the make streams tests there, but i get the following: > > > > np speedup > > 1 1.0 > > 2 1.51 > > 3 2.17 > > 4 2.66 > > 5 2.87 > > 6 3.06 > > 7 3.44 > > 8 3.84 > > 9 3.81 > > 10 3.17 > > 11 3.69 > > 12 3.81 > > 13 3.26 > > 14 3.51 > > 15 3.61 > > 16 3.81 > > 17 3.8 > > 18 3.64 > > 19 3.48 > > 20 4.01 > > > > So very modest scaling, this is about the same i get with my > application, how can i make it work faster? > > You can't, the memory bandwidth is the limiting factor on this machine > (not the number of cores) and there is nothing to be done about it. When > buying machines make sure that the memory bandwidth is an important factor > in the decision. > > Barry > > > > i am already using --map-by and --machinefile arguments for mpirun, > maybe this is also a conflict with the different MPI installations? > > > > Thanks, > > > > > > > > On Wed, Apr 25, 2018 at 11:51 AM, Karl Rupp > wrote: > > Hi Manuel, > > > > this looks like the wrong MPI gets used. You should see an increasing > number of processes, e.g. > > > > Number of MPI processes 1 Processor names node37 > > Triad: 6052.3571 Rate (MB/s) > > Number of MPI processes 2 Processor names node37 node37 > > Triad: 9138.9376 Rate (MB/s) > > Number of MPI processes 3 Processor names node37 node37 node37 > > Triad: 11077.5905 Rate (MB/s) > > Number of MPI processes 4 Processor names node37 node37 node37 node37 > > Triad: 12055.9123 Rate (MB/s) > > > > Best regards, > > Karli > > > > > > > > > > On 04/25/2018 08:26 PM, Manuel Valera wrote: > > Hi, > > > > I'm running scaling tests on my system to check why my scaling is so > poor, and after following the MPIVersion guidelines my scaling.log output > looks like this: > > > > Number of MPI processes 1 Processor names node37 > > Triad: 12856.9252 Rate (MB/s) > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Triad: 9138.3320 Rate (MB/s) > > Triad: 9945.0006 Rate (MB/s) > > Triad: 10480.8471 Rate (MB/s) > > Triad: 12055.4846 Rate (MB/s) > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Triad: 7394.1014 Rate (MB/s) > > Triad: 5528.9757 Rate (MB/s) > > Triad: 6052.7506 Rate (MB/s) > > Triad: 6188.5710 Rate (MB/s) > > Triad: 6944.4515 Rate (MB/s) > > Triad: 7407.1594 Rate (MB/s) > > Triad: 9508.1984 Rate (MB/s) > > Triad: 10699.7551 Rate (MB/s) > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Number of MPI processes 1 Processor names node37 > > Triad: 6682.3749 Rate (MB/s) > > Triad: 6825.3243 Rate (MB/s) > > Triad: 7217.8178 Rate (MB/s) > > Triad: 7525.1025 Rate (MB/s) > > Triad: 7882.1781 Rate (MB/s) > > Triad: 8071.1430 Rate (MB/s) > > Triad: 10341.9424 Rate (MB/s) > > Triad: 10418.4740 Rate (MB/s) > > > > > > Is this normal? i feel is different from what i get from an usual > streams test, how can i get it to work properly? > > > > Thanks, > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Wed Apr 25 16:12:23 2018 From: danyang.su at gmail.com (Danyang Su) Date: Wed, 25 Apr 2018 14:12:23 -0700 Subject: [petsc-users] DMSetLabelValue takes a lot of time for large domain In-Reply-To: References: <3ee3963e-3fbe-9440-6754-861ef9271ae8@gmail.com> Message-ID: On 2018-04-25 09:47 AM, Matthew Knepley wrote: > On Wed, Apr 25, 2018 at 12:40 PM, Danyang Su > wrote: > > Hi Matthew, > > In the worst case, every node/cell may have different label. > > Do not use Label for this. Its not an appropriate thing. If every cell > is different, just use the cell number. > Labels are for mapping a relatively small number of keys (like > material IDs) to sets of points (cells, vertices, etc.) > Its not a great data structure for a permutation. Yes. If there is small number of keys, it runs very fast, even for more than one million DMSetLabelValue calls. The performance just deteriorates as the number of keys increases. I cannot get avoid of DMSetLabelValue as node/cell index of original mesh is needed for the previous input file that uses some of global node/cell index to set value. But if I can get the natural order of nodes/cells from DMPlex, I can discard the use of DMSetLabelValue. Is there any function can do this job? Thanks, Danyang > > However, I still do not believe these numbers. The old code does a > string comparison every time. I will setup a test. > > ? ?Matt > > Below is one of the worst scenario with 102299 nodes and 102299 > different labels for test. I found the time cost increase during > the loop. The first 9300 loop takes least time (<0.5) while the > last 9300 loops takes much more time (>7.7), as shown below. If I > use larger mesh with >1 million nodes, it runs very very slowly in > this part. The PETSc is configured with optimization on. > > Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > --download-mpich --download-scalapack --download-parmetis > --download-metis --download-ptscotch --download-fblaslapack > --download-hypre --download-superlu_dist --download-hdf5=yes > --download-ctetgen --with-debugging=0 COPTFLAGS="-O3 -march=native > -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" > FOPTFLAGS="-O3 -march=native -mtune=native" > > istart iend progress CPU_Time time cost - old (sec) time cost > - new (sec) > 0 9299 0 1524670045.51166 > > 9300 18599 0.100010753 1524670045.99605 0.4843890667 0.497246027 > 18600 27899 0.200010747 1524670047.32635 1.330302 1.3820912838 > 27900 37199 0.300010741 1524670049.3066 1.9802515507 > 2.2439446449 > 37200 46499 0.400010765 1524670052.1594 2.852804184 3.0739262104 > 46500 55799 0.500010729 1524670055.90961 3.7502081394 > 3.9270553589 > 55800 65099 0.600010753 1524670060.47654 4.5669286251 > 4.7571902275 > 65100 74399 0.700010777 1524670066.0941 5.6175630093 > 5.7428796291 > 74400 83699 0.800010741 1524670072.53886 6.44475317 6.5761549473 > 83700 92998 0.900010765 1524670079.99072 7.4518604279 > 7.4606924057 > 92999 102298 1 1524670087.71066 7.7199423313 8.2424075603 > > > > old code > > ??????? do ipoint = 0, istart-1 > ????????? !c output time cost, use 1 processor to test > ????????? if (b_enable_output .and. rank == 0) then > ??????????? if (mod(ipoint,iprogress) == 0 .or. ipoint == > istart-1) then > ????????????? !write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart > ????????????? write(*,*) ipoint, > (ipoint+1.0)/istart,"time",MPI_Wtime() > ??????????? end if > ????????? end if > > ????????? call DMSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, & > ?????????????????????????????? ipoint+1,ierr) > ????????? CHKERRQ(ierr) > ??????? end do > > > new code > > ??????? call DMCreateLabel(dmda_flow%da,'cid_lg2g',ierr) > ??????? CHKERRQ(ierr) > > ??????? call DMGetLabel(dmda_flow%da,'cid_lg2g',label, ierr) > ??????? CHKERRQ(ierr) > > ??????? do ipoint = 0, istart-1 > ????????? !c output time cost, use 1 processor to test > ????????? if (b_enable_output .and. rank == 0) then > ??????????? if (mod(ipoint,iprogress) == 0 .or. ipoint == > istart-1) then > ????????????? !write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart > ????????????? write(*,*) ipoint, > (ipoint+1.0)/istart,"time",MPI_Wtime() > ??????????? end if > ????????? end if > > ????????? call DMLabelSetValue(label,ipoint,ipoint+1,ierr) > ????????? CHKERRQ(ierr) > ??????? end do > > Thanks, > > Danyang > > On 2018-04-25 03:16 AM, Matthew Knepley wrote: >> On Tue, Apr 24, 2018 at 11:57 PM, Danyang Su >> > wrote: >> >> Hi All, >> >> I use DMPlex in unstructured grid code and recently found >> DMSetLabelValue takes a lot of time for large problem, e.g., >> num. of cells > 1 million. In my code, I use >> >> >> I read your code wrong. For large loop, you should not use the >> convenience function. You should use >> >> DMPlexCreateFromCellList () >> >> >> DMGetLabel(dm, name, &label) >> >> >> Loop over all cells/nodes{ >> >> DMSetLabelValue >> >> >> Replace this by DMLabelSetValue(label, point, val) >> >> } >> >> DMPlexDistribute >> >> The code works fine except DMSetLabelValue takes a lot of >> time for large problem. I use DMSetLabelValue to set material >> id for all the nodes or cells so that each subdomain has a >> copy of material id. Is there any other functions that can be >> used more efficient, e.g. set labels by array, not 1 by 1? >> >> >> That should take much less time. >> >> ? Thanks, >> >> ? ? ?Matt >> >> Thanks, >> >> Danyang >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Apr 25 19:08:17 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 25 Apr 2018 19:08:17 -0500 Subject: [petsc-users] configuration option for PETSc on cluster In-Reply-To: References: <849c31e6-8d27-2def-c2e6-73754181fc4a@gmail.com> <7ad332fc-290d-c8f0-4a60-b0f4b7b02415@gmail.com> <5094d7bb-f74c-8fee-b1c4-3b3ded0aaea1@gmail.com> Message-ID: Samuel, thanks for the update. Glad the workaround gave a useable build. BTW - looking at configure.log - I think the option LDFLAGS="-dynamic" is interfering with some of these tests - in this case the default -fPIC test - hence this need for the workarround. I think the problem won't appear if the alternative of setting 'CRAYPE_LINK_TYPE=dynamic' env variable is used. Satish On Wed, 25 Apr 2018, Samuel Lanthaler wrote: > Dear Satish, > Your workaround worked! Great! > Thanks a lot for all your help, everyone. =) > > Cheers, > Samuel > > On 04/24/2018 06:33 PM, Satish Balay wrote: > > Hm - I'm not sure why configure didn't try and set -fPIC compiler options. > > > > Workarround is: > > > > CFLAGS=-fPIC FFLAGS=-fPICC CXXFLAGS=-fPIC > > > > or alternatively - you can disable sharedlibraries > > > > --with-shared-libraries=0 > > > > Satish > > > > On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > > > >> Dear Satish, > >> > >> As you suggested, the configure appears to work with > >> > >> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC --with-debugging=0 > >> --with-scalar-type=complex --download-scalapack --download-mumps=yes > >> --download-superlu --with-batch --known-mpi-shared-libraries=1 > >> --with-blaslapack-dir=${MKLROOT} --known-64-bit-blas-indices=0 > >> --LDFLAGS="-dynamic" > >> > >> However, after doing the reconfigure, I tried to compile using make, and > >> ran > >> into trouble with CLINKER: > >> > >> [...] > >> CC arch-complex/obj/sys/classes/draw/impls/x/xops.o > >> CLINKER arch-complex/lib/libpetsc.so.3.09.0 > >> Warning: > >> -dynamic was already seen on command line, overriding with -shared. > >> /usr/bin/ld: arch-complex/obj/sys/fsrc/somefort.o: relocation R_X86_64_32 > >> against `petscfortran5_' can not be used when making a shared object; > >> recompile with -fPIC > >> arch-complex/obj/sys/fsrc/somefort.o: error adding symbols: Bad value > >> [...] > >> **************************ERROR************************************* > >> Error during compile, check arch-complex/lib/petsc/conf/make.log > >> Send it and arch-complex/lib/petsc/conf/configure.log to > >> petsc-maint at mcs.anl.gov > >> ******************************************************************** > >> makefile:30: recipe for target 'all' failed > >> make: *** [all] Error 1 > >> > >> Any ideas? Again, I'm attaching the .log files. > >> > >> > >> > >> On 04/24/2018 04:14 PM, Satish Balay wrote: > >>> Sorry I should have clarified - its a configure option. > >>> > >>> Satish > >>> > >>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >>> > >>>> Sorry, I'm confused: Can I add this somehow when doing the ./configure? > >>>> Or > >>>> do > >>>> you mean to add it when compiling ex5f with the configuration I had > >>>> initially? > >>>> > >>>> Cheers, > >>>> Samuel > >>>> > >>>> > >>>> On 04/24/2018 04:03 PM, Satish Balay wrote: > >>>>> Ok - I do see: libmkl_intel_lp64.so libmkl_sequential.so libmkl_core.so > >>>>> - > >>>>> these should > >>>>> be the ones that get picked up. > >>>>> > >>>>> I do not know why 'cc' is picking up static libraries instead of the > >>>>> shared ones. Perhaps it needs an extra option > >>>>> > >>>>> 'LDFLAGS=-dynamic' > >>>>> > >>>>> Satish > >>>>> > >>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >>>>> > >>>>>> Dear Satish, > >>>>>> > >>>>>> I get the following when doing the ls: > >>>>>> > >>>>>> [lanthale at daint103 mkl]$ ls > >>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ > >>>>>> libmkl_ao_worker.so libmkl_blacs_sgimpt_lp64.so > >>>>>> libmkl_intel_lp64.a libmkl_sequential.so > >>>>>> libmkl_avx2.so libmkl_blas95_ilp64.a > >>>>>> libmkl_intel_lp64.so > >>>>>> libmkl_tbb_thread.a > >>>>>> libmkl_avx512_mic.so libmkl_blas95_lp64.a > >>>>>> libmkl_intel_thread.a > >>>>>> libmkl_tbb_thread.so > >>>>>> libmkl_avx512.so libmkl_cdft_core.a > >>>>>> libmkl_intel_thread.so > >>>>>> libmkl_vml_avx2.so > >>>>>> libmkl_avx.so libmkl_cdft_core.so > >>>>>> libmkl_lapack95_ilp64.a > >>>>>> libmkl_vml_avx512_mic.so > >>>>>> libmkl_blacs_intelmpi_ilp64.a libmkl_core.a libmkl_lapack95_lp64.a > >>>>>> libmkl_vml_avx512.so > >>>>>> libmkl_blacs_intelmpi_ilp64.so libmkl_core.so libmkl_mc3.so > >>>>>> libmkl_vml_avx.so > >>>>>> libmkl_blacs_intelmpi_lp64.a libmkl_def.so libmkl_mc.so > >>>>>> libmkl_vml_cmpt.so > >>>>>> libmkl_blacs_intelmpi_lp64.so libmkl_gf_ilp64.a libmkl_pgi_thread.a > >>>>>> libmkl_vml_def.so > >>>>>> libmkl_blacs_openmpi_ilp64.a libmkl_gf_ilp64.so libmkl_pgi_thread.so > >>>>>> libmkl_vml_mc2.so > >>>>>> libmkl_blacs_openmpi_ilp64.so libmkl_gf_lp64.a libmkl_rt.so > >>>>>> libmkl_vml_mc3.so > >>>>>> libmkl_blacs_openmpi_lp64.a libmkl_gf_lp64.so > >>>>>> libmkl_scalapack_ilp64.a > >>>>>> libmkl_vml_mc.so > >>>>>> libmkl_blacs_openmpi_lp64.so libmkl_gnu_thread.a > >>>>>> libmkl_scalapack_ilp64.so > >>>>>> locale > >>>>>> libmkl_blacs_sgimpt_ilp64.a libmkl_gnu_thread.so > >>>>>> libmkl_scalapack_lp64.a > >>>>>> libmkl_blacs_sgimpt_ilp64.so libmkl_intel_ilp64.a > >>>>>> libmkl_scalapack_lp64.so > >>>>>> libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so > >>>>>> libmkl_sequential.a > >>>>>> > >>>>>> > >>>>>> Cheers, > >>>>>> Samuel > >>>>>> > >>>>>> On 04/24/2018 03:50 PM, Satish Balay wrote: > >>>>>>> Executing: cc -o /tmp/petsc-tyd_Hm/config.libraries/conftest -O > >>>>>>> /tmp/petsc-tyd_Hm/config.libraries/conftest.o > >>>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 > >>>>>>> -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 > >>>>>>> -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl > >>>>>>> Possible ERROR while running linker: exit code 256 > >>>>>>> stderr: > >>>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(mkl_semaphore.o): > >>>>>>> In function `mkl_serv_load_inspector': > >>>>>>> mkl_semaphore.c:(.text+0x123): warning: Using 'dlopen' in statically > >>>>>>> linked > >>>>>>> applications requires at runtime the shared libraries from the glibc > >>>>>>> version > >>>>>>> used for linking > >>>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/libmkl_core.a(dgetrs.o): > >>>>>>> In function `mkl_lapack_dgetrs': > >>>>>>> dgetrs_gen.f:(.text+0x224): undefined reference to `mkl_blas_dtrsm' > >>>>>>> dgetrs_gen.f:(.text+0x2b1): undefined reference to `mkl_blas_dtrsm' > >>>>>>> dgetrs_gen.f:(.text+0x2e7): undefined reference to `mkl_lapack_dlaswp' > >>>>>>> <<<<<<<< > >>>>>>> > >>>>>>> For some reason the compiler is picking up static version of MKL - > >>>>>>> instead > >>>>>>> of shared libraries. > >>>>>>> > >>>>>>> What do you have for: > >>>>>>> > >>>>>>> ls > >>>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64/ > >>>>>>> > >>>>>>> Satish > >>>>>>> > >>>>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >>>>>>> > >>>>>>>> Dear Satish and Stefano, > >>>>>>>> > >>>>>>>> Thank you for your answers. I believe I had initially tried to use > >>>>>>>> the > >>>>>>>> option > >>>>>>>> --with-blaslapack-dir=[...], instead of specifying lib and include > >>>>>>>> directly. > >>>>>>>> But that gives me an error message: > >>>>>>>> > >>>>>>>> ******************************************************************************* > >>>>>>>> UNABLE to CONFIGURE with GIVEN OPTIONS (see > >>>>>>>> configure.log > >>>>>>>> for > >>>>>>>> details): > >>>>>>>> ------------------------------------------------------------------------------- > >>>>>>>> You set a value for --with-blaslapack-dir=, but > >>>>>>>> /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl cannot be > >>>>>>>> used > >>>>>>>> ******************************************************************************* > >>>>>>>> > >>>>>>>> I attach the new configure.log. Do you think there is something wrong > >>>>>>>> with > >>>>>>>> the > >>>>>>>> mkl version that I'm trying to use? Or is it only related to petsc? > >>>>>>>> > >>>>>>>> Cheers, > >>>>>>>> Samuel > >>>>>>>> > >>>>>>>> > >>>>>>>> On 04/24/2018 02:55 PM, Satish Balay wrote: > >>>>>>>>> On Tue, 24 Apr 2018, Samuel Lanthaler wrote: > >>>>>>>>> > >>>>>>>>>> Hi there, > >>>>>>>>>> > >>>>>>>>>> I was wondering if you could help me with the correct configuration > >>>>>>>>>> of > >>>>>>>>>> PETSc-dev version on a cluster (https://www.cscs.ch/)? I'm not sure > >>>>>>>>>> which > >>>>>>>>>> information would be useful to you, but basically the problem seems > >>>>>>>>>> to > >>>>>>>>>> be > >>>>>>>>>> in > >>>>>>>>>> correctly compiling it with intel compiler and the existing mkl > >>>>>>>>>> library. > >>>>>>>>>> > >>>>>>>>>> The pre-installed mkl version is > >>>>>>>>>> > >>>>>>>>>> intel/17.0.4.196 > >>>>>>>>>> > >>>>>>>>>> I have tried various things and finally, I got it at least to > >>>>>>>>>> compile > >>>>>>>>>> with > >>>>>>>>>> the > >>>>>>>>>> following options (options chosen in reference to the mkl link > >>>>>>>>>> advisor...): > >>>>>>>>>> > >>>>>>>>>> ./configure --with-fc=ftn --with-cc=cc --with-cxx=CC > >>>>>>>>>> --with-debugging=0 > >>>>>>>>>> --with-scalar-type=complex --download-scalapack > >>>>>>>>>> --download-mumps=yes > >>>>>>>>>> --download-superlu --with-batch --known-mpi-shared-libraries=1 > >>>>>>>>>> *--with-blaslapack-lib=*" > >>>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a > >>>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group > >>>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a > >>>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_sequential.a > >>>>>>>>>> ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm > >>>>>>>>>> -ldl" > >>>>>>>>>> *--with-blaslapack-include*="[/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include,/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include/intel64/lp64]" > >>>>>>>>> Can you remove the above 2 options [--with-blaslapack-lib, > >>>>>>>>> --with-blaslapack-include] and use: > >>>>>>>>> > >>>>>>>>> --with-blas-lapack-dir=${MKLROOT} > >>>>>>>>> > >>>>>>>>> And see if you still have this problem? > >>>>>>>>> > >>>>>>>>> Satish > >>>>>>>>> > >>>>>>>>>> --known-64-bit-blas-indices=0 > >>>>>>>>>> > >>>>>>>>>> After compilation, when trying to compile > >>>>>>>>>> /users/lanthale/petsc-git/src/snes/examples/tutorials/ex5f I get > >>>>>>>>>> linking > >>>>>>>>>> errors (attached). Would anyone be able to help me out here? I > >>>>>>>>>> really > >>>>>>>>>> don't > >>>>>>>>>> have a good understanding of this. > >>>>>>>>>> > >>>>>>>>>> I'm attaching the configuration.log file, as well as the linking > >>>>>>>>>> errors > >>>>>>>>>> when > >>>>>>>>>> trying to compile ex5f. > >>>>>>>>>> > >>>>>>>>>> Thank you very much in advance! > >>>>>>>>>> > >>>>>>>>>> Best regards, > >>>>>>>>>> Samuel > >>>>>>>>>> > >>>> > >> > > From loic.gouarin at math.u-psud.fr Thu Apr 26 03:05:31 2018 From: loic.gouarin at math.u-psud.fr (Loic Gouarin) Date: Thu, 26 Apr 2018 10:05:31 +0200 Subject: [petsc-users] MATIS and DMDA in petsc4py In-Reply-To: References: <9faeb719-6e56-3f99-b29c-2978f1149c83@math.u-psud.fr> Message-ID: <0b5fff73-9667-94d6-4a65-39cc0cdbbd55@math.u-psud.fr> The problem is solved with the new 3.9.0 release on conda-forge. Thanks Lisandro !! Loic Le 23/04/2018 ? 13:25, Loic Gouarin a ?crit?: > Thanks Lawrence for your reply. > > Le 23/04/2018 ? 12:36, Lawrence Mitchell a ?crit?: >>> On 23 Apr 2018, at 11:32, Loic Gouarin wrote: >>> >>> Hi, >>> >>> I try to use MATIS from petsc4py but I am not able to use setValuesLocal (I have a segmentation fault). And I don't understand why. >>> >>> Could you tell me if I did a mistake or if it's a bug ? >> This code works for me with: >> >> PETSc: 7bfdb6492ce >> petsc4py: 9137bf78e >> >> Perhaps you have a mismatching petsc4py/PETSc combo? > The problem probably comes from the conda package. I reinstalled a new > environment and tried again but I still have this issue. > > Did you compile petsc and petsc4py ? > > The versions that I use are those given on conda-forge : petsc-3.8.4 > and petsc4py-3.8.1. > > I did the following thing > > conda create -n petsc_tmp > source activate petsc_tmp > conda install petsc4py -c conda-forge > > and execute my script. > > Loic >> Lawrence >> >>> My example >>> ======= >>> from petsc4py import PETSc >>> import numpy as np >>> >>> n = 10 >>> da = PETSc.DMDA().create([n, n], dof=1, stencil_width=1) >>> da.setMatType(PETSc.Mat.Type.IS) >>> >>> A = da.createMatrix() >>> #ltog = da.getLGMap() >>> #A.setLGMap(ltog, ltog) >>> >>> elem = da.getElements() >>> melem = np.ones((4, 4)) >>> >>> for e in elem: >>> A.setValuesLocal(e, e, melem, PETSc.InsertMode.ADD_VALUES) >>> >>> Thanks, >>> Loic >>> >>> -- >>> Tel: 01 69 15 60 14 >>> http://www.math.u-psud.fr/~gouarin >>> https://github.com/gouarin >>> > > -- > Tel: 01 69 15 60 14 > http://www.math.u-psud.fr/~gouarin > https://github.com/gouarin -- Tel: 01 69 15 60 14 http://www.math.u-psud.fr/~gouarin https://github.com/gouarin -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Thu Apr 26 07:02:31 2018 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 26 Apr 2018 15:02:31 +0300 Subject: [petsc-users] MATIS and DMDA in petsc4py In-Reply-To: <0b5fff73-9667-94d6-4a65-39cc0cdbbd55@math.u-psud.fr> References: <9faeb719-6e56-3f99-b29c-2978f1149c83@math.u-psud.fr> <0b5fff73-9667-94d6-4a65-39cc0cdbbd55@math.u-psud.fr> Message-ID: On 26 April 2018 at 11:05, Loic Gouarin wrote: > The problem is solved with the new 3.9.0 release on conda-forge. > > Thanks Lisandro !! > Well, you are most welcome, but I don't remember doing anything in particular to get that fixed :-) -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From stefano.zampini at gmail.com Thu Apr 26 07:41:03 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Thu, 26 Apr 2018 12:41:03 +0000 Subject: [petsc-users] MATIS and DMDA in petsc4py In-Reply-To: References: <9faeb719-6e56-3f99-b29c-2978f1149c83@math.u-psud.fr> <0b5fff73-9667-94d6-4a65-39cc0cdbbd55@math.u-psud.fr> Message-ID: Without a stacktrace, I can say that the segfault is probably due to the fact that explicit MATIS support in DMDA is just from the latest release. Stefano Il Gio 26 Apr 2018, 15:03 Lisandro Dalcin ha scritto: > On 26 April 2018 at 11:05, Loic Gouarin > wrote: > > The problem is solved with the new 3.9.0 release on conda-forge. > > > > Thanks Lisandro !! > > > > Well, you are most welcome, but I don't remember doing anything in > particular to get that fixed :-) > > -- > Lisandro Dalcin > ============ > Research Scientist > Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > > 4700 King Abdullah University of Science and Technology > al-Khawarizmi Bldg (Bldg 1), Office # 0109 > Thuwal 23955-6900, Kingdom of Saudi Arabia > http://www.kaust.edu.sa > > Office Phone: +966 12 808-0459 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gcdiwan at gmail.com Thu Apr 26 09:46:01 2018 From: gcdiwan at gmail.com (Ganesh Diwan) Date: Thu, 26 Apr 2018 15:46:01 +0100 Subject: [petsc-users] understanding gmres output Message-ID: Dear Petsc developers, I am trying to learn to use PETSc GMRES with petsc4py and also trying to follow the codes in petsc/src/ksp/ksp/examples/tutorials. Here is the Python code I use to read a linear system and solve it with GMRES in PETSc (petsc4py) # gmrestest.py # usual imports import numpy as np import scipy.io as sio import scipy from sys import getrefcount from petsc4py import PETSc # read the matrix data mat_contents = sio.loadmat('data.mat') mat_A = mat_contents['A'] vec_b = mat_contents['b'] n = mat_contents['n'] vec_x = mat_contents['x'] mat_nnzA = mat_contents['nnzA'] # form the petsc matrices x = PETSc.Vec().createWithArray(vec_x) b = PETSc.Vec().createWithArray(vec_b) p1=mat_A.indptr p2=mat_A.indices p3=mat_A.data A = PETSc.Mat().createAIJ(size=mat_A.shape,csr=(p1,p2,p3)) A = scipy.transpose(A) # transpose the csr format as my matrix is column major originally A.assemblyBegin() A.assemblyEnd() # solve ksp = PETSc.KSP() ksp.create(PETSc.COMM_WORLD) ksp.setType('gmres') ksp.setOperators(A) ksp.setFromOptions() rtol = 1e-4 ksp.setTolerances(rtol, 1e-5, 10000., 10000) ksp.view() ksp.setConvergenceHistory() ksp.solve(b, x) # print print 'iterations = ', ksp.getIterationNumber() print 'residual = ', '{:.2e}'.format(ksp.getResidualNorm())# %.2E ksp.getResidualNorm() print 'norm(b)*rtol = ', '{:.2e}'.format(PETSc.Vec.norm(b)*rtol)# %.2E PETSc.Vec.norm(b)*rtol print 'converge reason# = ', ksp.getConvergedReason() print 'residuals at each iter = ', ksp.getConvergenceHistory() # Here is the output from the above code for a linear system of dimension 100 by 100. KSP Object: 1 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=0.0001, absolute=1e-05, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: 1 MPI processes type: ilu PC has not been set up so information may be incomplete out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=100, cols=100 total: nonzeros=2704, allocated nonzeros=2704 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 25 nodes, limit used is 5 iterations = 5 residual = 1.38e-03 norm(b)*rtol = 1.99e-01 converge reason# = 2 residuals at each iter = [ 2.05677686e+01 4.97916031e+00 4.82888782e-01 1.16849581e-01 8.87159777e-03 1.37992327e-03] Sorry if this sounds stupid, but I am trying to understand the output from PETSc by contrasting it with Matlab GMRES. I see that the residual < norm(b)*rtol for the 4th iteration itself, I am not sure then why GMRES continues for one more iteration. Secondly, how does one get total number of iterations? For eg. let's say if it takes 3 outer iterations each with the default restart of 30, then one would expect the length of residual vector to be 150. In the above specific example, it took GMRES a total of 5 iterations to converge. Does it mean the convergence was achieved without having to restart? Thank you, Ganesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From loic.gouarin at math.u-psud.fr Thu Apr 26 09:58:08 2018 From: loic.gouarin at math.u-psud.fr (Loic Gouarin) Date: Thu, 26 Apr 2018 16:58:08 +0200 Subject: [petsc-users] MATIS and DMDA in petsc4py In-Reply-To: References: <9faeb719-6e56-3f99-b29c-2978f1149c83@math.u-psud.fr> <0b5fff73-9667-94d6-4a65-39cc0cdbbd55@math.u-psud.fr> Message-ID: Thanks Stefano for your reply. I didn't know that matis was not supported for DMDA in previous version. This could be explain my segfault. I give you in attached file the stacktrace but I am not sure that it helps. Anyway, it works with 3.9 so it's fine for me. Thanks, Loic Le 26/04/2018 ? 14:41, Stefano Zampini a ?crit?: > Without a stacktrace, I can say that the segfault is probably due to > the fact that explicit? MATIS support in DMDA is just from the latest > release. > > Stefano > > Il Gio 26 Apr 2018, 15:03 Lisandro Dalcin > ha scritto: > > On 26 April 2018 at 11:05, Loic Gouarin > > > wrote: > > The problem is solved with the new 3.9.0 release on conda-forge. > > > > Thanks Lisandro !! > > > > Well, you are most welcome, but I don't remember doing anything in > particular to get that fixed :-) > > -- > Lisandro Dalcin > ============ > Research Scientist > Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > > 4700 King Abdullah University of Science and Technology > al-Khawarizmi Bldg (Bldg 1), Office # 0109 > Thuwal 23955-6900, Kingdom of Saudi Arabia > http://www.kaust.edu.sa > > Office Phone: +966 12 808-0459 > -- Tel: 01 69 15 60 14 http://www.math.u-psud.fr/~gouarin https://github.com/gouarin -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: strace_matis.log Type: text/x-log Size: 623487 bytes Desc: not available URL: From bsmith at mcs.anl.gov Thu Apr 26 13:24:47 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 26 Apr 2018 18:24:47 +0000 Subject: [petsc-users] understanding gmres output In-Reply-To: References: Message-ID: * You may/likely won't get the same convergence with Matlab not due to GMRES but due to differences in the ILU in Matlab and PETSc. Better to run with Jacobi or no preconditioner if you wish to get consistent results. * Check carefully if Matlab using left or right preconditioning for GMRES (PETSc uses left by default). > I see that the residual < norm(b)*rtol for the 4th iteration itself, I am not sure then why GMRES continues for one more iteration. * Since PETSc is using left preconditioning it is norm(B*b)*rtol that determines the convergence, not norm(b)*rtol. Where B is application of preconditioner. * The residual history for GMRES includes everything through any restarts so if it takes 90 total iterations (and the restart is 30) the residual history should have 90 values. > In the above specific example, it took GMRES a total of 5 iterations to converge. Does it mean the convergence was achieved without having to restart? In most cases if you need to restart then you need a better preconditioner. Barry > On Apr 26, 2018, at 9:46 AM, Ganesh Diwan wrote: > > Dear Petsc developers, > > I am trying to learn to use PETSc GMRES with petsc4py and also trying to follow the codes in petsc/src/ksp/ksp/examples/tutorials. Here is the Python code I use to read a linear system and solve it with GMRES in PETSc (petsc4py) > > # gmrestest.py > # usual imports > import numpy as np > import scipy.io as sio > import scipy > from sys import getrefcount > from petsc4py import PETSc > > # read the matrix data > mat_contents = sio.loadmat('data.mat') > mat_A = mat_contents['A'] > vec_b = mat_contents['b'] > n = mat_contents['n'] > vec_x = mat_contents['x'] > mat_nnzA = mat_contents['nnzA'] > > # form the petsc matrices > x = PETSc.Vec().createWithArray(vec_x) > b = PETSc.Vec().createWithArray(vec_b) > p1=mat_A.indptr > p2=mat_A.indices > p3=mat_A.data > A = PETSc.Mat().createAIJ(size=mat_A.shape,csr=(p1,p2,p3)) > A = scipy.transpose(A) # transpose the csr format as my matrix is column major originally > A.assemblyBegin() > A.assemblyEnd() > > # solve > ksp = PETSc.KSP() > ksp.create(PETSc.COMM_WORLD) > ksp.setType('gmres') > ksp.setOperators(A) > ksp.setFromOptions() > rtol = 1e-4 > ksp.setTolerances(rtol, 1e-5, 10000., 10000) > ksp.view() > ksp.setConvergenceHistory() > ksp.solve(b, x) > > # print > print 'iterations = ', ksp.getIterationNumber() > print 'residual = ', '{:.2e}'.format(ksp.getResidualNorm())# %.2E ksp.getResidualNorm() > print 'norm(b)*rtol = ', '{:.2e}'.format(PETSc.Vec.norm(b)*rtol)# %.2E PETSc.Vec.norm(b)*rtol > print 'converge reason# = ', ksp.getConvergedReason() > print 'residuals at each iter = ', ksp.getConvergenceHistory() > # > > Here is the output from the above code for a linear system of dimension 100 by 100. > > > KSP Object: 1 MPI processes > type: gmres > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=0.0001, absolute=1e-05, divergence=10000. > left preconditioning > using DEFAULT norm type for convergence test > PC Object: 1 MPI processes > type: ilu > PC has not been set up so information may be incomplete > out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=100, cols=100 > total: nonzeros=2704, allocated nonzeros=2704 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 25 nodes, limit used is 5 > iterations = 5 > residual = 1.38e-03 > norm(b)*rtol = 1.99e-01 > converge reason# = 2 > residuals at each iter = [ 2.05677686e+01 4.97916031e+00 4.82888782e-01 1.16849581e-01 > 8.87159777e-03 1.37992327e-03] > > Sorry if this sounds stupid, but I am trying to understand the output from PETSc by contrasting it with Matlab GMRES. > I see that the residual < norm(b)*rtol for the 4th iteration itself, I am not sure then why GMRES continues for one more iteration. Secondly, how does one get total number of iterations? For eg. let's say if it takes 3 outer iterations each with the default restart of 30, then one would expect the length of residual vector to be 150. In the above specific example, it took GMRES a total of 5 iterations to converge. Does it mean the convergence was achieved without having to restart? > > Thank you, Ganesh From danyang.su at gmail.com Fri Apr 27 01:09:23 2018 From: danyang.su at gmail.com (Danyang Su) Date: Thu, 26 Apr 2018 23:09:23 -0700 Subject: [petsc-users] Get vertex index of each cell in DMPlex after distribution In-Reply-To: References: <3ee3963e-3fbe-9440-6754-861ef9271ae8@gmail.com> Message-ID: <979e9fba-286f-9414-2ad6-6e652019db4d@gmail.com> Hi Matt, Sorry if this is a stupid question. In the previous code for unstructured grid, I create labels to mark the original node/cell index from VTK file and then distribute it so that each subdomain has a copy of its original node and cell index, as well as the PETSc numbering. Now I am trying to get avoid of using large number of keys in DMSetLabelValue since this costs lot of time for large problem. I can get the coordinates of subdomain after distribution by using DMGetCoordinatesLocal and DMGetCoordinateDM. How can I get the vertex index of each cell after distribution? Would you please give me a hint or functions that I can use. Thanks, Danyang On 18-04-25 02:12 PM, Danyang Su wrote: > On 2018-04-25 09:47 AM, Matthew Knepley wrote: >> On Wed, Apr 25, 2018 at 12:40 PM, Danyang Su > > wrote: >> >> Hi Matthew, >> >> In the worst case, every node/cell may have different label. >> >> Do not use Label for this. Its not an appropriate thing. If every >> cell is different, just use the cell number. >> Labels are for mapping a relatively small number of keys (like >> material IDs) to sets of points (cells, vertices, etc.) >> Its not a great data structure for a permutation. > Yes. If there is small number of keys, it runs very fast, even for > more than one million DMSetLabelValue calls. The performance just > deteriorates as the number of keys increases. > > I cannot get avoid of DMSetLabelValue as node/cell index of original > mesh is needed for the previous input file that uses some of global > node/cell index to set value. But if I can get the natural order of > nodes/cells from DMPlex, I can discard the use of DMSetLabelValue. Is > there any function can do this job? > > Thanks, > > Danyang >> >> However, I still do not believe these numbers. The old code does a >> string comparison every time. I will setup a test. >> >> Matt >> >> Below is one of the worst scenario with 102299 nodes and 102299 >> different labels for test. I found the time cost increase during >> the loop. The first 9300 loop takes least time (<0.5) while the >> last 9300 loops takes much more time (>7.7), as shown below. If I >> use larger mesh with >1 million nodes, it runs very very slowly >> in this part. The PETSc is configured with optimization on. >> >> Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran >> --download-mpich --download-scalapack --download-parmetis >> --download-metis --download-ptscotch --download-fblaslapack >> --download-hypre --download-superlu_dist --download-hdf5=yes >> --download-ctetgen --with-debugging=0 COPTFLAGS="-O3 >> -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native >> -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" >> >> istart iend progress CPU_Time time cost - old (sec) time >> cost - new (sec) >> 0 9299 0 1524670045.51166 >> >> 9300 18599 0.100010753 1524670045.99605 0.4843890667 >> 0.497246027 >> 18600 27899 0.200010747 1524670047.32635 1.330302 1.3820912838 >> 27900 37199 0.300010741 1524670049.3066 1.9802515507 >> 2.2439446449 >> 37200 46499 0.400010765 1524670052.1594 2.852804184 >> 3.0739262104 >> 46500 55799 0.500010729 1524670055.90961 3.7502081394 >> 3.9270553589 >> 55800 65099 0.600010753 1524670060.47654 4.5669286251 >> 4.7571902275 >> 65100 74399 0.700010777 1524670066.0941 5.6175630093 >> 5.7428796291 >> 74400 83699 0.800010741 1524670072.53886 6.44475317 >> 6.5761549473 >> 83700 92998 0.900010765 1524670079.99072 7.4518604279 >> 7.4606924057 >> 92999 102298 1 1524670087.71066 7.7199423313 8.2424075603 >> >> >> >> old code >> >> do ipoint = 0, istart-1 >> !c output time cost, use 1 processor to test >> if (b_enable_output .and. rank == 0) then >> if (mod(ipoint,iprogress) == 0 .or. ipoint == >> istart-1) then >> !write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart >> write(*,*) ipoint, >> (ipoint+1.0)/istart,"time",MPI_Wtime() >> end if >> end if >> >> call DMSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, & >> ipoint+1,ierr) >> CHKERRQ(ierr) >> end do >> >> >> new code >> >> call DMCreateLabel(dmda_flow%da,'cid_lg2g',ierr) >> CHKERRQ(ierr) >> >> call DMGetLabel(dmda_flow%da,'cid_lg2g',label, ierr) >> CHKERRQ(ierr) >> >> do ipoint = 0, istart-1 >> !c output time cost, use 1 processor to test >> if (b_enable_output .and. rank == 0) then >> if (mod(ipoint,iprogress) == 0 .or. ipoint == >> istart-1) then >> !write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart >> write(*,*) ipoint, >> (ipoint+1.0)/istart,"time",MPI_Wtime() >> end if >> end if >> >> call DMLabelSetValue(label,ipoint,ipoint+1,ierr) >> CHKERRQ(ierr) >> end do >> >> Thanks, >> >> Danyang >> >> On 2018-04-25 03:16 AM, Matthew Knepley wrote: >>> On Tue, Apr 24, 2018 at 11:57 PM, Danyang Su >>> > wrote: >>> >>> Hi All, >>> >>> I use DMPlex in unstructured grid code and recently found >>> DMSetLabelValue takes a lot of time for large problem, e.g., >>> num. of cells > 1 million. In my code, I use >>> >>> >>> I read your code wrong. For large loop, you should not use the >>> convenience function. You should use >>> >>> DMPlexCreateFromCellList () >>> >>> >>> DMGetLabel(dm, name, &label) >>> >>> >>> Loop over all cells/nodes{ >>> >>> DMSetLabelValue >>> >>> >>> Replace this by DMLabelSetValue(label, point, val) >>> >>> } >>> >>> DMPlexDistribute >>> >>> The code works fine except DMSetLabelValue takes a lot of >>> time for large problem. I use DMSetLabelValue to set >>> material id for all the nodes or cells so that each >>> subdomain has a copy of material id. Is there any other >>> functions that can be used more efficient, e.g. set labels >>> by array, not 1 by 1? >>> >>> >>> That should take much less time. >>> >>> Thanks, >>> >>> Matt >>> >>> Thanks, >>> >>> Danyang >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to >>> which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aroli.marcellinus at gmail.com Fri Apr 27 03:31:23 2018 From: aroli.marcellinus at gmail.com (Aroli Marcellinus) Date: Fri, 27 Apr 2018 17:31:23 +0900 Subject: [petsc-users] libpciaccess dependency Problem Message-ID: When I try to compile the program with PETSC, and run it with mpirun using qsub. And I have checked the log and it something like this: cep2017_heart: error while loading shared libraries: libpciaccess.so.0: cannot open shared object file: No such file or directory and I check ldd, the result is like this: ldd bin/cep2017_heart linux-vdso.so.1 => (0x00007fff99787000) libsundials_cvode.so.2 => /opt/Lib/sundials-2.7.0/lib/libsundials_cvode.so.2 (0x00002ac2b2262000) libsundials_nvecserial.so.2 => /opt/Lib/sundials-2.7.0/lib/libsundials_nvecserial.so.2 (0x00002ac2b248d000) libsundials_nvecparallel.so.2 => /opt/Lib/sundials-2.7.0/lib/libsundials_nvecparallel.so.2 (0x00002ac2b2692000) libX11.so.6 => /usr/lib64/libX11.so.6 (0x00000039f0c00000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002ac2b28c7000) libdl.so.2 => /lib64/libdl.so.2 (0x00000039efc00000) libmpi_usempif08.so.20 => /opt/Lib/openmpi-2.1.3/lib/libmpi_usempif08.so.20 (0x00002ac2b2c6a000) libmpi_usempi_ignore_tkr.so.20 => /opt/Lib/openmpi-2.1.3/lib/libmpi_usempi_ignore_tkr.so.20 (0x00002ac2b2e9d000) libmpi_mpifh.so.20 => /opt/Lib/openmpi-2.1.3/lib/libmpi_mpifh.so.20 (0x00002ac2b30a6000) libmpi.so.20 => /opt/Lib/openmpi-2.1.3/lib/libmpi.so.20 (0x00002ac2b3302000) libifport.so.5 => /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libifport.so.5 (0x00002ac2b35f5000) libifcore.so.5 => /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libifcore.so.5 (0x00002ac2b372a000) libimf.so => /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libimf.so (0x00002ac2b396d000) libsvml.so => /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libsvml.so (0x00002ac2b3d39000) libm.so.6 => /lib64/libm.so.6 (0x00000039f0000000) libirc.so => /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libirc.so (0x00002ac2b44ac000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00000039f0400000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000035b0800000) libz.so.1 => /lib64/libz.so.1 (0x00000037c7600000) libc.so.6 => /lib64/libc.so.6 (0x00000039ef800000) libXau.so.6 => /usr/lib64/libXau.so.6 (0x00000039f0800000) libXdmcp.so.6 => /usr/lib64/libXdmcp.so.6 (0x00000039f1000000) /lib64/ld-linux-x86-64.so.2 (0x00000039ef000000) libopen-rte.so.20 => /opt/Lib/openmpi-2.1.3/lib/libopen-rte.so.20 (0x00002ac2b45fb000) libopen-pal.so.20 => /opt/Lib/openmpi-2.1.3/lib/libopen-pal.so.20 (0x00002ac2b4884000) libpciaccess.so.0 => /usr/lib64/libpciaccess.so.0 (0x00002ac2b4b86000) librt.so.1 => /lib64/librt.so.1 (0x00000039f3c00000) libutil.so.1 => /lib64/libutil.so.1 (0x00000039fca00000) libintlc.so.5 => /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libintlc.so.5 (0x00002ac2b4d8d000) libifcoremt.so.5 => /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libifcoremt.so.5 (0x00002ac2b4eda000) So as you can see that the libpciaccess already coupled to the compiled program, but it's still error. So I know that libpciaccess maybe from the MPI libs (I'm using OpenMPI), but this kind of error is really bugging me. Even after I tried to use --disable-hwloc-pci when installing OpenMPI, it still got this kind of error,. What I want to know is: - Is it possible to remove the dependenciness of libpciaccess? - Or how to configure it so that PETSc or MPI doesn't use libpciaccess? In my case, I cannot use static because some of those dependencies doesn't have static libraries. I'm using: PETSc-3.9.0 OpenMPI-2.1.3 Aroli Marcellinus *Kumoh Institute of Technology**Computational Medicine Laboratory* 61 Daehak-ro (Sinpyeong-dong), Gumi, Gyeongbuk +82 10 9724 3957 KTalk ID: vondarkness -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 27 06:11:19 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 27 Apr 2018 07:11:19 -0400 Subject: [petsc-users] Get vertex index of each cell in DMPlex after distribution In-Reply-To: <979e9fba-286f-9414-2ad6-6e652019db4d@gmail.com> References: <3ee3963e-3fbe-9440-6754-861ef9271ae8@gmail.com> <979e9fba-286f-9414-2ad6-6e652019db4d@gmail.com> Message-ID: On Fri, Apr 27, 2018 at 2:09 AM, Danyang Su wrote: > Hi Matt, > > Sorry if this is a stupid question. > > In the previous code for unstructured grid, I create labels to mark the > original node/cell index from VTK file and then distribute it so that each > subdomain has a copy of its original node and cell index, as well as the > PETSc numbering. Now I am trying to get avoid of using large number of keys > in DMSetLabelValue since this costs lot of time for large problem. > > I can get the coordinates of subdomain after distribution by using > DMGetCoordinatesLocal and DMGetCoordinateDM. > > How can I get the vertex index of each cell after distribution? Would you > please give me a hint or functions that I can use. > You can permute the vectors back to the natural ordering using http://www.mcs.anl.gov/petsc/petsc-master/docs/manualpages/DMPLEX/DMPlexNaturalToGlobalBegin.html which says you have to call DMPlexSetUseNaturalSF() before distributing the mesh. It is tested in src/dm/impls/plex/examples/tests/ex15.c so you can see how its intended to work. It is very new and has not been tested by many people. I can see how you might want this for small tests. Why would you want it for production models? Thanks, Matt > Thanks, > > Danyang > > On 18-04-25 02:12 PM, Danyang Su wrote: > > On 2018-04-25 09:47 AM, Matthew Knepley wrote: > > On Wed, Apr 25, 2018 at 12:40 PM, Danyang Su wrote: > >> Hi Matthew, >> >> In the worst case, every node/cell may have different label. >> > Do not use Label for this. Its not an appropriate thing. If every cell is > different, just use the cell number. > Labels are for mapping a relatively small number of keys (like material > IDs) to sets of points (cells, vertices, etc.) > Its not a great data structure for a permutation. > > Yes. If there is small number of keys, it runs very fast, even for more > than one million DMSetLabelValue calls. The performance just deteriorates > as the number of keys increases. > > I cannot get avoid of DMSetLabelValue as node/cell index of original mesh > is needed for the previous input file that uses some of global node/cell > index to set value. But if I can get the natural order of nodes/cells from > DMPlex, I can discard the use of DMSetLabelValue. Is there any function can > do this job? > > Thanks, > > Danyang > > > However, I still do not believe these numbers. The old code does a string > comparison every time. I will setup a test. > > Matt > >> Below is one of the worst scenario with 102299 nodes and 102299 different >> labels for test. I found the time cost increase during the loop. The first >> 9300 loop takes least time (<0.5) while the last 9300 loops takes much more >> time (>7.7), as shown below. If I use larger mesh with >1 million nodes, it >> runs very very slowly in this part. The PETSc is configured with >> optimization on. >> >> Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran >> --download-mpich --download-scalapack --download-parmetis --download-metis >> --download-ptscotch --download-fblaslapack --download-hypre >> --download-superlu_dist --download-hdf5=yes --download-ctetgen >> --with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" >> CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native >> -mtune=native" >> >> istart iend progress CPU_Time time cost - old (sec) time cost - new (sec) >> 0 9299 0 1524670045.51166 >> >> 9300 18599 0.100010753 1524670045.99605 0.4843890667 0.497246027 >> 18600 27899 0.200010747 1524670047.32635 1.330302 1.3820912838 >> 27900 37199 0.300010741 1524670049.3066 1.9802515507 2.2439446449 >> 37200 46499 0.400010765 1524670052.1594 2.852804184 3.0739262104 >> 46500 55799 0.500010729 1524670055.90961 3.7502081394 3.9270553589 >> 55800 65099 0.600010753 1524670060.47654 4.5669286251 4.7571902275 >> 65100 74399 0.700010777 1524670066.0941 5.6175630093 5.7428796291 >> 74400 83699 0.800010741 1524670072.53886 6.44475317 6.5761549473 >> 83700 92998 0.900010765 1524670079.99072 7.4518604279 7.4606924057 >> 92999 102298 1 1524670087.71066 7.7199423313 8.2424075603 >> >> old code >> >> do ipoint = 0, istart-1 >> !c output time cost, use 1 processor to test >> if (b_enable_output .and. rank == 0) then >> if (mod(ipoint,iprogress) == 0 .or. ipoint == istart-1) then >> !write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart >> write(*,*) ipoint, (ipoint+1.0)/istart,"time",MPI_Wtime() >> end if >> end if >> >> call DMSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, & >> ipoint+1,ierr) >> CHKERRQ(ierr) >> end do >> >> >> new code >> >> call DMCreateLabel(dmda_flow%da,'cid_lg2g',ierr) >> CHKERRQ(ierr) >> >> call DMGetLabel(dmda_flow%da,'cid_lg2g',label, ierr) >> CHKERRQ(ierr) >> >> do ipoint = 0, istart-1 >> !c output time cost, use 1 processor to test >> if (b_enable_output .and. rank == 0) then >> if (mod(ipoint,iprogress) == 0 .or. ipoint == istart-1) then >> !write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart >> write(*,*) ipoint, (ipoint+1.0)/istart,"time",MPI_Wtime() >> end if >> end if >> >> call DMLabelSetValue(label,ipoint,ipoint+1,ierr) >> CHKERRQ(ierr) >> end do >> >> Thanks, >> >> Danyang >> >> On 2018-04-25 03:16 AM, Matthew Knepley wrote: >> >> On Tue, Apr 24, 2018 at 11:57 PM, Danyang Su >> wrote: >> >>> Hi All, >>> >>> I use DMPlex in unstructured grid code and recently found >>> DMSetLabelValue takes a lot of time for large problem, e.g., num. of cells >>> > 1 million. In my code, I use >>> >> >> I read your code wrong. For large loop, you should not use the >> convenience function. You should use >> >> >>> DMPlexCreateFromCellList () >> >> >> DMGetLabel(dm, name, &label) >> >> >>> >> >> >>> Loop over all cells/nodes{ >>> >>> DMSetLabelValue >>> >> >> Replace this by DMLabelSetValue(label, point, val) >> >> >>> } >>> >>> DMPlexDistribute >>> >>> The code works fine except DMSetLabelValue takes a lot of time for large >>> problem. I use DMSetLabelValue to set material id for all the nodes or >>> cells so that each subdomain has a copy of material id. Is there any other >>> functions that can be used more efficient, e.g. set labels by array, not 1 >>> by 1? >>> >> >> That should take much less time. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> Danyang >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Apr 27 08:11:59 2018 From: jed at jedbrown.org (Jed Brown) Date: Fri, 27 Apr 2018 07:11:59 -0600 Subject: [petsc-users] libpciaccess dependency Problem In-Reply-To: References: Message-ID: <87r2n07q00.fsf@jedbrown.org> Make sure your batch environment has the same modules loaded so you get the same versions of all the libraries. Aroli Marcellinus writes: > When I try to compile the program with PETSC, and run it with mpirun using > qsub. And I have checked the log and it something like this: > > cep2017_heart: error while loading shared libraries: libpciaccess.so.0: > cannot open shared object file: No such file or directory > > and I check ldd, the result is like this: > ldd bin/cep2017_heart > linux-vdso.so.1 => (0x00007fff99787000) > libsundials_cvode.so.2 => > /opt/Lib/sundials-2.7.0/lib/libsundials_cvode.so.2 (0x00002ac2b2262000) > libsundials_nvecserial.so.2 => > /opt/Lib/sundials-2.7.0/lib/libsundials_nvecserial.so.2 (0x00002ac2b248d000) > libsundials_nvecparallel.so.2 => > /opt/Lib/sundials-2.7.0/lib/libsundials_nvecparallel.so.2 > (0x00002ac2b2692000) > libX11.so.6 => /usr/lib64/libX11.so.6 (0x00000039f0c00000) > libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002ac2b28c7000) > libdl.so.2 => /lib64/libdl.so.2 (0x00000039efc00000) > libmpi_usempif08.so.20 => > /opt/Lib/openmpi-2.1.3/lib/libmpi_usempif08.so.20 (0x00002ac2b2c6a000) > libmpi_usempi_ignore_tkr.so.20 => > /opt/Lib/openmpi-2.1.3/lib/libmpi_usempi_ignore_tkr.so.20 > (0x00002ac2b2e9d000) > libmpi_mpifh.so.20 => /opt/Lib/openmpi-2.1.3/lib/libmpi_mpifh.so.20 > (0x00002ac2b30a6000) > libmpi.so.20 => /opt/Lib/openmpi-2.1.3/lib/libmpi.so.20 > (0x00002ac2b3302000) > libifport.so.5 => > /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libifport.so.5 > (0x00002ac2b35f5000) > libifcore.so.5 => > /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libifcore.so.5 > (0x00002ac2b372a000) > libimf.so => > /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libimf.so > (0x00002ac2b396d000) > libsvml.so => > /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libsvml.so > (0x00002ac2b3d39000) > libm.so.6 => /lib64/libm.so.6 (0x00000039f0000000) > libirc.so => > /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libirc.so > (0x00002ac2b44ac000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00000039f0400000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000035b0800000) > libz.so.1 => /lib64/libz.so.1 (0x00000037c7600000) > libc.so.6 => /lib64/libc.so.6 (0x00000039ef800000) > libXau.so.6 => /usr/lib64/libXau.so.6 (0x00000039f0800000) > libXdmcp.so.6 => /usr/lib64/libXdmcp.so.6 (0x00000039f1000000) > /lib64/ld-linux-x86-64.so.2 (0x00000039ef000000) > libopen-rte.so.20 => /opt/Lib/openmpi-2.1.3/lib/libopen-rte.so.20 > (0x00002ac2b45fb000) > libopen-pal.so.20 => /opt/Lib/openmpi-2.1.3/lib/libopen-pal.so.20 > (0x00002ac2b4884000) > libpciaccess.so.0 => /usr/lib64/libpciaccess.so.0 (0x00002ac2b4b86000) > librt.so.1 => /lib64/librt.so.1 (0x00000039f3c00000) > libutil.so.1 => /lib64/libutil.so.1 (0x00000039fca00000) > libintlc.so.5 => > /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libintlc.so.5 > (0x00002ac2b4d8d000) > libifcoremt.so.5 => > /opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64/libifcoremt.so.5 > (0x00002ac2b4eda000) > > > So as you can see that the libpciaccess already coupled to the compiled > program, but it's still error. So I know that libpciaccess maybe from the > MPI libs (I'm using OpenMPI), but this kind of error is really bugging me. > > Even after I tried to use --disable-hwloc-pci when installing OpenMPI, it > still got this kind of error,. > > What I want to know is: > - Is it possible to remove the dependenciness of libpciaccess? > - Or how to configure it so that PETSc or MPI doesn't use libpciaccess? > > In my case, I cannot use static because some of those dependencies doesn't > have static libraries. > > I'm using: > PETSc-3.9.0 > OpenMPI-2.1.3 > > > > > Aroli Marcellinus > > *Kumoh Institute of Technology**Computational Medicine Laboratory* > 61 Daehak-ro (Sinpyeong-dong), Gumi, Gyeongbuk > +82 10 9724 3957 > KTalk ID: vondarkness From danyang.su at gmail.com Fri Apr 27 10:29:22 2018 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 27 Apr 2018 08:29:22 -0700 Subject: [petsc-users] Get vertex index of each cell in DMPlex after distribution In-Reply-To: References: <3ee3963e-3fbe-9440-6754-861ef9271ae8@gmail.com> <979e9fba-286f-9414-2ad6-6e652019db4d@gmail.com> Message-ID: <63d7c860-21a5-3dfd-82b0-c101ee007608@gmail.com> On 2018-04-27 04:11 AM, Matthew Knepley wrote: > On Fri, Apr 27, 2018 at 2:09 AM, Danyang Su > wrote: > > Hi Matt, > > Sorry if this is a stupid question. > > In the previous code for unstructured grid, I create labels to > mark the original node/cell index from VTK file and then > distribute it so that each subdomain has a copy of its original > node and cell index, as well as the PETSc numbering. Now I am > trying to get avoid of using large number of keys in > DMSetLabelValue since this costs lot of time for large problem. > > I can get the coordinates of subdomain after distribution by using > DMGetCoordinatesLocal and DMGetCoordinateDM. > > How can I get the vertex index of each cell after distribution? > Would you please give me a hint or functions that I can use. > > You can permute the vectors back to the natural ordering using > > http://www.mcs.anl.gov/petsc/petsc-master/docs/manualpages/DMPLEX/DMPlexNaturalToGlobalBegin.html > > which says you have to call DMPlexSetUseNaturalSF() before > distributing the mesh. It is tested in > > ? src/dm/impls/plex/examples/tests/ex15.c > > so you can see how its intended to work. It is very new and has not > been tested by many people. > > I can see how you might want this for small tests. Why would you want > it for production models? Hi Matt, This is indeed what I need. As some of years old cases import initial conditions from external files, which are in natural ordering as the original mesh. Just want to make the code compatible to the old input files. Thanks, Danyang > > ? Thanks, > > ? ? Matt > > Thanks, > > Danyang > > > On 18-04-25 02:12 PM, Danyang Su wrote: >> On 2018-04-25 09:47 AM, Matthew Knepley wrote: >>> On Wed, Apr 25, 2018 at 12:40 PM, Danyang Su >>> > wrote: >>> >>> Hi Matthew, >>> >>> In the worst case, every node/cell may have different label. >>> >>> Do not use Label for this. Its not an appropriate thing. If >>> every cell is different, just use the cell number. >>> Labels are for mapping a relatively small number of keys (like >>> material IDs) to sets of points (cells, vertices, etc.) >>> Its not a great data structure for a permutation. >> Yes. If there is small number of keys, it runs very fast, even >> for more than one million DMSetLabelValue calls. The performance >> just deteriorates as the number of keys increases. >> >> I cannot get avoid of DMSetLabelValue as node/cell index of >> original mesh is needed for the previous input file that uses >> some of global node/cell index to set value. But if I can get the >> natural order of nodes/cells from DMPlex, I can discard the use >> of DMSetLabelValue. Is there any function can do this job? >> >> Thanks, >> >> Danyang >>> >>> However, I still do not believe these numbers. The old code does >>> a string comparison every time. I will setup a test. >>> >>> ? ?Matt >>> >>> Below is one of the worst scenario with 102299 nodes and >>> 102299 different labels for test. I found the time cost >>> increase during the loop. The first 9300 loop takes least >>> time (<0.5) while the last 9300 loops takes much more time >>> (>7.7), as shown below. If I use larger mesh with >1 million >>> nodes, it runs very very slowly in this part. The PETSc is >>> configured with optimization on. >>> >>> Configure options --with-cc=gcc --with-cxx=g++ >>> --with-fc=gfortran --download-mpich --download-scalapack >>> --download-parmetis --download-metis --download-ptscotch >>> --download-fblaslapack --download-hypre >>> --download-superlu_dist --download-hdf5=yes >>> --download-ctetgen --with-debugging=0 COPTFLAGS="-O3 >>> -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native >>> -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" >>> >>> istart iend progress CPU_Time time cost - old (sec) >>> time cost - new (sec) >>> 0 9299 0 1524670045.51166 >>> >>> 9300 18599 0.100010753 1524670045.99605 0.4843890667 >>> 0.497246027 >>> 18600 27899 0.200010747 1524670047.32635 1.330302 >>> 1.3820912838 >>> 27900 37199 0.300010741 1524670049.3066 1.9802515507 >>> 2.2439446449 >>> 37200 46499 0.400010765 1524670052.1594 2.852804184 >>> 3.0739262104 >>> 46500 55799 0.500010729 1524670055.90961 3.7502081394 >>> 3.9270553589 >>> 55800 65099 0.600010753 1524670060.47654 4.5669286251 >>> 4.7571902275 >>> 65100 74399 0.700010777 1524670066.0941 5.6175630093 >>> 5.7428796291 >>> 74400 83699 0.800010741 1524670072.53886 6.44475317 >>> 6.5761549473 >>> 83700 92998 0.900010765 1524670079.99072 7.4518604279 >>> 7.4606924057 >>> 92999 102298 1 1524670087.71066 7.7199423313 8.2424075603 >>> >>> >>> >>> old code >>> >>> ??????? do ipoint = 0, istart-1 >>> ????????? !c output time cost, use 1 processor to test >>> ????????? if (b_enable_output .and. rank == 0) then >>> ??????????? if (mod(ipoint,iprogress) == 0 .or. ipoint == >>> istart-1) then >>> !write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart >>> ????????????? write(*,*) ipoint, >>> (ipoint+1.0)/istart,"time",MPI_Wtime() >>> ??????????? end if >>> ????????? end if >>> >>> ????????? call DMSetLabelValue(dmda_flow%da,"cid_lg2g",ipoint, & >>> ipoint+1,ierr) >>> ????????? CHKERRQ(ierr) >>> ??????? end do >>> >>> >>> new code >>> >>> ??????? call DMCreateLabel(dmda_flow%da,'cid_lg2g',ierr) >>> ??????? CHKERRQ(ierr) >>> >>> ??????? call DMGetLabel(dmda_flow%da,'cid_lg2g',label, ierr) >>> ??????? CHKERRQ(ierr) >>> >>> ??????? do ipoint = 0, istart-1 >>> ????????? !c output time cost, use 1 processor to test >>> ????????? if (b_enable_output .and. rank == 0) then >>> ??????????? if (mod(ipoint,iprogress) == 0 .or. ipoint == >>> istart-1) then >>> !write(*,'(f3.1,1x)',advance="no") (ipoint+1.0)/istart >>> ????????????? write(*,*) ipoint, >>> (ipoint+1.0)/istart,"time",MPI_Wtime() >>> ??????????? end if >>> ????????? end if >>> >>> ????????? call DMLabelSetValue(label,ipoint,ipoint+1,ierr) >>> ????????? CHKERRQ(ierr) >>> ??????? end do >>> >>> Thanks, >>> >>> Danyang >>> >>> On 2018-04-25 03:16 AM, Matthew Knepley wrote: >>>> On Tue, Apr 24, 2018 at 11:57 PM, Danyang Su >>>> > wrote: >>>> >>>> Hi All, >>>> >>>> I use DMPlex in unstructured grid code and recently >>>> found DMSetLabelValue takes a lot of time for large >>>> problem, e.g., num. of cells > 1 million. In my code, I use >>>> >>>> >>>> I read your code wrong. For large loop, you should not use >>>> the convenience function. You should use >>>> >>>> DMPlexCreateFromCellList () >>>> >>>> >>>> DMGetLabel(dm, name, &label) >>>> >>>> >>>> Loop over all cells/nodes{ >>>> >>>> DMSetLabelValue >>>> >>>> >>>> Replace this by DMLabelSetValue(label, point, val) >>>> >>>> } >>>> >>>> DMPlexDistribute >>>> >>>> The code works fine except DMSetLabelValue takes a lot >>>> of time for large problem. I use DMSetLabelValue to set >>>> material id for all the nodes or cells so that each >>>> subdomain has a copy of material id. Is there any other >>>> functions that can be used more efficient, e.g. set >>>> labels by array, not 1 by 1? >>>> >>>> >>>> That should take much less time. >>>> >>>> ? Thanks, >>>> >>>> ? ? ?Matt >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin >>>> their experiments is infinitely more interesting than any >>>> results to which their experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to >>> which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedmud at gmail.com Fri Apr 27 11:58:29 2018 From: friedmud at gmail.com (Derek Gaston) Date: Fri, 27 Apr 2018 16:58:29 +0000 Subject: [petsc-users] Speed of MatGetRowSum Message-ID: I was wondering about the comment on MatGetRowSum that says: "This code is slow since it is not currently specialized for different formats" How slow are we talking about here? Is it actually going to loop over i,j from 0 to N on a sparse matrix? I need to get a vector that represents the "lumped" mass matrix: but I would like to get it fairly quickly. I would just compute the lumped matrix directly and then use MatGetDiag - but I actually want both (unlumped/consistent and lumped) and MatGetRowSum seemed like the best direction. Willing to take advice though! Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Fri Apr 27 12:01:06 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Fri, 27 Apr 2018 17:01:06 +0000 Subject: [petsc-users] Speed of MatGetRowSum In-Reply-To: References: Message-ID: If it's not specialized, as fast as one matrix vector multiplication (with all ones) Il Ven 27 Apr 2018, 19:58 Derek Gaston ha scritto: > I was wondering about the comment on MatGetRowSum that says: > > "This code is slow since it is not currently specialized for different > formats" > > How slow are we talking about here? Is it actually going to loop over i,j > from 0 to N on a sparse matrix? > > I need to get a vector that represents the "lumped" mass matrix: but I > would like to get it fairly quickly. I would just compute the lumped > matrix directly and then use MatGetDiag - but I actually want both > (unlumped/consistent and lumped) and MatGetRowSum seemed like the best > direction. Willing to take advice though! > > Derek > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 27 13:00:27 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 27 Apr 2018 18:00:27 +0000 Subject: [petsc-users] Speed of MatGetRowSum In-Reply-To: References: Message-ID: Here is the current code: ;-) PetscErrorCode MatGetRowSum(Mat mat, Vec v) { Vec ones; PetscErrorCode ierr; PetscFunctionBegin; PetscValidHeaderSpecific(mat,MAT_CLASSID,1); PetscValidType(mat,1); PetscValidHeaderSpecific(v,VEC_CLASSID,2); if (!mat->assembled) SETERRQ(PetscObjectComm((PetscObject)mat),PETSC_ERR_ARG_WRONGSTATE,"Not for unassembled matrix"); MatCheckPreallocated(mat,1); ierr = MatCreateVecs(mat,&ones,NULL);CHKERRQ(ierr); ierr = VecSet(ones,1.);CHKERRQ(ierr); ierr = MatMult(mat,ones,v);CHKERRQ(ierr); ierr = VecDestroy(&ones);CHKERRQ(ierr); PetscFunctionReturn(0); } An optimized for AIJ one would avoid creating the matrix and avoid the multiples by 1 in the summation and avoid the communication. I wouldn't worry about its performance; it won't be the bottle neck generally. Writing a custom routine for AIJ (seq and MPI) would be easy, just structure it like the MatMult_xxx() routine. Barry > On Apr 27, 2018, at 11:58 AM, Derek Gaston wrote: > > I was wondering about the comment on MatGetRowSum that says: > > "This code is slow since it is not currently specialized for different formats" > > How slow are we talking about here? Is it actually going to loop over i,j from 0 to N on a sparse matrix? > > I need to get a vector that represents the "lumped" mass matrix: but I would like to get it fairly quickly. I would just compute the lumped matrix directly and then use MatGetDiag - but I actually want both (unlumped/consistent and lumped) and MatGetRowSum seemed like the best direction. Willing to take advice though! > > Derek From friedmud at gmail.com Fri Apr 27 13:11:51 2018 From: friedmud at gmail.com (Derek Gaston) Date: Fri, 27 Apr 2018 18:11:51 +0000 Subject: [petsc-users] Speed of MatGetRowSum In-Reply-To: References: Message-ID: Thanks for your response! I suppose I should have just checked the code out myself :-) Makes sense - I won't worry about it then unless it shows up in profiling (which I doubt). I just wanted to make sure it wasn't an N^2 operation (which it's not since my matrices are very sparse). Thanks again, Derek On Fri, Apr 27, 2018 at 12:00 PM Smith, Barry F. wrote: > > Here is the current code: ;-) > > PetscErrorCode MatGetRowSum(Mat mat, Vec v) > { > Vec ones; > PetscErrorCode ierr; > > PetscFunctionBegin; > PetscValidHeaderSpecific(mat,MAT_CLASSID,1); > PetscValidType(mat,1); > PetscValidHeaderSpecific(v,VEC_CLASSID,2); > if (!mat->assembled) > SETERRQ(PetscObjectComm((PetscObject)mat),PETSC_ERR_ARG_WRONGSTATE,"Not for > unassembled matrix"); > MatCheckPreallocated(mat,1); > ierr = MatCreateVecs(mat,&ones,NULL);CHKERRQ(ierr); > ierr = VecSet(ones,1.);CHKERRQ(ierr); > ierr = MatMult(mat,ones,v);CHKERRQ(ierr); > ierr = VecDestroy(&ones);CHKERRQ(ierr); > PetscFunctionReturn(0); > } > > An optimized for AIJ one would avoid creating the matrix and avoid the > multiples by 1 in the summation and avoid the communication. > > I wouldn't worry about its performance; it won't be the bottle neck > generally. Writing a custom routine for AIJ (seq and MPI) would be easy, > just structure it like the MatMult_xxx() routine. > > Barry > > > > On Apr 27, 2018, at 11:58 AM, Derek Gaston wrote: > > > > I was wondering about the comment on MatGetRowSum that says: > > > > "This code is slow since it is not currently specialized for different > formats" > > > > How slow are we talking about here? Is it actually going to loop over > i,j from 0 to N on a sparse matrix? > > > > I need to get a vector that represents the "lumped" mass matrix: but I > would like to get it fairly quickly. I would just compute the lumped > matrix directly and then use MatGetDiag - but I actually want both > (unlumped/consistent and lumped) and MatGetRowSum seemed like the best > direction. Willing to take advice though! > > > > Derek > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Sat Apr 28 01:08:14 2018 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 27 Apr 2018 23:08:14 -0700 Subject: [petsc-users] Segmentation Violation in getting DMPlex coordinates Message-ID: <41a039f4-c4a1-ced7-f121-f25d539652f8@gmail.com> Hi All, I use DMPlex and need to get coordinates back after distribution. However, I always get segmentation violation in getting coords values in the following codes if using multiple processors. If only one processor is used, it works fine. For each processors, the off value starts from 0 which looks good. I also tried 0-based index, which gives the same error. Would any one help to check what is wrong here? idof 1 off 0 idof 2 off 0 idof 1 off 2 idof 2 off 2 idof 1 off 4 idof 2 off 4 idof 1 off 6 idof 2 off 6 idof 1 off 8 idof 2 off 8 DM :: distributedMesh, cda Vec :: gc PetscScalar, pointer :: coords(:) PetscSection :: cs ... call DMGetCoordinatesLocal(dmda_flow%da,gc,ierr) CHKERRQ(ierr) call DMGetCoordinateDM(dmda_flow%da,cda,ierr) CHKERRQ(ierr) call DMGetDefaultSection(cda,cs,ierr) CHKERRQ(ierr) call PetscSectionGetChart(cs,istart,iend,ierr) CHKERRQ(ierr) !c get coordinates array call DMDAVecGetArrayF90(cda,gc,coords,ierr) CHKERRQ(ierr) do ipoint = istart, iend-1 call PetscSectionGetDof(cs,ipoint,dof,ierr) CHKERRQ(ierr) call PetscSectionGetOffset(cs,ipoint,off,ierr) CHKERRQ(ierr) inode = ipoint-istart+1 if (cell_coords == coords_xyz) then nodes(inode)%x = coords(off+1) nodes(inode)%y = coords(off+2) nodes(inode)%z = coords(off+3) else if (cell_coords == coords_xy) then nodes(inode)%x = coords(off+1) nodes(inode)%y = coords(off+2) nodes(inode)%z = 0.0d0 else if (cell_coords == coords_yz) then nodes(inode)%x = 0.0d0 nodes(inode)%y = coords(off+1) nodes(inode)%z = coords(off+2) else if (cell_coords ==coords_xz) then nodes(inode)%x = coords(off+1) nodes(inode)%y = 0.0d0 nodes(inode)%z = coords(off+2) end if end do call DMDAVecRestoreArrayF90(cda,gc,coords,ierr) CHKERRQ(ierr) Thanks, Danyang From knepley at gmail.com Sat Apr 28 09:19:23 2018 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 28 Apr 2018 10:19:23 -0400 Subject: [petsc-users] Segmentation Violation in getting DMPlex coordinates In-Reply-To: <41a039f4-c4a1-ced7-f121-f25d539652f8@gmail.com> References: <41a039f4-c4a1-ced7-f121-f25d539652f8@gmail.com> Message-ID: On Sat, Apr 28, 2018 at 2:08 AM, Danyang Su wrote: > Hi All, > > I use DMPlex and need to get coordinates back after distribution. However, > I always get segmentation violation in getting coords values in the > following codes if using multiple processors. If only one processor is > used, it works fine. > > For each processors, the off value starts from 0 which looks good. I also > tried 0-based index, which gives the same error. Would any one help to > check what is wrong here? > > idof 1 off 0 > idof 2 off 0 > idof 1 off 2 > idof 2 off 2 > idof 1 off 4 > idof 2 off 4 > idof 1 off 6 > idof 2 off 6 > idof 1 off 8 > idof 2 off 8 > > > DM :: distributedMesh, cda > Vec :: gc > PetscScalar, pointer :: coords(:) > PetscSection :: cs > > ... > > call DMGetCoordinatesLocal(dmda_flow%da,gc,ierr) > CHKERRQ(ierr) > > call DMGetCoordinateDM(dmda_flow%da,cda,ierr) > CHKERRQ(ierr) > > call DMGetDefaultSection(cda,cs,ierr) > CHKERRQ(ierr) > > call PetscSectionGetChart(cs,istart,iend,ierr) > CHKERRQ(ierr) > > !c get coordinates array > call DMDAVecGetArrayF90(cda,gc,coords,ierr) > You cannot call DMDA function if you have a DMPlex. You jsut call VecGetArrayF90() Matt > CHKERRQ(ierr) > > do ipoint = istart, iend-1 > > call PetscSectionGetDof(cs,ipoint,dof,ierr) > CHKERRQ(ierr) > > call PetscSectionGetOffset(cs,ipoint,off,ierr) > CHKERRQ(ierr) > > inode = ipoint-istart+1 > > if (cell_coords == coords_xyz) then > nodes(inode)%x = coords(off+1) > nodes(inode)%y = coords(off+2) > nodes(inode)%z = coords(off+3) > else if (cell_coords == coords_xy) then > nodes(inode)%x = coords(off+1) > nodes(inode)%y = coords(off+2) > nodes(inode)%z = 0.0d0 > else if (cell_coords == coords_yz) then > nodes(inode)%x = 0.0d0 > nodes(inode)%y = coords(off+1) > nodes(inode)%z = coords(off+2) > else if (cell_coords ==coords_xz) then > nodes(inode)%x = coords(off+1) > nodes(inode)%y = 0.0d0 > nodes(inode)%z = coords(off+2) > end if > end do > > call DMDAVecRestoreArrayF90(cda,gc,coords,ierr) > CHKERRQ(ierr) > > Thanks, > > Danyang > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Apr 28 15:38:42 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sat, 28 Apr 2018 20:38:42 +0000 Subject: [petsc-users] Segmentation Violation in getting DMPlex coordinates In-Reply-To: References: <41a039f4-c4a1-ced7-f121-f25d539652f8@gmail.com> Message-ID: <064B39E5-2AFB-4167-BBF6-58CB200F0F76@anl.gov> Added runtime error checking for such incorrect calls in barry/dmda-calls-type-check > On Apr 28, 2018, at 9:19 AM, Matthew Knepley wrote: > > On Sat, Apr 28, 2018 at 2:08 AM, Danyang Su wrote: > Hi All, > > I use DMPlex and need to get coordinates back after distribution. However, I always get segmentation violation in getting coords values in the following codes if using multiple processors. If only one processor is used, it works fine. > > For each processors, the off value starts from 0 which looks good. I also tried 0-based index, which gives the same error. Would any one help to check what is wrong here? > > idof 1 off 0 > idof 2 off 0 > idof 1 off 2 > idof 2 off 2 > idof 1 off 4 > idof 2 off 4 > idof 1 off 6 > idof 2 off 6 > idof 1 off 8 > idof 2 off 8 > > > DM :: distributedMesh, cda > Vec :: gc > PetscScalar, pointer :: coords(:) > PetscSection :: cs > > ... > > call DMGetCoordinatesLocal(dmda_flow%da,gc,ierr) > CHKERRQ(ierr) > > call DMGetCoordinateDM(dmda_flow%da,cda,ierr) > CHKERRQ(ierr) > > call DMGetDefaultSection(cda,cs,ierr) > CHKERRQ(ierr) > > call PetscSectionGetChart(cs,istart,iend,ierr) > CHKERRQ(ierr) > > !c get coordinates array > call DMDAVecGetArrayF90(cda,gc,coords,ierr) > > You cannot call DMDA function if you have a DMPlex. You jsut call VecGetArrayF90() > > Matt > > CHKERRQ(ierr) > > do ipoint = istart, iend-1 > > call PetscSectionGetDof(cs,ipoint,dof,ierr) > CHKERRQ(ierr) > > call PetscSectionGetOffset(cs,ipoint,off,ierr) > CHKERRQ(ierr) > > inode = ipoint-istart+1 > > if (cell_coords == coords_xyz) then > nodes(inode)%x = coords(off+1) > nodes(inode)%y = coords(off+2) > nodes(inode)%z = coords(off+3) > else if (cell_coords == coords_xy) then > nodes(inode)%x = coords(off+1) > nodes(inode)%y = coords(off+2) > nodes(inode)%z = 0.0d0 > else if (cell_coords == coords_yz) then > nodes(inode)%x = 0.0d0 > nodes(inode)%y = coords(off+1) > nodes(inode)%z = coords(off+2) > else if (cell_coords ==coords_xz) then > nodes(inode)%x = coords(off+1) > nodes(inode)%y = 0.0d0 > nodes(inode)%z = coords(off+2) > end if > end do > > call DMDAVecRestoreArrayF90(cda,gc,coords,ierr) > CHKERRQ(ierr) > > Thanks, > > Danyang > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From danyang.su at gmail.com Sat Apr 28 17:27:11 2018 From: danyang.su at gmail.com (Danyang Su) Date: Sat, 28 Apr 2018 15:27:11 -0700 Subject: [petsc-users] Segmentation Violation in getting DMPlex coordinates In-Reply-To: <064B39E5-2AFB-4167-BBF6-58CB200F0F76@anl.gov> References: <41a039f4-c4a1-ced7-f121-f25d539652f8@gmail.com> <064B39E5-2AFB-4167-BBF6-58CB200F0F76@anl.gov> Message-ID: <0caf25b3-91f4-e203-3489-6509ddab349e@gmail.com> Hi Matt and Barry, Thanks for your quick response. After changing DMDAVecGetArrayF90 to VecGetArrayF90, everything works now. Thanks, Danyang On 18-04-28 01:38 PM, Smith, Barry F. wrote: > Added runtime error checking for such incorrect calls in barry/dmda-calls-type-check > > >> On Apr 28, 2018, at 9:19 AM, Matthew Knepley wrote: >> >> On Sat, Apr 28, 2018 at 2:08 AM, Danyang Su wrote: >> Hi All, >> >> I use DMPlex and need to get coordinates back after distribution. However, I always get segmentation violation in getting coords values in the following codes if using multiple processors. If only one processor is used, it works fine. >> >> For each processors, the off value starts from 0 which looks good. I also tried 0-based index, which gives the same error. Would any one help to check what is wrong here? >> >> idof 1 off 0 >> idof 2 off 0 >> idof 1 off 2 >> idof 2 off 2 >> idof 1 off 4 >> idof 2 off 4 >> idof 1 off 6 >> idof 2 off 6 >> idof 1 off 8 >> idof 2 off 8 >> >> >> DM :: distributedMesh, cda >> Vec :: gc >> PetscScalar, pointer :: coords(:) >> PetscSection :: cs >> >> ... >> >> call DMGetCoordinatesLocal(dmda_flow%da,gc,ierr) >> CHKERRQ(ierr) >> >> call DMGetCoordinateDM(dmda_flow%da,cda,ierr) >> CHKERRQ(ierr) >> >> call DMGetDefaultSection(cda,cs,ierr) >> CHKERRQ(ierr) >> >> call PetscSectionGetChart(cs,istart,iend,ierr) >> CHKERRQ(ierr) >> >> !c get coordinates array >> call DMDAVecGetArrayF90(cda,gc,coords,ierr) >> >> You cannot call DMDA function if you have a DMPlex. You jsut call VecGetArrayF90() >> >> Matt >> >> CHKERRQ(ierr) >> >> do ipoint = istart, iend-1 >> >> call PetscSectionGetDof(cs,ipoint,dof,ierr) >> CHKERRQ(ierr) >> >> call PetscSectionGetOffset(cs,ipoint,off,ierr) >> CHKERRQ(ierr) >> >> inode = ipoint-istart+1 >> >> if (cell_coords == coords_xyz) then >> nodes(inode)%x = coords(off+1) >> nodes(inode)%y = coords(off+2) >> nodes(inode)%z = coords(off+3) >> else if (cell_coords == coords_xy) then >> nodes(inode)%x = coords(off+1) >> nodes(inode)%y = coords(off+2) >> nodes(inode)%z = 0.0d0 >> else if (cell_coords == coords_yz) then >> nodes(inode)%x = 0.0d0 >> nodes(inode)%y = coords(off+1) >> nodes(inode)%z = coords(off+2) >> else if (cell_coords ==coords_xz) then >> nodes(inode)%x = coords(off+1) >> nodes(inode)%y = 0.0d0 >> nodes(inode)%z = coords(off+2) >> end if >> end do >> >> call DMDAVecRestoreArrayF90(cda,gc,coords,ierr) >> CHKERRQ(ierr) >> >> Thanks, >> >> Danyang >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ From olk548 at mail.usask.ca Sat Apr 28 20:05:12 2018 From: olk548 at mail.usask.ca (Oleksandr Koshkarov) Date: Sat, 28 Apr 2018 19:05:12 -0600 Subject: [petsc-users] Finite difference Jacobian in TS context Message-ID: Hello All, I hope someone can help :) I think I am doing something wrong, but cannot understand what. I have a huge time dependent system with 3d DMDA data structure and I am evolving it with explicit Runge-Kutta by using TS and basically only using "TSSetRHSFunction". Now I want to repeat it with implicit time stepper (for now Crank-Nicolson) and I am trying to provide finite difference Jacobian and I am failing miserably. I also cannot find appropriate example in PETSc tutorial (if you can point me to working example, it would be great). Here is my best attempt (what wrong with it?): ? DMDACreate3d(PETSC_COMM_WORLD, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, ??? ?????? DMDA_STENCIL_STAR, ??? ?????? NX, NY, NZ, ??? ?????? PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, ??? ?????? 2*3+NC*NS, ??? ?????? 1, ??? ?????? NULL,? NULL, NULL,? &da); ? DMSetUp(da); ? DMCreateGlobalVector(da,&x); ? TSCreate(PETSC_COMM_WORLD,&ts); ? TSSetProblemType(ts,TS_NONLINEAR); ? TSSetRHSFunction(ts,NULL,compute_RHS,NULL); ? TSSetMaxTime(ts,T_FINAL); ? TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER); ? TSSetDM(ts,da); ? TSSetType(ts,TSCN); //it works with: TSSetType(ts,TSRK); ? set_IC(da,x); ? TSSetTimeStep(ts,DT); ? TSSetSolution(ts,x); ? TSGetSNES(ts,&snes); ? SNESGetKSP(snes,&ksp); ? KSPGetPC(ksp,&pc); ? PCSetType(pc,PCNONE); ? DMSetMatType(da,MATAIJ); ? DMCreateMatrix(da,&J); ? ISColoring iscoloring; ? MatFDColoring? matfdcoloring; ? DMCreateColoring(da,IS_COLORING_GLOBAL,&iscoloring); ? MatFDColoringCreate(J,iscoloring,&matfdcoloring); ? MatFDColoringSetType(matfdcoloring,MATMFFD_DS); // I think I do something wrong in the following 3 lines ? PetscErrorCode (*temp_f)(SNES,Vec,Vec,void*); ? SNESGetFunction(snes,NULL,&temp_f,NULL); ? MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode (*)(void))temp_f,NULL); ? MatFDColoringSetUp(J,iscoloring,matfdcoloring); SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,matfdcoloring); ? ISColoringDestroy(&iscoloring); ? TSSolve(ts,x); Thank you, Oleksandr Koshkarov. From bsmith at mcs.anl.gov Sat Apr 28 20:20:27 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 29 Apr 2018 01:20:27 +0000 Subject: [petsc-users] Finite difference Jacobian in TS context In-Reply-To: References: Message-ID: <72B587E7-3100-44F6-946C-2203D4AAD627@anl.gov> ~/Src/petsc/src/ts/examples/tutorials $ grep SNESComputeJacobianDefaultColor *.c ex10.c: ierr = SNESSetJacobian(snes,A,B,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); ex15.c: ierr = SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); ex17.c: ierr = SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); I don't think you need to explicitly create the MatFDColoring object. Please take a look at ex15.c and see if you can get it working like that example. If that doesn't work let us know and we can take a closer look at it. Barry > On Apr 28, 2018, at 8:05 PM, Oleksandr Koshkarov wrote: > > Hello All, > > I hope someone can help :) I think I am doing something wrong, but cannot understand what. I have a huge time dependent system with 3d DMDA data structure and I am evolving it with explicit Runge-Kutta by using TS and basically only using "TSSetRHSFunction". Now I want to repeat it with implicit time stepper (for now Crank-Nicolson) and I am trying to provide finite difference Jacobian and I am failing miserably. I also cannot find appropriate example in PETSc tutorial (if you can point me to working example, it would be great). > > Here is my best attempt (what wrong with it?): > > DMDACreate3d(PETSC_COMM_WORLD, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, > DMDA_STENCIL_STAR, > NX, NY, NZ, > PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, > 2*3+NC*NS, > 1, > NULL, NULL, NULL, &da); > DMSetUp(da); > DMCreateGlobalVector(da,&x); > TSCreate(PETSC_COMM_WORLD,&ts); > TSSetProblemType(ts,TS_NONLINEAR); > TSSetRHSFunction(ts,NULL,compute_RHS,NULL); > TSSetMaxTime(ts,T_FINAL); > TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER); > TSSetDM(ts,da); > TSSetType(ts,TSCN); //it works with: TSSetType(ts,TSRK); > set_IC(da,x); > TSSetTimeStep(ts,DT); > TSSetSolution(ts,x); > > TSGetSNES(ts,&snes); > SNESGetKSP(snes,&ksp); > KSPGetPC(ksp,&pc); > PCSetType(pc,PCNONE); > > DMSetMatType(da,MATAIJ); > DMCreateMatrix(da,&J); > ISColoring iscoloring; > MatFDColoring matfdcoloring; > DMCreateColoring(da,IS_COLORING_GLOBAL,&iscoloring); > MatFDColoringCreate(J,iscoloring,&matfdcoloring); > > MatFDColoringSetType(matfdcoloring,MATMFFD_DS); > > // I think I do something wrong in the following 3 lines > > PetscErrorCode (*temp_f)(SNES,Vec,Vec,void*); > SNESGetFunction(snes,NULL,&temp_f,NULL); > MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode (*)(void))temp_f,NULL); > > MatFDColoringSetUp(J,iscoloring,matfdcoloring); > > SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,matfdcoloring); > ISColoringDestroy(&iscoloring); > > TSSolve(ts,x); > > Thank you, > > Oleksandr Koshkarov. > From mfadams at lbl.gov Sun Apr 29 10:38:59 2018 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 29 Apr 2018 11:38:59 -0400 Subject: [petsc-users] config error with hypre on Summit Message-ID: I'm getting an error configure hypre on the new IBM at ORNL, Summit. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 3392578 bytes Desc: not available URL: From knepley at gmail.com Sun Apr 29 11:24:02 2018 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 29 Apr 2018 12:24:02 -0400 Subject: [petsc-users] config error with hypre on Summit In-Reply-To: References: Message-ID: On Sun, Apr 29, 2018 at 11:38 AM, Mark Adams wrote: > I'm getting an error configure hypre on the new IBM at ORNL, Summit. > I can spot your error right away. You are trying to build Hypre ;) It looks like autoconf is failing for this system, since the config.guess script is old. This is one for the sysadmin I think. Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedmud at gmail.com Sun Apr 29 12:24:05 2018 From: friedmud at gmail.com (Derek Gaston) Date: Sun, 29 Apr 2018 17:24:05 +0000 Subject: [petsc-users] Initial Guess for KSP in SNES Message-ID: I'm interested in setting the initial guess for the first KSP solve in SNES (used in a transient calculation) - and whether anyone thinks it could be a good thing to do. I see some previous discussion on this from Jed and Matt here: https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-September/022779.html https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-September/022780.html I realize that the Newton solver is solving for the update... but that doesn't necessarily mean that guessing 0 is the best plan. For instance: for fairly linear problems in time (or slightly nonlinear depending on how you look at it!) it might be a good idea to guess the first update to be the difference between the previous two time steps (or: more generally some sort of "projected" update based on your time integration scheme). Indeed - for a perfectly linear problem in time (with constant dt) this would yield a "perfect" guess for the linear solver and it would return immediately. I suppose that if you use a "predictor" for predicting the guess for the nonlinear system (which we often do - but not always)... then 0 for your first update is probably a decent choice (what else would you choose?). But if you're simply using the old timestep solution (which is what is commonly done)... then that first update is more likely to look like the difference between the last two timesteps than it looks like 0. Thoughts? Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sun Apr 29 12:54:22 2018 From: jed at jedbrown.org (Jed Brown) Date: Sun, 29 Apr 2018 11:54:22 -0600 Subject: [petsc-users] Initial Guess for KSP in SNES In-Reply-To: References: Message-ID: <87bme1x5ip.fsf@jedbrown.org> Why do you want it to be an initial guess for the linear problem rather than for the nonlinear problem? I think you can use SNESSetUpdate() and in that function, SNESGetSolutionUpdate() which you can set to whatever you want the initial guess to be. Derek Gaston writes: > I'm interested in setting the initial guess for the first KSP solve in SNES > (used in a transient calculation) - and whether anyone thinks it could be a > good thing to do. > > I see some previous discussion on this from Jed and Matt here: > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-September/022779.html > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-September/022780.html > > I realize that the Newton solver is solving for the update... but that > doesn't necessarily mean that guessing 0 is the best plan. For instance: > for fairly linear problems in time (or slightly nonlinear depending on how > you look at it!) it might be a good idea to guess the first update to be > the difference between the previous two time steps (or: more generally some > sort of "projected" update based on your time integration scheme). > > Indeed - for a perfectly linear problem in time (with constant dt) this > would yield a "perfect" guess for the linear solver and it would return > immediately. > > I suppose that if you use a "predictor" for predicting the guess for the > nonlinear system (which we often do - but not always)... then 0 for your > first update is probably a decent choice (what else would you choose?). > But if you're simply using the old timestep solution (which is what is > commonly done)... then that first update is more likely to look like the > difference between the last two timesteps than it looks like 0. > > Thoughts? > > Derek From friedmud at gmail.com Sun Apr 29 13:14:15 2018 From: friedmud at gmail.com (Derek Gaston) Date: Sun, 29 Apr 2018 18:14:15 +0000 Subject: [petsc-users] Initial Guess for KSP in SNES In-Reply-To: <87bme1x5ip.fsf@jedbrown.org> References: <87bme1x5ip.fsf@jedbrown.org> Message-ID: I suppose that's a good point. This came about because I happen to be solving linear problems right now - and I'm just using KSP - but I'm still solving for an "update" as in a Newton method. So, naturally, I wanted to "predict" the update of that linear problem each timestep. That got me thinking about what we generally choose as the initial guess for the KSP in SNES so I went looking. I think the moral of the story here is, as per usual, making good guesses for the solution of the nonlinear solve is a good idea. It obviously helps Newton and it makes the initial guess of 0 for the solution of the update also make more sense. Also: I hadn't really considered the effect of an initial guess on a Krylov solver before. It's intuitively obvious that providing a good initial guess helps Newton (for instance, guessing within the ball of convergence)... but I haven't seen much work on how the initial guess effects Krylov solvers before. A bit of googling led me to this paper which is interesting: http://dodo.inm.ras.ru/vassilevski/wp-content/uploads/2015/01/jcp219_210-227.pdf do you know of another reference that is related? Thanks for the response, Derek On Sun, Apr 29, 2018 at 11:54 AM Jed Brown wrote: > Why do you want it to be an initial guess for the linear problem rather > than for the nonlinear problem? > > I think you can use SNESSetUpdate() and in that function, > SNESGetSolutionUpdate() which you can set to whatever you want the > initial guess to be. > > Derek Gaston writes: > > > I'm interested in setting the initial guess for the first KSP solve in > SNES > > (used in a transient calculation) - and whether anyone thinks it could > be a > > good thing to do. > > > > I see some previous discussion on this from Jed and Matt here: > > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-September/022779.html > > > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-September/022780.html > > > > I realize that the Newton solver is solving for the update... but that > > doesn't necessarily mean that guessing 0 is the best plan. For instance: > > for fairly linear problems in time (or slightly nonlinear depending on > how > > you look at it!) it might be a good idea to guess the first update to be > > the difference between the previous two time steps (or: more generally > some > > sort of "projected" update based on your time integration scheme). > > > > Indeed - for a perfectly linear problem in time (with constant dt) this > > would yield a "perfect" guess for the linear solver and it would return > > immediately. > > > > I suppose that if you use a "predictor" for predicting the guess for the > > nonlinear system (which we often do - but not always)... then 0 for your > > first update is probably a decent choice (what else would you choose?). > > But if you're simply using the old timestep solution (which is what is > > commonly done)... then that first update is more likely to look like the > > difference between the last two timesteps than it looks like 0. > > > > Thoughts? > > > > Derek > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Sun Apr 29 13:18:05 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Sun, 29 Apr 2018 18:18:05 +0000 Subject: [petsc-users] Initial Guess for KSP in SNES In-Reply-To: References: <87bme1x5ip.fsf@jedbrown.org> Message-ID: For linear time dependent problems you can try KSPGUESSFISCHER and KSPGUESSPOD, that project the linear system on a lower dimensional manifold. Note that there are also integrators, like bdf, that already computed a fairly good initial guess for Newton. Il Dom 29 Apr 2018, 21:14 Derek Gaston ha scritto: > I suppose that's a good point. > > This came about because I happen to be solving linear problems right now - > and I'm just using KSP - but I'm still solving for an "update" as in a > Newton method. So, naturally, I wanted to "predict" the update of that > linear problem each timestep. That got me thinking about what we generally > choose as the initial guess for the KSP in SNES so I went looking. > > I think the moral of the story here is, as per usual, making good guesses > for the solution of the nonlinear solve is a good idea. It obviously helps > Newton and it makes the initial guess of 0 for the solution of the update > also make more sense. > > Also: I hadn't really considered the effect of an initial guess on a > Krylov solver before. It's intuitively obvious that providing a good > initial guess helps Newton (for instance, guessing within the ball of > convergence)... but I haven't seen much work on how the initial guess > effects Krylov solvers before. A bit of googling led me to this paper > which is interesting: > http://dodo.inm.ras.ru/vassilevski/wp-content/uploads/2015/01/jcp219_210-227.pdf do > you know of another reference that is related? > > Thanks for the response, > Derek > > On Sun, Apr 29, 2018 at 11:54 AM Jed Brown wrote: > >> Why do you want it to be an initial guess for the linear problem rather >> than for the nonlinear problem? >> >> I think you can use SNESSetUpdate() and in that function, >> SNESGetSolutionUpdate() which you can set to whatever you want the >> initial guess to be. >> >> Derek Gaston writes: >> >> > I'm interested in setting the initial guess for the first KSP solve in >> SNES >> > (used in a transient calculation) - and whether anyone thinks it could >> be a >> > good thing to do. >> > >> > I see some previous discussion on this from Jed and Matt here: >> > >> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-September/022779.html >> > >> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-September/022780.html >> > >> > I realize that the Newton solver is solving for the update... but that >> > doesn't necessarily mean that guessing 0 is the best plan. For >> instance: >> > for fairly linear problems in time (or slightly nonlinear depending on >> how >> > you look at it!) it might be a good idea to guess the first update to be >> > the difference between the previous two time steps (or: more generally >> some >> > sort of "projected" update based on your time integration scheme). >> > >> > Indeed - for a perfectly linear problem in time (with constant dt) this >> > would yield a "perfect" guess for the linear solver and it would return >> > immediately. >> > >> > I suppose that if you use a "predictor" for predicting the guess for the >> > nonlinear system (which we often do - but not always)... then 0 for your >> > first update is probably a decent choice (what else would you choose?). >> > But if you're simply using the old timestep solution (which is what is >> > commonly done)... then that first update is more likely to look like the >> > difference between the last two timesteps than it looks like 0. >> > >> > Thoughts? >> > >> > Derek >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedmud at gmail.com Sun Apr 29 13:26:09 2018 From: friedmud at gmail.com (Derek Gaston) Date: Sun, 29 Apr 2018 18:26:09 +0000 Subject: [petsc-users] Initial Guess for KSP in SNES In-Reply-To: References: <87bme1x5ip.fsf@jedbrown.org> Message-ID: Nice! Thanks Stefano - I'll check those out. I actually don't solve linear problems very often so I don't know about all the technology available :-) Derek On Sun, Apr 29, 2018 at 12:18 PM Stefano Zampini wrote: > For linear time dependent problems you can try KSPGUESSFISCHER and > KSPGUESSPOD, that project the linear system on a lower dimensional > manifold. Note that there are also integrators, like bdf, that already > computed a fairly good initial guess for Newton. > > Il Dom 29 Apr 2018, 21:14 Derek Gaston ha scritto: > >> I suppose that's a good point. >> >> This came about because I happen to be solving linear problems right now >> - and I'm just using KSP - but I'm still solving for an "update" as in a >> Newton method. So, naturally, I wanted to "predict" the update of that >> linear problem each timestep. That got me thinking about what we generally >> choose as the initial guess for the KSP in SNES so I went looking. >> >> I think the moral of the story here is, as per usual, making good guesses >> for the solution of the nonlinear solve is a good idea. It obviously helps >> Newton and it makes the initial guess of 0 for the solution of the update >> also make more sense. >> >> Also: I hadn't really considered the effect of an initial guess on a >> Krylov solver before. It's intuitively obvious that providing a good >> initial guess helps Newton (for instance, guessing within the ball of >> convergence)... but I haven't seen much work on how the initial guess >> effects Krylov solvers before. A bit of googling led me to this paper >> which is interesting: >> http://dodo.inm.ras.ru/vassilevski/wp-content/uploads/2015/01/jcp219_210-227.pdf do >> you know of another reference that is related? >> >> Thanks for the response, >> Derek >> >> On Sun, Apr 29, 2018 at 11:54 AM Jed Brown wrote: >> >>> Why do you want it to be an initial guess for the linear problem rather >>> than for the nonlinear problem? >>> >>> I think you can use SNESSetUpdate() and in that function, >>> SNESGetSolutionUpdate() which you can set to whatever you want the >>> initial guess to be. >>> >>> Derek Gaston writes: >>> >>> > I'm interested in setting the initial guess for the first KSP solve in >>> SNES >>> > (used in a transient calculation) - and whether anyone thinks it could >>> be a >>> > good thing to do. >>> > >>> > I see some previous discussion on this from Jed and Matt here: >>> > >>> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-September/022779.html >>> > >>> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-September/022780.html >>> > >>> > I realize that the Newton solver is solving for the update... but that >>> > doesn't necessarily mean that guessing 0 is the best plan. For >>> instance: >>> > for fairly linear problems in time (or slightly nonlinear depending on >>> how >>> > you look at it!) it might be a good idea to guess the first update to >>> be >>> > the difference between the previous two time steps (or: more generally >>> some >>> > sort of "projected" update based on your time integration scheme). >>> > >>> > Indeed - for a perfectly linear problem in time (with constant dt) this >>> > would yield a "perfect" guess for the linear solver and it would return >>> > immediately. >>> > >>> > I suppose that if you use a "predictor" for predicting the guess for >>> the >>> > nonlinear system (which we often do - but not always)... then 0 for >>> your >>> > first update is probably a decent choice (what else would you choose?). >>> > But if you're simply using the old timestep solution (which is what is >>> > commonly done)... then that first update is more likely to look like >>> the >>> > difference between the last two timesteps than it looks like 0. >>> > >>> > Thoughts? >>> > >>> > Derek >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtmills at anl.gov Sun Apr 29 13:46:01 2018 From: rtmills at anl.gov (Richard Tran Mills) Date: Sun, 29 Apr 2018 13:46:01 -0500 Subject: [petsc-users] config error with hypre on Summit In-Reply-To: References: Message-ID: Mark, I asked OLCF if I could get access to Summit so that I could ensure that the latest PETSc builds and works on there and was told "NO". If you are associated with an allocation on there, though, then I could get on and poke around if the PI is willing to approve me on their project. (I already have an OLCF account, just no access to pre-production Summit as I'm not associated with an allocation on there.) Should we ask the PI of your allocation to get me an account on there? Pat: I see you cc'ed on this message thread. You are doing a terrible job of being "retired"! =) Cheers, Richard On Sun, Apr 29, 2018 at 11:24 AM, Matthew Knepley wrote: > On Sun, Apr 29, 2018 at 11:38 AM, Mark Adams wrote: > >> I'm getting an error configure hypre on the new IBM at ORNL, Summit. >> > > I can spot your error right away. You are trying to build Hypre ;) > > It looks like autoconf is failing for this system, since the config.guess > script is old. This is > one for the sysadmin I think. > > Thanks, > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olk548 at mail.usask.ca Sun Apr 29 14:56:22 2018 From: olk548 at mail.usask.ca (Oleksandr Koshkarov) Date: Sun, 29 Apr 2018 13:56:22 -0600 Subject: [petsc-users] Finite difference Jacobian in TS context In-Reply-To: <72B587E7-3100-44F6-946C-2203D4AAD627@anl.gov> References: <72B587E7-3100-44F6-946C-2203D4AAD627@anl.gov> Message-ID: Hello all, Sorry for stupid question, but do "SNESComputeJacobianDefault" or "SNESComputeJacobianDefaultColor" make Jacobian matrix free? (I was thinking so from the start... but now it seems I am totally wrong) Best regards, Oleksandr. On 04/28/2018 07:20 PM, Smith, Barry F. wrote: > ~/Src/petsc/src/ts/examples/tutorials > $ grep SNESComputeJacobianDefaultColor *.c > ex10.c: ierr = SNESSetJacobian(snes,A,B,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); > ex15.c: ierr = SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); > ex17.c: ierr = SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); > > I don't think you need to explicitly create the MatFDColoring object. > > Please take a look at ex15.c and see if you can get it working like that example. If that doesn't work let us know and we can take a closer look at it. > > > Barry > > >> On Apr 28, 2018, at 8:05 PM, Oleksandr Koshkarov wrote: >> >> Hello All, >> >> I hope someone can help :) I think I am doing something wrong, but cannot understand what. I have a huge time dependent system with 3d DMDA data structure and I am evolving it with explicit Runge-Kutta by using TS and basically only using "TSSetRHSFunction". Now I want to repeat it with implicit time stepper (for now Crank-Nicolson) and I am trying to provide finite difference Jacobian and I am failing miserably. I also cannot find appropriate example in PETSc tutorial (if you can point me to working example, it would be great). >> >> Here is my best attempt (what wrong with it?): >> >> DMDACreate3d(PETSC_COMM_WORLD, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, >> DMDA_STENCIL_STAR, >> NX, NY, NZ, >> PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, >> 2*3+NC*NS, >> 1, >> NULL, NULL, NULL, &da); >> DMSetUp(da); >> DMCreateGlobalVector(da,&x); >> TSCreate(PETSC_COMM_WORLD,&ts); >> TSSetProblemType(ts,TS_NONLINEAR); >> TSSetRHSFunction(ts,NULL,compute_RHS,NULL); >> TSSetMaxTime(ts,T_FINAL); >> TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER); >> TSSetDM(ts,da); >> TSSetType(ts,TSCN); //it works with: TSSetType(ts,TSRK); >> set_IC(da,x); >> TSSetTimeStep(ts,DT); >> TSSetSolution(ts,x); >> >> TSGetSNES(ts,&snes); >> SNESGetKSP(snes,&ksp); >> KSPGetPC(ksp,&pc); >> PCSetType(pc,PCNONE); >> >> DMSetMatType(da,MATAIJ); >> DMCreateMatrix(da,&J); >> ISColoring iscoloring; >> MatFDColoring matfdcoloring; >> DMCreateColoring(da,IS_COLORING_GLOBAL,&iscoloring); >> MatFDColoringCreate(J,iscoloring,&matfdcoloring); >> >> MatFDColoringSetType(matfdcoloring,MATMFFD_DS); >> >> // I think I do something wrong in the following 3 lines >> >> PetscErrorCode (*temp_f)(SNES,Vec,Vec,void*); >> SNESGetFunction(snes,NULL,&temp_f,NULL); >> MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode (*)(void))temp_f,NULL); >> >> MatFDColoringSetUp(J,iscoloring,matfdcoloring); >> >> SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,matfdcoloring); >> ISColoringDestroy(&iscoloring); >> >> TSSolve(ts,x); >> >> Thank you, >> >> Oleksandr Koshkarov. >> From olk548 at mail.usask.ca Sun Apr 29 17:40:23 2018 From: olk548 at mail.usask.ca (Oleksandr Koshkarov) Date: Sun, 29 Apr 2018 16:40:23 -0600 Subject: [petsc-users] Finite difference Jacobian in TS context In-Reply-To: <72B587E7-3100-44F6-946C-2203D4AAD627@anl.gov> References: <72B587E7-3100-44F6-946C-2203D4AAD627@anl.gov> Message-ID: <046d9b1d-c901-665f-887f-9e662ef14065@mail.usask.ca> Dear All, sorry for spam because of my poor PETSc knowledge (I am just starting with this nice framework). I think, I figured part of it out. However, I want to point that src/ts/examples/tutorials/ex15.c is misleading. (or maybe it is a bug?) in this example we have TSGetSNES(ts,&snes); MatCreateSNESMF(snes,&Jmf); SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefault,NULL); // or this: SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefaultColor,0); which implies (I think) that Jacobian would be matrix free. And if one would use PCNONE for preconditioning the matrix would never be allocated. However, it seems in reality it allocates matrix. To avoid it, I used MatCreateSNESMF(snes,&J); SNESSetJacobian(snes,J,J,MatMFFDComputeJacobian,0); which seems to work fine. I am not sure I fully understand the difference and i have zero intuition and I also have no idea what happens with preconditioning in this case. If someone have some useful comets, please share :) (I read the relevant section in PETSc manual, but still not fully understanding what I should use when) Thank you and again sorry for the spam, Oleksandr. On 04/28/2018 07:20 PM, Smith, Barry F. wrote: > ~/Src/petsc/src/ts/examples/tutorials > $ grep SNESComputeJacobianDefaultColor *.c > ex10.c: ierr = SNESSetJacobian(snes,A,B,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); > ex15.c: ierr = SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); > ex17.c: ierr = SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); > > I don't think you need to explicitly create the MatFDColoring object. > > Please take a look at ex15.c and see if you can get it working like that example. If that doesn't work let us know and we can take a closer look at it. > > > Barry > > >> On Apr 28, 2018, at 8:05 PM, Oleksandr Koshkarov wrote: >> >> Hello All, >> >> I hope someone can help :) I think I am doing something wrong, but cannot understand what. I have a huge time dependent system with 3d DMDA data structure and I am evolving it with explicit Runge-Kutta by using TS and basically only using "TSSetRHSFunction". Now I want to repeat it with implicit time stepper (for now Crank-Nicolson) and I am trying to provide finite difference Jacobian and I am failing miserably. I also cannot find appropriate example in PETSc tutorial (if you can point me to working example, it would be great). >> >> Here is my best attempt (what wrong with it?): >> >> DMDACreate3d(PETSC_COMM_WORLD, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, >> DMDA_STENCIL_STAR, >> NX, NY, NZ, >> PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, >> 2*3+NC*NS, >> 1, >> NULL, NULL, NULL, &da); >> DMSetUp(da); >> DMCreateGlobalVector(da,&x); >> TSCreate(PETSC_COMM_WORLD,&ts); >> TSSetProblemType(ts,TS_NONLINEAR); >> TSSetRHSFunction(ts,NULL,compute_RHS,NULL); >> TSSetMaxTime(ts,T_FINAL); >> TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER); >> TSSetDM(ts,da); >> TSSetType(ts,TSCN); //it works with: TSSetType(ts,TSRK); >> set_IC(da,x); >> TSSetTimeStep(ts,DT); >> TSSetSolution(ts,x); >> >> TSGetSNES(ts,&snes); >> SNESGetKSP(snes,&ksp); >> KSPGetPC(ksp,&pc); >> PCSetType(pc,PCNONE); >> >> DMSetMatType(da,MATAIJ); >> DMCreateMatrix(da,&J); >> ISColoring iscoloring; >> MatFDColoring matfdcoloring; >> DMCreateColoring(da,IS_COLORING_GLOBAL,&iscoloring); >> MatFDColoringCreate(J,iscoloring,&matfdcoloring); >> >> MatFDColoringSetType(matfdcoloring,MATMFFD_DS); >> >> // I think I do something wrong in the following 3 lines >> >> PetscErrorCode (*temp_f)(SNES,Vec,Vec,void*); >> SNESGetFunction(snes,NULL,&temp_f,NULL); >> MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode (*)(void))temp_f,NULL); >> >> MatFDColoringSetUp(J,iscoloring,matfdcoloring); >> >> SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,matfdcoloring); >> ISColoringDestroy(&iscoloring); >> >> TSSolve(ts,x); >> >> Thank you, >> >> Oleksandr Koshkarov. >> From bsmith at mcs.anl.gov Sun Apr 29 18:26:55 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 29 Apr 2018 23:26:55 +0000 Subject: [petsc-users] Finite difference Jacobian in TS context In-Reply-To: <046d9b1d-c901-665f-887f-9e662ef14065@mail.usask.ca> References: <72B587E7-3100-44F6-946C-2203D4AAD627@anl.gov> <046d9b1d-c901-665f-887f-9e662ef14065@mail.usask.ca> Message-ID: > On Apr 29, 2018, at 5:40 PM, Oleksandr Koshkarov wrote: > > Dear All, > > sorry for spam because of my poor PETSc knowledge (I am just starting with this nice framework). > > I think, I figured part of it out. However, I want to point that src/ts/examples/tutorials/ex15.c is misleading. (or maybe it is a bug?) > > in this example we have > > TSGetSNES(ts,&snes); > MatCreateSNESMF(snes,&Jmf); > SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefault,NULL); > > // or this: SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefaultColor,0); > > which implies (I think) that Jacobian would be matrix free. And if one would use PCNONE for preconditioning the matrix would never be allocated. However, it seems in reality it allocates matrix. The above uses matrix-free to do matrix-vector products for linear system but constructs the preconditioner by building the Jacobian via differencing and then using that matrix to build the preconditioner. > > To avoid it, I used > > MatCreateSNESMF(snes,&J); > > SNESSetJacobian(snes,J,J,MatMFFDComputeJacobian,0); > > which seems to work fine. I am not sure I fully understand the difference and i have zero intuition and I also have no idea what happens with preconditioning in this case. If someone have some useful comets, please share :) (I read the relevant section in PETSc manual, but still not fully understanding what I should use when) This does not build the Jacobian (only has matrix free matrix vector products) so requires much less memory but likely for large problem sizes the linear system solve will be slow (require many iterations) or won't converge at all. The conditioning of the linear system depends on the exact problem you are solving and the type of discretization you are using. There is no easy rules that always apply but for most discretizations of PDEs the number of iterations needed by the linear solver increases with the problem sizes. This means for most large problems matrix-free (without building any sort of jacobean and preconditioner) is impractical and one needs to pay the price of using more memory to get reasonable convergence. Barry > > Thank you and again sorry for the spam, > Oleksandr. > > On 04/28/2018 07:20 PM, Smith, Barry F. wrote: >> ~/Src/petsc/src/ts/examples/tutorials >> $ grep SNESComputeJacobianDefaultColor *.c >> ex10.c: ierr = SNESSetJacobian(snes,A,B,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); >> ex15.c: ierr = SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); >> ex17.c: ierr = SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); >> >> I don't think you need to explicitly create the MatFDColoring object. >> >> Please take a look at ex15.c and see if you can get it working like that example. If that doesn't work let us know and we can take a closer look at it. >> >> >> Barry >> >> >>> On Apr 28, 2018, at 8:05 PM, Oleksandr Koshkarov wrote: >>> >>> Hello All, >>> >>> I hope someone can help :) I think I am doing something wrong, but cannot understand what. I have a huge time dependent system with 3d DMDA data structure and I am evolving it with explicit Runge-Kutta by using TS and basically only using "TSSetRHSFunction". Now I want to repeat it with implicit time stepper (for now Crank-Nicolson) and I am trying to provide finite difference Jacobian and I am failing miserably. I also cannot find appropriate example in PETSc tutorial (if you can point me to working example, it would be great). >>> >>> Here is my best attempt (what wrong with it?): >>> >>> DMDACreate3d(PETSC_COMM_WORLD, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, >>> DMDA_STENCIL_STAR, >>> NX, NY, NZ, >>> PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, >>> 2*3+NC*NS, >>> 1, >>> NULL, NULL, NULL, &da); >>> DMSetUp(da); >>> DMCreateGlobalVector(da,&x); >>> TSCreate(PETSC_COMM_WORLD,&ts); >>> TSSetProblemType(ts,TS_NONLINEAR); >>> TSSetRHSFunction(ts,NULL,compute_RHS,NULL); >>> TSSetMaxTime(ts,T_FINAL); >>> TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER); >>> TSSetDM(ts,da); >>> TSSetType(ts,TSCN); //it works with: TSSetType(ts,TSRK); >>> set_IC(da,x); >>> TSSetTimeStep(ts,DT); >>> TSSetSolution(ts,x); >>> >>> TSGetSNES(ts,&snes); >>> SNESGetKSP(snes,&ksp); >>> KSPGetPC(ksp,&pc); >>> PCSetType(pc,PCNONE); >>> >>> DMSetMatType(da,MATAIJ); >>> DMCreateMatrix(da,&J); >>> ISColoring iscoloring; >>> MatFDColoring matfdcoloring; >>> DMCreateColoring(da,IS_COLORING_GLOBAL,&iscoloring); >>> MatFDColoringCreate(J,iscoloring,&matfdcoloring); >>> >>> MatFDColoringSetType(matfdcoloring,MATMFFD_DS); >>> >>> // I think I do something wrong in the following 3 lines >>> >>> PetscErrorCode (*temp_f)(SNES,Vec,Vec,void*); >>> SNESGetFunction(snes,NULL,&temp_f,NULL); >>> MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode (*)(void))temp_f,NULL); >>> >>> MatFDColoringSetUp(J,iscoloring,matfdcoloring); >>> >>> SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,matfdcoloring); >>> ISColoringDestroy(&iscoloring); >>> >>> TSSolve(ts,x); >>> >>> Thank you, >>> >>> Oleksandr Koshkarov. >>> > From olk548 at mail.usask.ca Sun Apr 29 18:44:30 2018 From: olk548 at mail.usask.ca (Oleksandr Koshkarov) Date: Sun, 29 Apr 2018 17:44:30 -0600 Subject: [petsc-users] Finite difference Jacobian in TS context In-Reply-To: References: <72B587E7-3100-44F6-946C-2203D4AAD627@anl.gov> <046d9b1d-c901-665f-887f-9e662ef14065@mail.usask.ca> Message-ID: <84346982-c9aa-fad7-ee96-64488c5a50d2@mail.usask.ca> Thank you. A little clarification: > The above uses matrix-free to do matrix-vector products for linear system but constructs the preconditioner by building the Jacobian via differencing and then using that matrix to build the preconditioner. So SNESComputeJacobianDefaultColor will use memory (construct Jacobian) even if preconditioning is set to PCNONE and J=NULL? (that is what I saw in my example) > This does not build the Jacobian (only has matrix free matrix vector products) so requires much less memory but likely for large problem sizes the linear system solve will be slow (require many iterations) or won't converge at all. The conditioning of the linear system depends on the exact problem you are solving and the type of discretization you are using. There is no easy rules that always apply but for most discretizations of PDEs the number of iterations needed by the linear solver increases with the problem sizes. This means for most large problems matrix-free (without building any sort of jacobean and preconditioner) is impractical and one needs to pay the price of using more memory to get reasonable convergence. So the following command? ignores precoditioning? Can I still use something like PCSHELL? SNESSetJacobian(snes,J,J,MatMFFDComputeJacobian,0); p.s. for my problems if probably be unrealistic to construct Jacobian (state size will start from N > 1000^4). Thank you, Oleksandr. On 04/29/2018 05:26 PM, Smith, Barry F. wrote: > >> On Apr 29, 2018, at 5:40 PM, Oleksandr Koshkarov wrote: >> >> Dear All, >> >> sorry for spam because of my poor PETSc knowledge (I am just starting with this nice framework). >> >> I think, I figured part of it out. However, I want to point that src/ts/examples/tutorials/ex15.c is misleading. (or maybe it is a bug?) >> >> in this example we have >> >> TSGetSNES(ts,&snes); >> MatCreateSNESMF(snes,&Jmf); >> SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefault,NULL); >> >> // or this: SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefaultColor,0); >> >> which implies (I think) that Jacobian would be matrix free. And if one would use PCNONE for preconditioning the matrix would never be allocated. However, it seems in reality it allocates matrix. > The above uses matrix-free to do matrix-vector products for linear system but constructs the preconditioner by building the Jacobian via differencing and then using that matrix to build the preconditioner. > >> To avoid it, I used >> >> MatCreateSNESMF(snes,&J); >> >> SNESSetJacobian(snes,J,J,MatMFFDComputeJacobian,0); >> >> which seems to work fine. I am not sure I fully understand the difference and i have zero intuition and I also have no idea what happens with preconditioning in this case. If someone have some useful comets, please share :) (I read the relevant section in PETSc manual, but still not fully understanding what I should use when) > This does not build the Jacobian (only has matrix free matrix vector products) so requires much less memory but likely for large problem sizes the linear system solve will be slow (require many iterations) or won't converge at all. The conditioning of the linear system depends on the exact problem you are solving and the type of discretization you are using. There is no easy rules that always apply but for most discretizations of PDEs the number of iterations needed by the linear solver increases with the problem sizes. This means for most large problems matrix-free (without building any sort of jacobean and preconditioner) is impractical and one needs to pay the price of using more memory to get reasonable convergence. > > Barry > >> Thank you and again sorry for the spam, >> Oleksandr. >> >> On 04/28/2018 07:20 PM, Smith, Barry F. wrote: >>> ~/Src/petsc/src/ts/examples/tutorials >>> $ grep SNESComputeJacobianDefaultColor *.c >>> ex10.c: ierr = SNESSetJacobian(snes,A,B,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); >>> ex15.c: ierr = SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); >>> ex17.c: ierr = SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); >>> >>> I don't think you need to explicitly create the MatFDColoring object. >>> >>> Please take a look at ex15.c and see if you can get it working like that example. If that doesn't work let us know and we can take a closer look at it. >>> >>> >>> Barry >>> >>> >>>> On Apr 28, 2018, at 8:05 PM, Oleksandr Koshkarov wrote: >>>> >>>> Hello All, >>>> >>>> I hope someone can help :) I think I am doing something wrong, but cannot understand what. I have a huge time dependent system with 3d DMDA data structure and I am evolving it with explicit Runge-Kutta by using TS and basically only using "TSSetRHSFunction". Now I want to repeat it with implicit time stepper (for now Crank-Nicolson) and I am trying to provide finite difference Jacobian and I am failing miserably. I also cannot find appropriate example in PETSc tutorial (if you can point me to working example, it would be great). >>>> >>>> Here is my best attempt (what wrong with it?): >>>> >>>> DMDACreate3d(PETSC_COMM_WORLD, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, >>>> DMDA_STENCIL_STAR, >>>> NX, NY, NZ, >>>> PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, >>>> 2*3+NC*NS, >>>> 1, >>>> NULL, NULL, NULL, &da); >>>> DMSetUp(da); >>>> DMCreateGlobalVector(da,&x); >>>> TSCreate(PETSC_COMM_WORLD,&ts); >>>> TSSetProblemType(ts,TS_NONLINEAR); >>>> TSSetRHSFunction(ts,NULL,compute_RHS,NULL); >>>> TSSetMaxTime(ts,T_FINAL); >>>> TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER); >>>> TSSetDM(ts,da); >>>> TSSetType(ts,TSCN); //it works with: TSSetType(ts,TSRK); >>>> set_IC(da,x); >>>> TSSetTimeStep(ts,DT); >>>> TSSetSolution(ts,x); >>>> >>>> TSGetSNES(ts,&snes); >>>> SNESGetKSP(snes,&ksp); >>>> KSPGetPC(ksp,&pc); >>>> PCSetType(pc,PCNONE); >>>> >>>> DMSetMatType(da,MATAIJ); >>>> DMCreateMatrix(da,&J); >>>> ISColoring iscoloring; >>>> MatFDColoring matfdcoloring; >>>> DMCreateColoring(da,IS_COLORING_GLOBAL,&iscoloring); >>>> MatFDColoringCreate(J,iscoloring,&matfdcoloring); >>>> >>>> MatFDColoringSetType(matfdcoloring,MATMFFD_DS); >>>> >>>> // I think I do something wrong in the following 3 lines >>>> >>>> PetscErrorCode (*temp_f)(SNES,Vec,Vec,void*); >>>> SNESGetFunction(snes,NULL,&temp_f,NULL); >>>> MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode (*)(void))temp_f,NULL); >>>> >>>> MatFDColoringSetUp(J,iscoloring,matfdcoloring); >>>> >>>> SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,matfdcoloring); >>>> ISColoringDestroy(&iscoloring); >>>> >>>> TSSolve(ts,x); >>>> >>>> Thank you, >>>> >>>> Oleksandr Koshkarov. >>>> From knepley at gmail.com Sun Apr 29 19:03:46 2018 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 29 Apr 2018 20:03:46 -0400 Subject: [petsc-users] Finite difference Jacobian in TS context In-Reply-To: <84346982-c9aa-fad7-ee96-64488c5a50d2@mail.usask.ca> References: <72B587E7-3100-44F6-946C-2203D4AAD627@anl.gov> <046d9b1d-c901-665f-887f-9e662ef14065@mail.usask.ca> <84346982-c9aa-fad7-ee96-64488c5a50d2@mail.usask.ca> Message-ID: On Sun, Apr 29, 2018 at 7:44 PM, Oleksandr Koshkarov wrote: > Thank you. A little clarification: > > The above uses matrix-free to do matrix-vector products for linear >> system but constructs the preconditioner by building the Jacobian via >> differencing and then using that matrix to build the preconditioner. >> > So SNESComputeJacobianDefaultColor will use memory (construct Jacobian) > even if preconditioning is set to PCNONE and J=NULL? (that is what I saw in > my example) > 1) SNESComputeJacobianDefaultColor() creates the Jacobian, fullstop. 2) When you pass J=NULL, the Jacobian is created automatically by DMDA 3) The PC has no influence on the assembly process. > This does not build the Jacobian (only has matrix free matrix vector >> products) so requires much less memory but likely for large problem sizes >> the linear system solve will be slow (require many iterations) or won't >> converge at all. The conditioning of the linear system depends on the >> exact problem you are solving and the type of discretization you are using. >> There is no easy rules that always apply but for most discretizations of >> PDEs the number of iterations needed by the linear solver increases with >> the problem sizes. This means for most large problems matrix-free (without >> building any sort of jacobean and preconditioner) is impractical and one >> needs to pay the price of using more memory to get reasonable convergence. >> > So the following command ignores precoditioning? Ignore is the wrong word. How do you form a preconditioner from an object with no values in it? > Can I still use something like PCSHELL? > What exactly would you do in your shell? If you are only using the action of an operator, it usually equivalent to some Krylov method. > SNESSetJacobian(snes,J,J,MatMFFDComputeJacobian,0); > > p.s. for my problems if probably be unrealistic to construct Jacobian > (state size will start from N > 1000^4). > The number of dofs does not tell us anything. You would need to know the sparsity. People regularly solve problems with billions of unknowns. Thanks, Matt > Thank you, > > Oleksandr. > > > On 04/29/2018 05:26 PM, Smith, Barry F. wrote: > >> >> On Apr 29, 2018, at 5:40 PM, Oleksandr Koshkarov >>> wrote: >>> >>> Dear All, >>> >>> sorry for spam because of my poor PETSc knowledge (I am just starting >>> with this nice framework). >>> >>> I think, I figured part of it out. However, I want to point that >>> src/ts/examples/tutorials/ex15.c is misleading. (or maybe it is a bug?) >>> >>> in this example we have >>> >>> TSGetSNES(ts,&snes); >>> MatCreateSNESMF(snes,&Jmf); >>> SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefault,NULL); >>> >>> // or this: SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefaultColor, >>> 0); >>> >>> which implies (I think) that Jacobian would be matrix free. And if one >>> would use PCNONE for preconditioning the matrix would never be allocated. >>> However, it seems in reality it allocates matrix. >>> >> The above uses matrix-free to do matrix-vector products for linear >> system but constructs the preconditioner by building the Jacobian via >> differencing and then using that matrix to build the preconditioner. >> >> To avoid it, I used >>> >>> MatCreateSNESMF(snes,&J); >>> >>> SNESSetJacobian(snes,J,J,MatMFFDComputeJacobian,0); >>> >>> which seems to work fine. I am not sure I fully understand the >>> difference and i have zero intuition and I also have no idea what happens >>> with preconditioning in this case. If someone have some useful comets, >>> please share :) (I read the relevant section in PETSc manual, but still not >>> fully understanding what I should use when) >>> >> This does not build the Jacobian (only has matrix free matrix vector >> products) so requires much less memory but likely for large problem sizes >> the linear system solve will be slow (require many iterations) or won't >> converge at all. The conditioning of the linear system depends on the >> exact problem you are solving and the type of discretization you are using. >> There is no easy rules that always apply but for most discretizations of >> PDEs the number of iterations needed by the linear solver increases with >> the problem sizes. This means for most large problems matrix-free (without >> building any sort of jacobean and preconditioner) is impractical and one >> needs to pay the price of using more memory to get reasonable convergence. >> >> Barry >> >> Thank you and again sorry for the spam, >>> Oleksandr. >>> >>> On 04/28/2018 07:20 PM, Smith, Barry F. wrote: >>> >>>> ~/Src/petsc/src/ts/examples/tutorials >>>> $ grep SNESComputeJacobianDefaultColor *.c >>>> ex10.c: ierr = SNESSetJacobian(snes,A,B,SNESC >>>> omputeJacobianDefaultColor,0);CHKERRQ(ierr); >>>> ex15.c: ierr = SNESSetJacobian(snes,Jmf,J,SNE >>>> SComputeJacobianDefaultColor,0);CHKERRQ(ierr); >>>> ex17.c: ierr = SNESSetJacobian(snes,J,J,SNESC >>>> omputeJacobianDefaultColor,0);CHKERRQ(ierr); >>>> >>>> I don't think you need to explicitly create the MatFDColoring >>>> object. >>>> >>>> Please take a look at ex15.c and see if you can get it working like >>>> that example. If that doesn't work let us know and we can take a closer >>>> look at it. >>>> >>>> >>>> Barry >>>> >>>> >>>> On Apr 28, 2018, at 8:05 PM, Oleksandr Koshkarov >>>>> wrote: >>>>> >>>>> Hello All, >>>>> >>>>> I hope someone can help :) I think I am doing something wrong, but >>>>> cannot understand what. I have a huge time dependent system with 3d DMDA >>>>> data structure and I am evolving it with explicit Runge-Kutta by using TS >>>>> and basically only using "TSSetRHSFunction". Now I want to repeat it with >>>>> implicit time stepper (for now Crank-Nicolson) and I am trying to provide >>>>> finite difference Jacobian and I am failing miserably. I also cannot find >>>>> appropriate example in PETSc tutorial (if you can point me to working >>>>> example, it would be great). >>>>> >>>>> Here is my best attempt (what wrong with it?): >>>>> >>>>> DMDACreate3d(PETSC_COMM_WORLD, DM_BOUNDARY_PERIODIC, >>>>> DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, >>>>> DMDA_STENCIL_STAR, >>>>> NX, NY, NZ, >>>>> PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, >>>>> 2*3+NC*NS, >>>>> 1, >>>>> NULL, NULL, NULL, &da); >>>>> DMSetUp(da); >>>>> DMCreateGlobalVector(da,&x); >>>>> TSCreate(PETSC_COMM_WORLD,&ts); >>>>> TSSetProblemType(ts,TS_NONLINEAR); >>>>> TSSetRHSFunction(ts,NULL,compute_RHS,NULL); >>>>> TSSetMaxTime(ts,T_FINAL); >>>>> TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER); >>>>> TSSetDM(ts,da); >>>>> TSSetType(ts,TSCN); //it works with: TSSetType(ts,TSRK); >>>>> set_IC(da,x); >>>>> TSSetTimeStep(ts,DT); >>>>> TSSetSolution(ts,x); >>>>> >>>>> TSGetSNES(ts,&snes); >>>>> SNESGetKSP(snes,&ksp); >>>>> KSPGetPC(ksp,&pc); >>>>> PCSetType(pc,PCNONE); >>>>> >>>>> DMSetMatType(da,MATAIJ); >>>>> DMCreateMatrix(da,&J); >>>>> ISColoring iscoloring; >>>>> MatFDColoring matfdcoloring; >>>>> DMCreateColoring(da,IS_COLORING_GLOBAL,&iscoloring); >>>>> MatFDColoringCreate(J,iscoloring,&matfdcoloring); >>>>> >>>>> MatFDColoringSetType(matfdcoloring,MATMFFD_DS); >>>>> >>>>> // I think I do something wrong in the following 3 lines >>>>> >>>>> PetscErrorCode (*temp_f)(SNES,Vec,Vec,void*); >>>>> SNESGetFunction(snes,NULL,&temp_f,NULL); >>>>> MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode >>>>> (*)(void))temp_f,NULL); >>>>> >>>>> MatFDColoringSetUp(J,iscoloring,matfdcoloring); >>>>> >>>>> SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor, >>>>> matfdcoloring); >>>>> ISColoringDestroy(&iscoloring); >>>>> >>>>> TSSolve(ts,x); >>>>> >>>>> Thank you, >>>>> >>>>> Oleksandr Koshkarov. >>>>> >>>>> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From olk548 at mail.usask.ca Sun Apr 29 19:29:10 2018 From: olk548 at mail.usask.ca (Oleksandr Koshkarov) Date: Sun, 29 Apr 2018 18:29:10 -0600 Subject: [petsc-users] Finite difference Jacobian in TS context In-Reply-To: References: <72B587E7-3100-44F6-946C-2203D4AAD627@anl.gov> <046d9b1d-c901-665f-887f-9e662ef14065@mail.usask.ca> <84346982-c9aa-fad7-ee96-64488c5a50d2@mail.usask.ca> Message-ID: Thank you, it clarifies a lot :) > Ignore is the wrong word. How do you form a preconditioner from an > object with no values in it? > > Can I still use something like PCSHELL? > > > What exactly would you do in your shell? If you are only using the > action of an operator, it > usually equivalent to some Krylov method. Yes, I was thinking to use FD matrix free Jacobian and for preconditioning to use Krylov method to invert the linearized Jacobian which also would be FD matrix free. Sounds bad? Well, from your advices I convinced to try constructing actual Jacobian and use it as Jacobian and matrix from which petsc would construct preconditioner. p.s. Do I reply to emails correctly? or should I replay to petsc-users at mcs.anl.gov only? Thank you, Oleksandr. On 04/29/2018 06:03 PM, Matthew Knepley wrote: > On Sun, Apr 29, 2018 at 7:44 PM, Oleksandr Koshkarov > > wrote: > > Thank you. A little clarification: > > ? ? The above uses matrix-free to do matrix-vector products > for linear system but constructs the preconditioner by > building the Jacobian via differencing and then using that > matrix to build the preconditioner. > > So SNESComputeJacobianDefaultColor will use memory (construct > Jacobian) even if preconditioning is set to PCNONE and J=NULL? > (that is what I saw in my example) > > > 1) SNESComputeJacobianDefaultColor() creates the Jacobian, fullstop. > > 2) When you pass J=NULL, the Jacobian is created automatically by DMDA > > 3) The PC has no influence on the assembly process. > > ? ? This does not build the Jacobian (only has matrix free > matrix vector products) so requires much less memory but > likely for large problem sizes the linear system solve will be > slow (require many iterations) or won't converge at all.? ?The > conditioning of the linear system depends on the exact problem > you are solving and the type of discretization you are using. > There is no easy rules that always apply but for most > discretizations of PDEs the number of iterations needed by the > linear solver increases with the problem sizes. This means for > most large problems matrix-free (without building any sort of > jacobean and preconditioner) is impractical and one needs to > pay the price of using more memory to get reasonable convergence. > > So the following command? ignores precoditioning? > > > Ignore is the wrong word. How do you form a preconditioner from an > object with no values in it? > > Can I still use something like PCSHELL? > > > What exactly would you do in your shell? If you are only using the > action of an operator, it > usually equivalent to some Krylov method. > > SNESSetJacobian(snes,J,J,MatMFFDComputeJacobian,0); > > p.s. for my problems if probably be unrealistic to construct > Jacobian (state size will start from N > 1000^4). > > > The number of dofs does not tell us anything. You would need to know > the sparsity. People regularly solve > problems with billions of unknowns. > > ? Thanks, > > ? ? ?Matt > > Thank you, > > Oleksandr. > > > On 04/29/2018 05:26 PM, Smith, Barry F. wrote: > > > On Apr 29, 2018, at 5:40 PM, Oleksandr Koshkarov > > wrote: > > Dear All, > > sorry for spam because of my poor PETSc knowledge (I am > just starting with this nice framework). > > I think, I figured part of it out. However, I want to > point that src/ts/examples/tutorials/ex15.c is misleading. > (or maybe it is a bug?) > > in this example we have > > TSGetSNES(ts,&snes); > MatCreateSNESMF(snes,&Jmf); > SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefault,NULL); > > // or this: > SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefaultColor,0); > > which implies (I think) that Jacobian would be matrix > free. And if one would use PCNONE for preconditioning the > matrix would never be allocated. However, it seems in > reality it allocates matrix. > > ? ? The above uses matrix-free to do matrix-vector products > for linear system but constructs the preconditioner by > building the Jacobian via differencing and then using that > matrix to build the preconditioner. > > To avoid it, I used > > MatCreateSNESMF(snes,&J); > > SNESSetJacobian(snes,J,J,MatMFFDComputeJacobian,0); > > which seems to work fine. I am not sure I fully understand > the difference and i have zero intuition and I also have > no idea what happens with preconditioning in this case. If > someone have some useful comets, please share :) (I read > the relevant section in PETSc manual, but still not fully > understanding what I should use when) > > ? ? ?This does not build the Jacobian (only has matrix free > matrix vector products) so requires much less memory but > likely for large problem sizes the linear system solve will be > slow (require many iterations) or won't converge at all.? ?The > conditioning of the linear system depends on the exact problem > you are solving and the type of discretization you are using. > There is no easy rules that always apply but for most > discretizations of PDEs the number of iterations needed by the > linear solver increases with the problem sizes. This means for > most large problems matrix-free (without building any sort of > jacobean and preconditioner) is impractical and one needs to > pay the price of using more memory to get reasonable convergence. > > ? ?Barry > > Thank you and again sorry for the spam, > Oleksandr. > > On 04/28/2018 07:20 PM, Smith, Barry F. wrote: > > ~/Src/petsc/src/ts/examples/tutorials > $ grep SNESComputeJacobianDefaultColor *.c > ex10.c:? ? ierr = > SNESSetJacobian(snes,A,B,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); > ex15.c:? ? ? ierr = > SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); > ex17.c:? ? ierr = > SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr); > > ? ? I don't think you need to explicitly create the > MatFDColoring object. > > ? ? Please take a look at ex15.c and see if you can > get it working like that example. If that doesn't work > let us know and we can take a closer look at it. > > > ? ? Barry > > > On Apr 28, 2018, at 8:05 PM, Oleksandr Koshkarov > > wrote: > > Hello All, > > I hope someone can help :) I think I am doing > something wrong, but cannot understand what. I > have a huge time dependent system with 3d DMDA > data structure and I am evolving it with explicit > Runge-Kutta by using TS and basically only using > "TSSetRHSFunction". Now I want to repeat it with > implicit time stepper (for now Crank-Nicolson) and > I am trying to provide finite difference Jacobian > and I am failing miserably. I also cannot find > appropriate example in PETSc tutorial (if you can > point me to working example, it would be great). > > Here is my best attempt (what wrong with it?): > > ? ?DMDACreate3d(PETSC_COMM_WORLD, > DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, > DM_BOUNDARY_PERIODIC, > ? ? ? ? ? ? DMDA_STENCIL_STAR, > ? ? ? ? ? ? NX, NY, NZ, > ? ? ? ? ? ? PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, > ? ? ? ? ? ? 2*3+NC*NS, > ? ? ? ? ? ? 1, > ? ? ? ? ? ? NULL,? NULL, NULL,? &da); > ? ?DMSetUp(da); > ? ?DMCreateGlobalVector(da,&x); > ? ?TSCreate(PETSC_COMM_WORLD,&ts); > ? ?TSSetProblemType(ts,TS_NONLINEAR); > ? ?TSSetRHSFunction(ts,NULL,compute_RHS,NULL); > ? ?TSSetMaxTime(ts,T_FINAL); > ? ?TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER); > ? ?TSSetDM(ts,da); > ? ?TSSetType(ts,TSCN); //it works with: > TSSetType(ts,TSRK); > ? ?set_IC(da,x); > ? ?TSSetTimeStep(ts,DT); > ? ?TSSetSolution(ts,x); > > ? ?TSGetSNES(ts,&snes); > ? ?SNESGetKSP(snes,&ksp); > ? ?KSPGetPC(ksp,&pc); > ? ?PCSetType(pc,PCNONE); > > ? ?DMSetMatType(da,MATAIJ); > ? ?DMCreateMatrix(da,&J); > ? ?ISColoring iscoloring; > ? ?MatFDColoring? matfdcoloring; > ? > ?DMCreateColoring(da,IS_COLORING_GLOBAL,&iscoloring); > ? ?MatFDColoringCreate(J,iscoloring,&matfdcoloring); > > ? ?MatFDColoringSetType(matfdcoloring,MATMFFD_DS); > > // I think I do something wrong in the following 3 > lines > > ? ?PetscErrorCode (*temp_f)(SNES,Vec,Vec,void*); > ? ?SNESGetFunction(snes,NULL,&temp_f,NULL); > ? > ?MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode > (*)(void))temp_f,NULL); > > ? ?MatFDColoringSetUp(J,iscoloring,matfdcoloring); > > SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,matfdcoloring); > ? ?ISColoringDestroy(&iscoloring); > > ? ?TSSolve(ts,x); > > Thank you, > > Oleksandr Koshkarov. > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 30 05:38:57 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 30 Apr 2018 06:38:57 -0400 Subject: [petsc-users] Finite difference Jacobian in TS context In-Reply-To: References: <72B587E7-3100-44F6-946C-2203D4AAD627@anl.gov> <046d9b1d-c901-665f-887f-9e662ef14065@mail.usask.ca> <84346982-c9aa-fad7-ee96-64488c5a50d2@mail.usask.ca> Message-ID: On Sun, Apr 29, 2018 at 8:29 PM, Oleksandr Koshkarov wrote: > Thank you, it clarifies a lot :) > > Ignore is the wrong word. How do you form a preconditioner from an object > with no values in it? > > >> Can I still use something like PCSHELL? >> > > What exactly would you do in your shell? If you are only using the action > of an operator, it > usually equivalent to some Krylov method. > > > Yes, I was thinking to use FD matrix free Jacobian and for preconditioning > to use Krylov method to invert the linearized Jacobian which also would be > FD matrix free. Sounds bad? > It will not be scalable, unless you are using the Boundary Element Method. PDE operators have condition number that grows pretty rapidly with size. > Well, from your advices I convinced to try constructing actual Jacobian > and use it as Jacobian and matrix from which petsc would construct > preconditioner. > Or the first option: Apply J matrix-free for the action in a Krylov method, but build a perhaps simplified operator for preconditioning. If there is no obvious simplification, then just build the whole thing. Thanks, Matt > p.s. Do I reply to emails correctly? or should I replay to > petsc-users at mcs.anl.gov only? > > Thank you, > Oleksandr. > > On 04/29/2018 06:03 PM, Matthew Knepley wrote: > > On Sun, Apr 29, 2018 at 7:44 PM, Oleksandr Koshkarov > wrote: > >> Thank you. A little clarification: >> >> The above uses matrix-free to do matrix-vector products for linear >>> system but constructs the preconditioner by building the Jacobian via >>> differencing and then using that matrix to build the preconditioner. >>> >> So SNESComputeJacobianDefaultColor will use memory (construct Jacobian) >> even if preconditioning is set to PCNONE and J=NULL? (that is what I saw in >> my example) >> > > 1) SNESComputeJacobianDefaultColor() creates the Jacobian, fullstop. > > 2) When you pass J=NULL, the Jacobian is created automatically by DMDA > > 3) The PC has no influence on the assembly process. > > >> This does not build the Jacobian (only has matrix free matrix vector >>> products) so requires much less memory but likely for large problem sizes >>> the linear system solve will be slow (require many iterations) or won't >>> converge at all. The conditioning of the linear system depends on the >>> exact problem you are solving and the type of discretization you are using. >>> There is no easy rules that always apply but for most discretizations of >>> PDEs the number of iterations needed by the linear solver increases with >>> the problem sizes. This means for most large problems matrix-free (without >>> building any sort of jacobean and preconditioner) is impractical and one >>> needs to pay the price of using more memory to get reasonable convergence. >>> >> So the following command ignores precoditioning? > > > Ignore is the wrong word. How do you form a preconditioner from an object > with no values in it? > > >> Can I still use something like PCSHELL? >> > > What exactly would you do in your shell? If you are only using the action > of an operator, it > usually equivalent to some Krylov method. > > >> SNESSetJacobian(snes,J,J,MatMFFDComputeJacobian,0); >> >> p.s. for my problems if probably be unrealistic to construct Jacobian >> (state size will start from N > 1000^4). >> > > The number of dofs does not tell us anything. You would need to know the > sparsity. People regularly solve > problems with billions of unknowns. > > Thanks, > > Matt > > >> Thank you, >> >> Oleksandr. >> >> >> On 04/29/2018 05:26 PM, Smith, Barry F. wrote: >> >>> >>> On Apr 29, 2018, at 5:40 PM, Oleksandr Koshkarov >>>> wrote: >>>> >>>> Dear All, >>>> >>>> sorry for spam because of my poor PETSc knowledge (I am just starting >>>> with this nice framework). >>>> >>>> I think, I figured part of it out. However, I want to point that >>>> src/ts/examples/tutorials/ex15.c is misleading. (or maybe it is a bug?) >>>> >>>> in this example we have >>>> >>>> TSGetSNES(ts,&snes); >>>> MatCreateSNESMF(snes,&Jmf); >>>> SNESSetJacobian(snes,Jmf,J,SNESComputeJacobianDefault,NULL); >>>> >>>> // or this: SNESSetJacobian(snes,Jmf,J,SNE >>>> SComputeJacobianDefaultColor,0); >>>> >>>> which implies (I think) that Jacobian would be matrix free. And if one >>>> would use PCNONE for preconditioning the matrix would never be allocated. >>>> However, it seems in reality it allocates matrix. >>>> >>> The above uses matrix-free to do matrix-vector products for linear >>> system but constructs the preconditioner by building the Jacobian via >>> differencing and then using that matrix to build the preconditioner. >>> >>> To avoid it, I used >>>> >>>> MatCreateSNESMF(snes,&J); >>>> >>>> SNESSetJacobian(snes,J,J,MatMFFDComputeJacobian,0); >>>> >>>> which seems to work fine. I am not sure I fully understand the >>>> difference and i have zero intuition and I also have no idea what happens >>>> with preconditioning in this case. If someone have some useful comets, >>>> please share :) (I read the relevant section in PETSc manual, but still not >>>> fully understanding what I should use when) >>>> >>> This does not build the Jacobian (only has matrix free matrix >>> vector products) so requires much less memory but likely for large problem >>> sizes the linear system solve will be slow (require many iterations) or >>> won't converge at all. The conditioning of the linear system depends on >>> the exact problem you are solving and the type of discretization you are >>> using. There is no easy rules that always apply but for most >>> discretizations of PDEs the number of iterations needed by the linear >>> solver increases with the problem sizes. This means for most large problems >>> matrix-free (without building any sort of jacobean and preconditioner) is >>> impractical and one needs to pay the price of using more memory to get >>> reasonable convergence. >>> >>> Barry >>> >>> Thank you and again sorry for the spam, >>>> Oleksandr. >>>> >>>> On 04/28/2018 07:20 PM, Smith, Barry F. wrote: >>>> >>>>> ~/Src/petsc/src/ts/examples/tutorials >>>>> $ grep SNESComputeJacobianDefaultColor *.c >>>>> ex10.c: ierr = SNESSetJacobian(snes,A,B,SNESC >>>>> omputeJacobianDefaultColor,0);CHKERRQ(ierr); >>>>> ex15.c: ierr = SNESSetJacobian(snes,Jmf,J,SNE >>>>> SComputeJacobianDefaultColor,0);CHKERRQ(ierr); >>>>> ex17.c: ierr = SNESSetJacobian(snes,J,J,SNESC >>>>> omputeJacobianDefaultColor,0);CHKERRQ(ierr); >>>>> >>>>> I don't think you need to explicitly create the MatFDColoring >>>>> object. >>>>> >>>>> Please take a look at ex15.c and see if you can get it working >>>>> like that example. If that doesn't work let us know and we can take a >>>>> closer look at it. >>>>> >>>>> >>>>> Barry >>>>> >>>>> >>>>> On Apr 28, 2018, at 8:05 PM, Oleksandr Koshkarov >>>>>> wrote: >>>>>> >>>>>> Hello All, >>>>>> >>>>>> I hope someone can help :) I think I am doing something wrong, but >>>>>> cannot understand what. I have a huge time dependent system with 3d DMDA >>>>>> data structure and I am evolving it with explicit Runge-Kutta by using TS >>>>>> and basically only using "TSSetRHSFunction". Now I want to repeat it with >>>>>> implicit time stepper (for now Crank-Nicolson) and I am trying to provide >>>>>> finite difference Jacobian and I am failing miserably. I also cannot find >>>>>> appropriate example in PETSc tutorial (if you can point me to working >>>>>> example, it would be great). >>>>>> >>>>>> Here is my best attempt (what wrong with it?): >>>>>> >>>>>> DMDACreate3d(PETSC_COMM_WORLD, DM_BOUNDARY_PERIODIC, >>>>>> DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, >>>>>> DMDA_STENCIL_STAR, >>>>>> NX, NY, NZ, >>>>>> PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, >>>>>> 2*3+NC*NS, >>>>>> 1, >>>>>> NULL, NULL, NULL, &da); >>>>>> DMSetUp(da); >>>>>> DMCreateGlobalVector(da,&x); >>>>>> TSCreate(PETSC_COMM_WORLD,&ts); >>>>>> TSSetProblemType(ts,TS_NONLINEAR); >>>>>> TSSetRHSFunction(ts,NULL,compute_RHS,NULL); >>>>>> TSSetMaxTime(ts,T_FINAL); >>>>>> TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER); >>>>>> TSSetDM(ts,da); >>>>>> TSSetType(ts,TSCN); //it works with: TSSetType(ts,TSRK); >>>>>> set_IC(da,x); >>>>>> TSSetTimeStep(ts,DT); >>>>>> TSSetSolution(ts,x); >>>>>> >>>>>> TSGetSNES(ts,&snes); >>>>>> SNESGetKSP(snes,&ksp); >>>>>> KSPGetPC(ksp,&pc); >>>>>> PCSetType(pc,PCNONE); >>>>>> >>>>>> DMSetMatType(da,MATAIJ); >>>>>> DMCreateMatrix(da,&J); >>>>>> ISColoring iscoloring; >>>>>> MatFDColoring matfdcoloring; >>>>>> DMCreateColoring(da,IS_COLORING_GLOBAL,&iscoloring); >>>>>> MatFDColoringCreate(J,iscoloring,&matfdcoloring); >>>>>> >>>>>> MatFDColoringSetType(matfdcoloring,MATMFFD_DS); >>>>>> >>>>>> // I think I do something wrong in the following 3 lines >>>>>> >>>>>> PetscErrorCode (*temp_f)(SNES,Vec,Vec,void*); >>>>>> SNESGetFunction(snes,NULL,&temp_f,NULL); >>>>>> MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode >>>>>> (*)(void))temp_f,NULL); >>>>>> >>>>>> MatFDColoringSetUp(J,iscoloring,matfdcoloring); >>>>>> >>>>>> SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,mat >>>>>> fdcoloring); >>>>>> ISColoringDestroy(&iscoloring); >>>>>> >>>>>> TSSolve(ts,x); >>>>>> >>>>>> Thank you, >>>>>> >>>>>> Oleksandr Koshkarov. >>>>>> >>>>>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From afrah.nacib at gmail.com Mon Apr 30 06:20:32 2018 From: afrah.nacib at gmail.com (Afrah Najib) Date: Mon, 30 Apr 2018 14:20:32 +0300 Subject: [petsc-users] PDSLin hybrid solver performance with PETSc Message-ID: Hi all, I am testing PDSLin hybrid solver performance with PETSc krylov method on a SPD matrix called audik_1 matrix(format is ijv full matrix ordered by rows) https://sparse.tamu.edu/GHS_psdef/audikw_1 I am also running on cluster with Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz processor and 28 core , 128 ; 512GB per node but it takes a lot of time for Schur complement factorization. Compiler Subdomains processors Precond TmatrixPartition TmatrixDist Tfactorise TcompApproSchur TprepApproSchur Tfactoschur Tprecond_Total Tsolve gcc 8 128 Sparse 23.49532 0.6009915 6.059546 212.8243 0.04027608 38.74885 257.6796 13.69609 gcc 8 16 Sparse 11.864 0.4385703 27.23499 1390.153 0.5523289 55.20706 1473.207 4.213825 gcc 8 32 Sparse 9.912792 0.2592737 16.79125 934.3571 0.2372452 49.39175 1000.813 37.2029 gcc 8 64 Sparse 12.61929 0.2336042 8.2316 632.839 0.1113939 52.21551 693.4153 13.94034 My PETSc in stallation configuration is like this: export PETSC_DIR=/X/petsc export PETSC_ARCH=arch-real ./configure PETSC_DIR=/X/petsc \ prefix=/X/petsc/petsc-install \ PETSC_ARCH=arch-real \ --with-debugging=0 \ --with-cc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc \ --with-cxx=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicxx \ --with-fc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 \ --with-valgrind=1 \ --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 \ --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 \ --with-blas-lapack-dir=/Y/progs/intel/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64. *My slurm scrpt content is like this:* #!/bin/bash -l #SBATCH -J "Paudik16" #SBATCH -A X #SBATCH -p X #SBATCH --exclusive #SBATCH -o Paudik_8_2_16.%j.out #SBATCH -e Paudik_8_2_16.%j.err #SBATCH -N 8 #SBATCH --ntasks-per-node=2 #SBATCH --cpus-per-task=1 export KMP_AFFINITY=verbose,disabled export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK} export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} module purge module load gcc/7.1.0 module load mpi/openmpi-3.0.0-gcc-7.1.0 module load intel/mkl_2017.0.098 file=input.in matrixFile=~/dataDir/ijv/audikw_1.ijv mpirun -np $SLURM_NTASKS ./dtest_pdslin $file $matrixFile *I attached an output of one file along with the input configuration of PDSLin.* Any idea of how to enhance these results. Note in PDSLin, I tried with parallel Schur permutation but I got the following error : [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [1]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [1]PETSC ERROR: to get more information on the crash. [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on e/Xactly when Open MPI kills them. -------------------------------------------------------------------------- [13]PETSC ERROR: ------------------------------------------------------------------------ [13]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [13]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [13]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [13]PETSC ERROR: to get more information on the crash. [11]PETSC ERROR: ------------------------------------------------------------------------ [11]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [11]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [11]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [11]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [11]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [11]PETSC ERROR: to get more information on the crash. [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [3]PETSC ERROR: to get more information on the crash. [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Signal received [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown [3]PETSC ERROR: [9]PETSC ERROR: ------------------------------------------------------------------------ [9]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [9]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [9]PETSC ERROR: to get more information on the crash. [7]PETSC ERROR: ------------------------------------------------------------------------ [7]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [7]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [7]PETSC ERROR: to get more information on the crash. [5]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [5]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [5]PETSC ERROR: to get more information on the crash. [5]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [5]PETSC ERROR: Signal received [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [5]PETSC ERROR: Petsc Release Version 3.9.0, unknown [5]PETSC ERROR: [15]PETSC ERROR: ------------------------------------------------------------------------ [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [15]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [13]PETSC ERROR: Signal received [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [13]PETSC ERROR: Petsc Release Version 3.9.0, unknown [13]PETSC ERROR: ./dtest_pdslin on a named s060 by afarea Mon Apr 30 05:29:53 2018 [13]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [11]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [11]PETSC ERROR: Signal received [11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [11]PETSC ERROR: Petsc Release Version 3.9.0, unknown [11]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 05:29:53 2018 [11]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [3]PETSC ERROR: #1 User provided function() line 0 in unknown file [9]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [9]PETSC ERROR: Signal received [9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [9]PETSC ERROR: Petsc Release Version 3.9.0, unknown [9]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 05:29:53 2018 [9]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [7]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [7]PETSC ERROR: Signal received [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [7]PETSC ERROR: Petsc Release Version 3.9.0, unknown [7]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 05:29:53 2018 [7]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [7]PETSC ERROR: #1 User provided function() line 0 in unknown file ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 [5]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [5]PETSC ERROR: #1 User provided function() line 0 in unknown file configure using --with-debugging=yes, recompile, link, and run [15]PETSC ERROR: to get more information on the crash. [15]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [15]PETSC ERROR: Signal received [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018 [13]PETSC ERROR: #1 User provided function() line 0 in unknown file [11]PETSC ERROR: #1 User provided function() line 0 in unknown file [9]PETSC ERROR: #1 User provided function() line 0 in unknown file [15]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [15]PETSC ERROR: #1 User provided function() line 0 in unknown file [1]PETSC ERROR: Signal received [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018 [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [1]PETSC ERROR: #1 User provided function() line 0 in unknown file [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018 [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [1]PETSC ERROR: #1 User provided function() line 0 in unknown file [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [6]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [6]PETSC ERROR: to get more information on the crash. [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [6]PETSC ERROR: Signal received [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [6]PETSC ERROR: Petsc Release Version 3.9.0, unknown [6]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 05:29:53 2018 [6]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [6]PETSC ERROR: #1 User provided function() line 0 in unknown file [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [4]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [4]PETSC ERROR: to get more information on the crash. [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [4]PETSC ERROR: Signal received [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 [4]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [4]PETSC ERROR: #1 User provided function() line 0 in unknown file [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018 [0]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [0]PETSC ERROR: #1 User provided function() line 0 in unknown file [10]PETSC ERROR: ------------------------------------------------------------------------ [10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [10]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [10]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [10]PETSC ERROR: to get more information on the crash. [10]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [10]PETSC ERROR: Signal received [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [10]PETSC ERROR: Petsc Release Version 3.9.0, unknown [10]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 05:29:53 2018 [8]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 05:29:53 2018 [8]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [8]PETSC ERROR: #1 User provided function() line 0 in unknown file [14]PETSC ERROR: ------------------------------------------------------------------------ [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [14]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [14]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [14]PETSC ERROR: to get more information on the crash. [14]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [14]PETSC ERROR: Signal received [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [14]PETSC ERROR: Petsc Release Version 3.9.0, unknown [14]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018 [14]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [14]PETSC ERROR: #1 User provided function() line 0 in unknown file [15]PETSC ERROR: ------------------------------------------------------------------------ [15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [15]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [15]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [15]PETSC ERROR: to get more information on the crash. [15]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [15]PETSC ERROR: Signal received [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018 [15]PETSC ERROR: #2 User provided function() line 0 in unknown file [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [2]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [2]PETSC ERROR: to get more information on the crash. [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Signal received [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.9.0, unknown [2]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 [2]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [2]PETSC ERROR: #1 User provided function() line 0 in unknown file [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [3]PETSC ERROR: to get more information on the crash. [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Signal received [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown [3]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [3]PETSC ERROR: #2 User provided function() line 0 in unknown file [f002:38724] 17 more processes have sent help message help-mpi-api.t/Xt / mpi-abort [f002:38724] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Paudik_8_2_16.100234.out Type: application/octet-stream Size: 33176 bytes Desc: not available URL: From knepley at gmail.com Mon Apr 30 06:41:11 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 30 Apr 2018 07:41:11 -0400 Subject: [petsc-users] PDSLin hybrid solver performance with PETSc In-Reply-To: References: Message-ID: On Mon, Apr 30, 2018 at 7:20 AM, Afrah Najib wrote: > Hi all, > > I am testing PDSLin hybrid solver performance with PETSc krylov method on > a SPD matrix called audik_1 matrix(format is ijv full matrix ordered by > rows) > https://sparse.tamu.edu/GHS_psdef/audikw_1 > > I am also running on cluster with Intel(R) Xeon(R) CPU E5-2680 v4 @ > 2.40GHz processor and 28 core , 128 ; 512GB per node > 1) For any solver performance question, we need to see the output of -log_view -ksp_monitor_true_residual -ksp_converged_reason -ksp_view 2) Without knowing how many nodes (not cores) you are running on, we cannot evaluate the speedup, see http://www.mcs.anl.gov/petsc/documentation/faq.html#computers 3) Your SEGV is not occurring in a PETSc function. If it were, the signal handler would report the function name. Thanks, Matt but it takes a lot of time for Schur complement factorization. > > > Compiler Subdomains processors Precond TmatrixPartition TmatrixDist > Tfactorise TcompApproSchur TprepApproSchur Tfactoschur Tprecond_Total > Tsolve > gcc 8 128 Sparse 23.49532 0.6009915 6.059546 212.8243 0.04027608 38.74885 > 257.6796 13.69609 > gcc 8 16 Sparse 11.864 0.4385703 27.23499 1390.153 0.5523289 55.20706 > 1473.207 4.213825 > gcc 8 32 Sparse 9.912792 0.2592737 16.79125 934.3571 0.2372452 49.39175 > 1000.813 37.2029 > gcc 8 64 Sparse 12.61929 0.2336042 8.2316 632.839 0.1113939 52.21551 > 693.4153 13.94034 > > > My PETSc in stallation configuration is like this: > > export PETSC_DIR=/X/petsc > export PETSC_ARCH=arch-real > > ./configure PETSC_DIR=/X/petsc \ > prefix=/X/petsc/petsc-install \ > PETSC_ARCH=arch-real \ > --with-debugging=0 \ > --with-cc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc \ > --with-cxx=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicxx \ > --with-fc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 \ > --with-valgrind=1 \ > --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 \ > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 \ > --with-blas-lapack-dir=/Y/progs/intel/compilers_and_ > libraries_2017.0.098/linux/mkl/lib/intel64. > > *My slurm scrpt content is like this:* > > #!/bin/bash -l > #SBATCH -J "Paudik16" > #SBATCH -A X > #SBATCH -p X > #SBATCH --exclusive > #SBATCH -o Paudik_8_2_16.%j.out > #SBATCH -e Paudik_8_2_16.%j.err > > #SBATCH -N 8 > #SBATCH --ntasks-per-node=2 > #SBATCH --cpus-per-task=1 > > export KMP_AFFINITY=verbose,disabled > export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK} > export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} > > module purge > > module load gcc/7.1.0 > module load mpi/openmpi-3.0.0-gcc-7.1.0 > module load intel/mkl_2017.0.098 > > file=input.in > matrixFile=~/dataDir/ijv/audikw_1.ijv > > mpirun -np $SLURM_NTASKS ./dtest_pdslin $file $matrixFile > > > > *I attached an output of one file along with the input configuration of > PDSLin.* > > Any idea of how to enhance these results. > > Note in PDSLin, I tried with parallel Schur permutation but I got the > following error : > > [1]PETSC ERROR: ------------------------------ > ------------------------------------------ > [1]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [1]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS > /X to find memory corruption errors > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [1]PETSC ERROR: to get more information on the crash. > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > e/Xactly when Open MPI kills them. > -------------------------------------------------------------------------- > [13]PETSC ERROR: ------------------------------ > ------------------------------------------ > [13]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > [13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [13]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [13]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > [13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > [13]PETSC ERROR: to get more information on the crash. > [11]PETSC ERROR: ------------------------------ > ------------------------------------------ > [11]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > [11]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [11]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [11]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > [11]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > [11]PETSC ERROR: to get more information on the crash. > [3]PETSC ERROR: ------------------------------ > ------------------------------------------ > [3]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS > /X to find memory corruption errors > [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [3]PETSC ERROR: to get more information on the crash. > [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [3]PETSC ERROR: Signal received > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [3]PETSC ERROR: [9]PETSC ERROR: ------------------------------ > ------------------------------------------ > [9]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > [9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [9]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS > /X to find memory corruption errors > [9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [9]PETSC ERROR: to get more information on the crash. > [7]PETSC ERROR: ------------------------------ > ------------------------------------------ > [7]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [7]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS > /X to find memory corruption errors > [7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [7]PETSC ERROR: to get more information on the crash. > [5]PETSC ERROR: ------------------------------ > ------------------------------------------ > [5]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [5]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS > /X to find memory corruption errors > [5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [5]PETSC ERROR: to get more information on the crash. > [5]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [5]PETSC ERROR: Signal received > [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [5]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [5]PETSC ERROR: [15]PETSC ERROR: ------------------------------ > ------------------------------------------ > [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [15]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [13]PETSC ERROR: Signal received > [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [13]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [13]PETSC ERROR: ./dtest_pdslin on a named s060 by afarea Mon Apr 30 > 05:29:53 2018 > [13]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [11]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [11]PETSC ERROR: Signal received > [11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [11]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [11]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 > 05:29:53 2018 > [11]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 > [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [3]PETSC ERROR: #1 User provided function() line 0 in unknown file > [9]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [9]PETSC ERROR: Signal received > [9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [9]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [9]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 > 05:29:53 2018 > [9]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [7]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [7]PETSC ERROR: Signal received > [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [7]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [7]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 > 05:29:53 2018 > [7]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [7]PETSC ERROR: #1 User provided function() line 0 in unknown file > ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 > [5]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [5]PETSC ERROR: #1 User provided function() line 0 in unknown file > configure using --with-debugging=yes, recompile, link, and run > [15]PETSC ERROR: to get more information on the crash. > [15]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [15]PETSC ERROR: Signal received > [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 > 05:29:53 2018 > [13]PETSC ERROR: #1 User provided function() line 0 in unknown file > [11]PETSC ERROR: #1 User provided function() line 0 in unknown file > [9]PETSC ERROR: #1 User provided function() line 0 in unknown file > [15]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [15]PETSC ERROR: #1 User provided function() line 0 in unknown file > [1]PETSC ERROR: Signal received > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 > 05:29:53 2018 > [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 > 05:29:53 2018 > [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > [6]PETSC ERROR: ------------------------------ > ------------------------------------------ > [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [6]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS > /X to find memory corruption errors > [6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [6]PETSC ERROR: to get more information on the crash. > [6]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [6]PETSC ERROR: Signal received > [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [6]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [6]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 > 05:29:53 2018 > [6]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [6]PETSC ERROR: #1 User provided function() line 0 in unknown file > [4]PETSC ERROR: ------------------------------ > ------------------------------------------ > [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [4]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS > /X to find memory corruption errors > [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [4]PETSC ERROR: to get more information on the crash. > [4]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [4]PETSC ERROR: Signal received > [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 > 05:29:53 2018 > [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 > 05:29:53 2018 > [4]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [4]PETSC ERROR: #1 User provided function() line 0 in unknown file > [0]PETSC ERROR: ------------------------------ > ------------------------------------------ > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS > /X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [0]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 > 05:29:53 2018 > [0]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > [10]PETSC ERROR: ------------------------------ > ------------------------------------------ > [10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [10]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [10]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > [10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > [10]PETSC ERROR: to get more information on the crash. > [10]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [10]PETSC ERROR: Signal received > [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [10]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [10]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 > 05:29:53 2018 > > [8]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 > 05:29:53 2018 > [8]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [8]PETSC ERROR: #1 User provided function() line 0 in unknown file > [14]PETSC ERROR: ------------------------------ > ------------------------------------------ > [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [14]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > [14]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > [14]PETSC ERROR: to get more information on the crash. > [14]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [14]PETSC ERROR: Signal received > [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [14]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [14]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 > 05:29:53 2018 > [14]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [14]PETSC ERROR: #1 User provided function() line 0 in unknown file > [15]PETSC ERROR: ------------------------------ > ------------------------------------------ > [15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [15]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > [15]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > [15]PETSC ERROR: to get more information on the crash. > [15]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [15]PETSC ERROR: Signal received > [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 > 05:29:53 2018 > [15]PETSC ERROR: #2 User provided function() line 0 in unknown file > [2]PETSC ERROR: ------------------------------ > ------------------------------------------ > [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [2]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS > /X to find memory corruption errors > [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [2]PETSC ERROR: to get more information on the crash. > [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: Signal received > [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [2]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [2]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 > 05:29:53 2018 > [2]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [2]PETSC ERROR: #1 User provided function() line 0 in unknown file > [3]PETSC ERROR: ------------------------------ > ------------------------------------------ > [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS > /X to find memory corruption errors > [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [3]PETSC ERROR: to get more information on the crash. > [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [3]PETSC ERROR: Signal received > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [3]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 > 05:29:53 2018 > [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > [3]PETSC ERROR: #2 User provided function() line 0 in unknown file > [f002:38724] 17 more processes have sent help message help-mpi-api.t/Xt / > mpi-abort > [f002:38724] Set MCA parameter "orte_base_help_aggregate" to 0 to see all > help / error messages > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Apr 30 10:39:52 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 30 Apr 2018 15:39:52 +0000 Subject: [petsc-users] PDSLin hybrid solver performance with PETSc In-Reply-To: References: Message-ID: <2C3AF510-5926-4A8F-B754-A450E0C4081E@anl.gov> Looks like there is a divide by a variable that is zero in pdslin where the code doesn't first check if the value is not zero. You can try using the option -start_in_debugger and in each debugger window use the debugger option to catch floating point exceptions and then type cont to continue the run. Yes this is annoying when you have so many debugger windows but is the only simple way to determine immediately the line of code that is producing the floating point exception. Barry > On Apr 30, 2018, at 6:20 AM, Afrah Najib wrote: > > Hi all, > > I am testing PDSLin hybrid solver performance with PETSc krylov method on a SPD matrix called audik_1 matrix(format is ijv full matrix ordered by rows) > https://sparse.tamu.edu/GHS_psdef/audikw_1 > > I am also running on cluster with Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz processor and 28 core , 128 ; 512GB per node > > but it takes a lot of time for Schur complement factorization. > > Compiler Subdomains processors Precond TmatrixPartition TmatrixDist Tfactorise TcompApproSchur TprepApproSchur Tfactoschur Tprecond_Total Tsolve > gcc 8 128 Sparse 23.49532 0.6009915 6.059546 212.8243 0.04027608 38.74885 257.6796 13.69609 > gcc 8 16 Sparse 11.864 0.4385703 27.23499 1390.153 0.5523289 55.20706 1473.207 4.213825 > gcc 8 32 Sparse 9.912792 0.2592737 16.79125 934.3571 0.2372452 49.39175 1000.813 37.2029 > gcc 8 64 Sparse 12.61929 0.2336042 8.2316 632.839 0.1113939 52.21551 693.4153 13.94034 > > My PETSc in stallation configuration is like this: > > export PETSC_DIR=/X/petsc > export PETSC_ARCH=arch-real > > ./configure PETSC_DIR=/X/petsc \ > prefix=/X/petsc/petsc-install \ > PETSC_ARCH=arch-real \ > --with-debugging=0 \ > --with-cc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc \ > --with-cxx=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicxx \ > --with-fc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 \ > --with-valgrind=1 \ > --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 \ > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 \ > --with-blas-lapack-dir=/Y/progs/intel/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64. > > My slurm scrpt content is like this: > > #!/bin/bash -l > #SBATCH -J "Paudik16" > #SBATCH -A X > #SBATCH -p X > #SBATCH --exclusive > #SBATCH -o Paudik_8_2_16.%j.out > #SBATCH -e Paudik_8_2_16.%j.err > > #SBATCH -N 8 > #SBATCH --ntasks-per-node=2 > #SBATCH --cpus-per-task=1 > > export KMP_AFFINITY=verbose,disabled > export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK} > export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} > > module purge > > module load gcc/7.1.0 > module load mpi/openmpi-3.0.0-gcc-7.1.0 > module load intel/mkl_2017.0.098 > > file=input.in > matrixFile=~/dataDir/ijv/audikw_1.ijv > > mpirun -np $SLURM_NTASKS ./dtest_pdslin $file $matrixFile > > > > I attached an output of one file along with the input configuration of PDSLin. > > Any idea of how to enhance these results. > > Note in PDSLin, I tried with parallel Schur permutation but I got the following error : > > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [1]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [1]PETSC ERROR: to get more information on the crash. > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > e/Xactly when Open MPI kills them. > -------------------------------------------------------------------------- > [13]PETSC ERROR: ------------------------------------------------------------------------ > [13]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [13]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [13]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [13]PETSC ERROR: to get more information on the crash. > [11]PETSC ERROR: ------------------------------------------------------------------------ > [11]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [11]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [11]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [11]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [11]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [11]PETSC ERROR: to get more information on the crash. > [3]PETSC ERROR: ------------------------------------------------------------------------ > [3]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [3]PETSC ERROR: to get more information on the crash. > [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [3]PETSC ERROR: Signal received > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [3]PETSC ERROR: [9]PETSC ERROR: ------------------------------------------------------------------------ > [9]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [9]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [9]PETSC ERROR: to get more information on the crash. > [7]PETSC ERROR: ------------------------------------------------------------------------ > [7]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [7]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [7]PETSC ERROR: to get more information on the crash. > [5]PETSC ERROR: ------------------------------------------------------------------------ > [5]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [5]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [5]PETSC ERROR: to get more information on the crash. > [5]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [5]PETSC ERROR: Signal received > [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [5]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [5]PETSC ERROR: [15]PETSC ERROR: ------------------------------------------------------------------------ > [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [15]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [13]PETSC ERROR: Signal received > [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [13]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [13]PETSC ERROR: ./dtest_pdslin on a named s060 by afarea Mon Apr 30 05:29:53 2018 > [13]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [11]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [11]PETSC ERROR: Signal received > [11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [11]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [11]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 05:29:53 2018 > [11]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 > [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [3]PETSC ERROR: #1 User provided function() line 0 in unknown file > [9]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [9]PETSC ERROR: Signal received > [9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [9]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [9]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 05:29:53 2018 > [9]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [7]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [7]PETSC ERROR: Signal received > [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [7]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [7]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 05:29:53 2018 > [7]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [7]PETSC ERROR: #1 User provided function() line 0 in unknown file > ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 > [5]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [5]PETSC ERROR: #1 User provided function() line 0 in unknown file > configure using --with-debugging=yes, recompile, link, and run > [15]PETSC ERROR: to get more information on the crash. > [15]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [15]PETSC ERROR: Signal received > [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018 > [13]PETSC ERROR: #1 User provided function() line 0 in unknown file > [11]PETSC ERROR: #1 User provided function() line 0 in unknown file > [9]PETSC ERROR: #1 User provided function() line 0 in unknown file > [15]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [15]PETSC ERROR: #1 User provided function() line 0 in unknown file > [1]PETSC ERROR: Signal received > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018 > [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018 > [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > [6]PETSC ERROR: ------------------------------------------------------------------------ > [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [6]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [6]PETSC ERROR: to get more information on the crash. > [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [6]PETSC ERROR: Signal received > [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [6]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [6]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 05:29:53 2018 > [6]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [6]PETSC ERROR: #1 User provided function() line 0 in unknown file > [4]PETSC ERROR: ------------------------------------------------------------------------ > [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [4]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [4]PETSC ERROR: to get more information on the crash. > [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [4]PETSC ERROR: Signal received > [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 > [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 > [4]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [4]PETSC ERROR: #1 User provided function() line 0 in unknown file > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [0]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018 > [0]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > [10]PETSC ERROR: ------------------------------------------------------------------------ > [10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [10]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [10]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [10]PETSC ERROR: to get more information on the crash. > [10]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [10]PETSC ERROR: Signal received > [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [10]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [10]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 05:29:53 2018 > > [8]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 05:29:53 2018 > [8]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [8]PETSC ERROR: #1 User provided function() line 0 in unknown file > [14]PETSC ERROR: ------------------------------------------------------------------------ > [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [14]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [14]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [14]PETSC ERROR: to get more information on the crash. > [14]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [14]PETSC ERROR: Signal received > [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [14]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [14]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018 > [14]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [14]PETSC ERROR: #1 User provided function() line 0 in unknown file > [15]PETSC ERROR: ------------------------------------------------------------------------ > [15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [15]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [15]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [15]PETSC ERROR: to get more information on the crash. > [15]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [15]PETSC ERROR: Signal received > [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018 > [15]PETSC ERROR: #2 User provided function() line 0 in unknown file > [2]PETSC ERROR: ------------------------------------------------------------------------ > [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [2]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [2]PETSC ERROR: to get more information on the crash. > [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [2]PETSC ERROR: Signal received > [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [2]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [2]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 > [2]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [2]PETSC ERROR: #1 User provided function() line 0 in unknown file > [3]PETSC ERROR: ------------------------------------------------------------------------ > [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [3]PETSC ERROR: to get more information on the crash. > [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [3]PETSC ERROR: Signal received > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [3]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 > [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 > [3]PETSC ERROR: #2 User provided function() line 0 in unknown file > [f002:38724] 17 more processes have sent help message help-mpi-api.t/Xt / mpi-abort > [f002:38724] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages > > > From afrah.nacib at gmail.com Mon Apr 30 14:31:04 2018 From: afrah.nacib at gmail.com (Afrah Najib) Date: Mon, 30 Apr 2018 22:31:04 +0300 Subject: [petsc-users] Fwd: PDSLin hybrid solver performance with PETSc In-Reply-To: <2C3AF510-5926-4A8F-B754-A450E0C4081E@anl.gov> References: <2C3AF510-5926-4A8F-B754-A450E0C4081E@anl.gov> Message-ID: I beg your burden, how can I use those tools of PETSc in PDSLin program ... valgrind is working in such "mpirun -n $n valgrind ./dtest_pdslin $inputFIle $matrixFile " but these options are not working ---------- Forwarded message ---------- From: Smith, Barry F. Date: 30 April 2018 at 18:39 Subject: Re: [petsc-users] PDSLin hybrid solver performance with PETSc To: Afrah Najib Cc: PETSc , "Xiaoye S. Li" , Ichitaro Yamazaki , Ichitaro Yamazaki < ic.yamazaki at gmail.com> Looks like there is a divide by a variable that is zero in pdslin where the code doesn't first check if the value is not zero. You can try using the option -start_in_debugger and in each debugger window use the debugger option to catch floating point exceptions and then type cont to continue the run. Yes this is annoying when you have so many debugger windows but is the only simple way to determine immediately the line of code that is producing the floating point exception. Barry > On Apr 30, 2018, at 6:20 AM, Afrah Najib wrote: > > Hi all, > > I am testing PDSLin hybrid solver performance with PETSc krylov method on a SPD matrix called audik_1 matrix(format is ijv full matrix ordered by rows) > https://sparse.tamu.edu/GHS_psdef/audikw_1 > > I am also running on cluster with Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz processor and 28 core , 128 ; 512GB per node > > but it takes a lot of time for Schur complement factorization. > > Compiler Subdomains processors Precond TmatrixPartition TmatrixDist Tfactorise TcompApproSchur TprepApproSchur Tfactoschur Tprecond_Total Tsolve > gcc 8 128 Sparse 23.49532 0.6009915 6.059546 212.8243 0.04027608 38.74885 257.6796 13.69609 > gcc 8 16 Sparse 11.864 0.4385703 27.23499 1390.153 0.5523289 55.20706 1473.207 4.213825 > gcc 8 32 Sparse 9.912792 0.2592737 16.79125 934.3571 0.2372452 49.39175 1000.813 37.2029 > gcc 8 64 Sparse 12.61929 0.2336042 8.2316 632.839 0.1113939 52.21551 693.4153 13.94034 > > My PETSc in stallation configuration is like this: > > export PETSC_DIR=/X/petsc > export PETSC_ARCH=arch-real > > ./configure PETSC_DIR=/X/petsc \ > prefix=/X/petsc/petsc-install \ > PETSC_ARCH=arch-real \ > --with-debugging=0 \ > --with-cc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc \ > --with-cxx=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicxx \ > --with-fc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 \ > --with-valgrind=1 \ > --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 \ > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 \ > --with-blas-lapack-dir=/Y/progs/intel/compilers_and_ libraries_2017.0.098/linux/mkl/lib/intel64. > > My slurm scrpt content is like this: > > #!/bin/bash -l > #SBATCH -J "Paudik16" > #SBATCH -A X > #SBATCH -p X > #SBATCH --exclusive > #SBATCH -o Paudik_8_2_16.%j.out > #SBATCH -e Paudik_8_2_16.%j.err > > #SBATCH -N 8 > #SBATCH --ntasks-per-node=2 > #SBATCH --cpus-per-task=1 > > export KMP_AFFINITY=verbose,disabled > export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK} > export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} > > module purge > > module load gcc/7.1.0 > module load mpi/openmpi-3.0.0-gcc-7.1.0 > module load intel/mkl_2017.0.098 > > file=input.in > matrixFile=~/dataDir/ijv/audikw_1.ijv > > mpirun -np $SLURM_NTASKS ./dtest_pdslin $file $matrixFile > > > > I attached an output of one file along with the input configuration of PDSLin. > > Any idea of how to enhance these results. > > Note in PDSLin, I tried with parallel Schur permutation but I got the following error : > > [1]PETSC ERROR: ------------------------------ ------------------------------------------ > [1]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [1]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [1]PETSC ERROR: to get more information on the crash. > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > e/Xactly when Open MPI kills them. > -------------------------------------------------------------------------- > [13]PETSC ERROR: ------------------------------ ------------------------------------------ > [13]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [13]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [13]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [13]PETSC ERROR: to get more information on the crash. > [11]PETSC ERROR: ------------------------------ ------------------------------------------ > [11]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [11]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [11]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [11]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [11]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [11]PETSC ERROR: to get more information on the crash. > [3]PETSC ERROR: ------------------------------ ------------------------------------------ > [3]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [3]PETSC ERROR: to get more information on the crash. > [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [3]PETSC ERROR: Signal received > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [3]PETSC ERROR: [9]PETSC ERROR: ------------------------------ ------------------------------------------ > [9]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [9]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [9]PETSC ERROR: to get more information on the crash. > [7]PETSC ERROR: ------------------------------ ------------------------------------------ > [7]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [7]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [7]PETSC ERROR: to get more information on the crash. > [5]PETSC ERROR: ------------------------------ ------------------------------------------ > [5]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [5]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [5]PETSC ERROR: to get more information on the crash. > [5]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [5]PETSC ERROR: Signal received > [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [5]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [5]PETSC ERROR: [15]PETSC ERROR: ------------------------------ ------------------------------------------ > [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero > [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [15]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [13]PETSC ERROR: Signal received > [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [13]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [13]PETSC ERROR: ./dtest_pdslin on a named s060 by afarea Mon Apr 30 05:29:53 2018 > [13]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [11]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [11]PETSC ERROR: Signal received > [11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [11]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [11]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 05:29:53 2018 > [11]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 > [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [3]PETSC ERROR: #1 User provided function() line 0 in unknown file > [9]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [9]PETSC ERROR: Signal received > [9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [9]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [9]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 05:29:53 2018 > [9]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [7]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [7]PETSC ERROR: Signal received > [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [7]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [7]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 05:29:53 2018 > [7]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [7]PETSC ERROR: #1 User provided function() line 0 in unknown file > ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 > [5]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [5]PETSC ERROR: #1 User provided function() line 0 in unknown file > configure using --with-debugging=yes, recompile, link, and run > [15]PETSC ERROR: to get more information on the crash. > [15]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [15]PETSC ERROR: Signal received > [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018 > [13]PETSC ERROR: #1 User provided function() line 0 in unknown file > [11]PETSC ERROR: #1 User provided function() line 0 in unknown file > [9]PETSC ERROR: #1 User provided function() line 0 in unknown file > [15]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [15]PETSC ERROR: #1 User provided function() line 0 in unknown file > [1]PETSC ERROR: Signal received > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018 > [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018 > [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > [6]PETSC ERROR: ------------------------------ ------------------------------------------ > [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [6]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [6]PETSC ERROR: to get more information on the crash. > [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [6]PETSC ERROR: Signal received > [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [6]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [6]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 05:29:53 2018 > [6]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [6]PETSC ERROR: #1 User provided function() line 0 in unknown file > [4]PETSC ERROR: ------------------------------ ------------------------------------------ > [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [4]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [4]PETSC ERROR: to get more information on the crash. > [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [4]PETSC ERROR: Signal received > [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 > [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 > [4]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [4]PETSC ERROR: #1 User provided function() line 0 in unknown file > [0]PETSC ERROR: ------------------------------ ------------------------------------------ > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [0]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018 > [0]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > [10]PETSC ERROR: ------------------------------ ------------------------------------------ > [10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [10]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [10]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [10]PETSC ERROR: to get more information on the crash. > [10]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [10]PETSC ERROR: Signal received > [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [10]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [10]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 05:29:53 2018 > > [8]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 05:29:53 2018 > [8]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [8]PETSC ERROR: #1 User provided function() line 0 in unknown file > [14]PETSC ERROR: ------------------------------ ------------------------------------------ > [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [14]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [14]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [14]PETSC ERROR: to get more information on the crash. > [14]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [14]PETSC ERROR: Signal received > [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [14]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [14]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018 > [14]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [14]PETSC ERROR: #1 User provided function() line 0 in unknown file > [15]PETSC ERROR: ------------------------------ ------------------------------------------ > [15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [15]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [15]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [15]PETSC ERROR: to get more information on the crash. > [15]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [15]PETSC ERROR: Signal received > [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018 > [15]PETSC ERROR: #2 User provided function() line 0 in unknown file > [2]PETSC ERROR: ------------------------------ ------------------------------------------ > [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [2]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [2]PETSC ERROR: to get more information on the crash. > [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [2]PETSC ERROR: Signal received > [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [2]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [2]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 > [2]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [2]PETSC ERROR: #1 User provided function() line 0 in unknown file > [3]PETSC ERROR: ------------------------------ ------------------------------------------ > [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ documentation/faq.html#valgrind > [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors > [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [3]PETSC ERROR: to get more information on the crash. > [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [3]PETSC ERROR: Signal received > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown > [3]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 > [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ X/mkl/lib/intel64 > [3]PETSC ERROR: #2 User provided function() line 0 in unknown file > [f002:38724] 17 more processes have sent help message help-mpi-api.t/Xt / mpi-abort > [f002:38724] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From afrah.nacib at gmail.com Mon Apr 30 15:56:23 2018 From: afrah.nacib at gmail.com (Afrah Najib) Date: Mon, 30 Apr 2018 23:56:23 +0300 Subject: [petsc-users] petsc-users Digest, Vol 112, Issue 141 In-Reply-To: References: Message-ID: This word is wrong burden ... pardon me ... I am sorry ... There is no edit here On 30 April 2018 at 22:31, wrote: > Send petsc-users mailing list submissions to > petsc-users at mcs.anl.gov > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > or, via email, send a message with subject or body 'help' to > petsc-users-request at mcs.anl.gov > > You can reach the person managing the list at > petsc-users-owner at mcs.anl.gov > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of petsc-users digest..." > > > Today's Topics: > > 1. Fwd: PDSLin hybrid solver performance with PETSc (Afrah Najib) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 30 Apr 2018 22:31:04 +0300 > From: Afrah Najib > To: PETSc > Subject: [petsc-users] Fwd: PDSLin hybrid solver performance with > PETSc > Message-ID: > gmail.com> > Content-Type: text/plain; charset="utf-8" > > I beg your burden, how can I use those tools of PETSc in PDSLin program ... > > valgrind is working in such "mpirun -n $n valgrind ./dtest_pdslin > $inputFIle $matrixFile " > > but these options are not working > > ---------- Forwarded message ---------- > From: Smith, Barry F. > Date: 30 April 2018 at 18:39 > Subject: Re: [petsc-users] PDSLin hybrid solver performance with PETSc > To: Afrah Najib > Cc: PETSc , "Xiaoye S. Li" , > Ichitaro Yamazaki , Ichitaro Yamazaki < > ic.yamazaki at gmail.com> > > > > Looks like there is a divide by a variable that is zero in pdslin where > the code doesn't first check if the value is not zero. > > You can try using the option -start_in_debugger and in each debugger > window use the debugger option to catch floating point exceptions and then > type cont to continue the run. Yes this is annoying when you have so many > debugger windows but is the only simple way to determine immediately the > line of code that is producing the floating point exception. > > Barry > > > > On Apr 30, 2018, at 6:20 AM, Afrah Najib wrote: > > > > Hi all, > > > > I am testing PDSLin hybrid solver performance with PETSc krylov method on > a SPD matrix called audik_1 matrix(format is ijv full matrix ordered by > rows) > > https://sparse.tamu.edu/GHS_psdef/audikw_1 > > > > I am also running on cluster with Intel(R) Xeon(R) CPU E5-2680 v4 @ > 2.40GHz processor and 28 core , 128 ; 512GB per node > > > > but it takes a lot of time for Schur complement factorization. > > > > Compiler Subdomains processors Precond TmatrixPartition > TmatrixDist Tfactorise TcompApproSchur TprepApproSchur > Tfactoschur Tprecond_Total Tsolve > > gcc 8 128 Sparse 23.49532 0.6009915 6.059546 > 212.8243 0.04027608 38.74885 257.6796 > 13.69609 > > gcc 8 16 Sparse 11.864 0.4385703 27.23499 > 1390.153 0.5523289 55.20706 1473.207 4.213825 > > gcc 8 32 Sparse 9.912792 0.2592737 16.79125 > 934.3571 0.2372452 49.39175 1000.813 37.2029 > > gcc 8 64 Sparse 12.61929 0.2336042 8.2316 > 632.839 0.1113939 52.21551 693.4153 13.94034 > > > > My PETSc in stallation configuration is like this: > > > > export PETSC_DIR=/X/petsc > > export PETSC_ARCH=arch-real > > > > ./configure PETSC_DIR=/X/petsc \ > > prefix=/X/petsc/petsc-install \ > > PETSC_ARCH=arch-real \ > > --with-debugging=0 \ > > --with-cc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc \ > > --with-cxx=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicxx \ > > --with-fc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 \ > > --with-valgrind=1 \ > > --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 \ > > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 \ > > --with-blas-lapack-dir=/Y/progs/intel/compilers_and_ > libraries_2017.0.098/linux/mkl/lib/intel64. > > > > My slurm scrpt content is like this: > > > > #!/bin/bash -l > > #SBATCH -J "Paudik16" > > #SBATCH -A X > > #SBATCH -p X > > #SBATCH --exclusive > > #SBATCH -o Paudik_8_2_16.%j.out > > #SBATCH -e Paudik_8_2_16.%j.err > > > > #SBATCH -N 8 > > #SBATCH --ntasks-per-node=2 > > #SBATCH --cpus-per-task=1 > > > > export KMP_AFFINITY=verbose,disabled > > export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK} > > export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} > > > > module purge > > > > module load gcc/7.1.0 > > module load mpi/openmpi-3.0.0-gcc-7.1.0 > > module load intel/mkl_2017.0.098 > > > > file=input.in > > matrixFile=~/dataDir/ijv/audikw_1.ijv > > > > mpirun -np $SLURM_NTASKS ./dtest_pdslin $file $matrixFile > > > > > > > > I attached an output of one file along with the input configuration of > PDSLin. > > > > Any idea of how to enhance these results. > > > > Note in PDSLin, I tried with parallel Schur permutation but I got the > following error : > > > > [1]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [1]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [1]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [1]PETSC ERROR: to get more information on the crash. > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > ------------------------------------------------------------ > -------------- > > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD > > with errorcode 59. > > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > > You may or may not see output from other processes, depending on > > e/Xactly when Open MPI kills them. > > ------------------------------------------------------------ > -------------- > > [13]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [13]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [13]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [13]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [13]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > > [13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [13]PETSC ERROR: to get more information on the crash. > > [11]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [11]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [11]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [11]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [11]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > > [11]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [11]PETSC ERROR: to get more information on the crash. > > [3]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [3]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [3]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [3]PETSC ERROR: to get more information on the crash. > > [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [3]PETSC ERROR: Signal received > > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [3]PETSC ERROR: [9]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [9]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [9]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [9]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [9]PETSC ERROR: to get more information on the crash. > > [7]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [7]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [7]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [7]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [7]PETSC ERROR: to get more information on the crash. > > [5]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [5]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [5]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [5]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [5]PETSC ERROR: to get more information on the crash. > > [5]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [5]PETSC ERROR: Signal received > > [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [5]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [5]PETSC ERROR: [15]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [15]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [15]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [13]PETSC ERROR: Signal received > > [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [13]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [13]PETSC ERROR: ./dtest_pdslin on a named s060 by afarea Mon Apr 30 > 05:29:53 2018 > > [13]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [11]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [11]PETSC ERROR: Signal received > > [11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [11]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [11]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 > 05:29:53 2018 > > [11]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 > > [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [3]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [9]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [9]PETSC ERROR: Signal received > > [9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [9]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [9]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 > 05:29:53 2018 > > [9]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [7]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [7]PETSC ERROR: Signal received > > [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [7]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [7]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 > 05:29:53 2018 > > [7]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [7]PETSC ERROR: #1 User provided function() line 0 in unknown file > > ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 > > [5]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [5]PETSC ERROR: #1 User provided function() line 0 in unknown file > > configure using --with-debugging=yes, recompile, link, and run > > [15]PETSC ERROR: to get more information on the crash. > > [15]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [15]PETSC ERROR: Signal received > > [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 > 05:29:53 2018 > > [13]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [11]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [9]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [15]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [15]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [1]PETSC ERROR: Signal received > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 > 05:29:53 2018 > > [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 > 05:29:53 2018 > > [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [6]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [6]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [6]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [6]PETSC ERROR: to get more information on the crash. > > [6]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [6]PETSC ERROR: Signal received > > [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [6]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [6]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 > 05:29:53 2018 > > [6]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [6]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [4]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [4]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [4]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [4]PETSC ERROR: to get more information on the crash. > > [4]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [4]PETSC ERROR: Signal received > > [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 > 05:29:53 2018 > > [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 > 05:29:53 2018 > > [4]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [4]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [0]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [0]PETSC ERROR: to get more information on the crash. > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Signal received > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [0]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 > 05:29:53 2018 > > [0]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [10]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [10]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [10]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [10]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > > [10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [10]PETSC ERROR: to get more information on the crash. > > [10]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [10]PETSC ERROR: Signal received > > [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [10]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [10]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 > 05:29:53 2018 > > > > [8]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 > 05:29:53 2018 > > [8]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [8]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [14]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [14]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [14]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > > [14]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [14]PETSC ERROR: to get more information on the crash. > > [14]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [14]PETSC ERROR: Signal received > > [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [14]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [14]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 > 05:29:53 2018 > > [14]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [14]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [15]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [15]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [15]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > > [15]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [15]PETSC ERROR: to get more information on the crash. > > [15]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [15]PETSC ERROR: Signal received > > [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 > 05:29:53 2018 > > [15]PETSC ERROR: #2 User provided function() line 0 in unknown file > > [2]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [2]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [2]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [2]PETSC ERROR: to get more information on the crash. > > [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [2]PETSC ERROR: Signal received > > [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [2]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [2]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 > 05:29:53 2018 > > [2]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [2]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [3]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [3]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [3]PETSC ERROR: to get more information on the crash. > > [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [3]PETSC ERROR: Signal received > > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [3]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 > 05:29:53 2018 > > [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [3]PETSC ERROR: #2 User provided function() line 0 in unknown file > > [f002:38724] 17 more processes have sent help message help-mpi-api.t/Xt / > mpi-abort > > [f002:38724] Set MCA parameter "orte_base_help_aggregate" to 0 to see all > help / error messages > > > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: attachments/20180430/94bc4793/attachment.html> > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > petsc-users mailing list > petsc-users at mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > > > ------------------------------ > > End of petsc-users Digest, Vol 112, Issue 141 > ********************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: