From knepley at gmail.com Sat Nov 1 11:40:51 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 1 Nov 2014 11:40:51 -0500 Subject: [petsc-users] Reading data from a file into a DM created vector In-Reply-To: <491840EA-7604-4E78-B152-4A9C96F3A691@gmail.com> References: <491840EA-7604-4E78-B152-4A9C96F3A691@gmail.com> Message-ID: On Fri, Oct 31, 2014 at 4:15 PM, Justin Chang wrote: > Matt, > > Thanks for the response. One more question: > > If I go with approach 2, will manually setting the constraints/indices and > its values be compatible with the DMPlexSNESComputeResidual/JacobianFEM > routines? When I look at the source code of those routines it seems the > constrained values are added to the local solution vectors via > DMPlexInsertBoundaryValuesFEM. If I choose not to use DMPlexAddBoundary and > wish to manually declare my boundary values, i assume I should call > DMPlexProjectField with mode INSERT_BC_VALUES? > Yes, exactly. I use DMPlexProjectFunctionLabelLocal(), https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plexfem.c?at=master#cl-576 Thanks, Matt > Thanks, > Justin > > Sent from my iPhone > > On Oct 30, 2014, at 4:24 PM, Matthew Knepley wrote: > > On Thu, Oct 30, 2014 at 4:16 PM, Justin Chang wrote: > >> Matt, thanks for the quick response. >> >> What about for (dirichlet) boundary conditions? Would it be possible to >> do something similar for those, like using those PetscSectionSetConstraint >> functions? >> > > Yes. There are generally two ways of handling Dirichlet conditions: > > 1) Replace those rows of the Jacobian with the identity and put the > boundary value in the rhs. I find > this cumbersome for nonlinear solves. > > 2) Remove these unknowns from the system using > PetscSectionSetConstraintDof() and ConstratinIndices(). > > The DMPlexAddBoundary() functions are automating this processing by > marking boundaries using DMLabels, > and then constraining dofs on those boundaries. > > Thanks, > > Matt > > >> Thanks, >> Justin >> >> On Thu, Oct 30, 2014 at 3:35 PM, Matthew Knepley >> wrote: >> >>> On Thu, Oct 30, 2014 at 3:29 PM, Justin Chang >>> wrote: >>> >>>> Hi all, >>>> >>>> So I am writing an FEM code where it reads input data (e.g., auxiliary >>>> coefficients, source terms, etc) from a text file. I have preprocessed the >>>> data text file so that each vertex point has its corresponding data. For >>>> instance, if my source term for a diffusion problem has a sin or cos >>>> function of the coordinates, then this data is already tabulated and simply >>>> needs to be fed into my PETSc program. The data text file containing both >>>> the auxiliary data and the coordinate/connectivities will also be used to >>>> provide the input arguments for the DMPlexCreateFromDAG() function. >>>> >>>> Normally, I would hard-code the sin/cos functions into the source code >>>> itself, but i want my program to be "universal" in that it can take as >>>> input any user-defined function for not only these auxiliaries but for >>>> other things like boundary conditions. I see that PETSc has a lot of >>>> examples on how to read data into vectors and matrices, but I guess my >>>> question is is there a way to project data from a text file into a vector >>>> that has already been created with a defined DM structure? >>>> >>> >>> If you have the functions available, I think it is far more universal to >>> use the function itself, since then you can be independent >>> of mesh and discretization when specifying input and BC. >>> >>> However, if you want to read it in, and you guarantee that it matches >>> the mesh and discretization, I think the easiest thing to do is >>> demand that it come in the same order as the vertices and use >>> >>> VecGetArray(V, &a); >>> for (v = vStart, i = 0; v < vEnd; ++v) { >>> PetscSectionGetDof(s, v, &dof); >>> PetscSectionGetOffset(s, v, &off); >>> for (d = 0; d < dof; ++d) a[off+d] = text_data[i++]; >>> } >>> VecRestoreArray(V, &a); >>> >>> Matt >>> >>> >>>> Thanks, >>>> Justin >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Sat Nov 1 12:20:29 2014 From: jychang48 at gmail.com (Justin Chang) Date: Sat, 1 Nov 2014 12:20:29 -0500 Subject: [petsc-users] Reading data from a file into a DM created vector In-Reply-To: References: <491840EA-7604-4E78-B152-4A9C96F3A691@gmail.com> Message-ID: Thanks for the response. On a related note, In the source code of SNES ex12, where is the PetscSection actually being created? I can't seem to find anywhere in the code or its routines where the creation has been explicitly called. Because when the discretization/problem is define, I imagine a PetscSection for the fields and constraints has to be invoked somewhere. It seems to me as soon as DMPlexAddBoundary() is called it creates the PetscSectionConstraints but I can't seem to really confirm this within the source of that function. The reason I want to know this is because if I really were to declare my own constraints, I want to make sure that the PetscSection for the field variables has already been set up. Thanks, Justin Sent from my iPhone > On Nov 1, 2014, at 11:40 AM, Matthew Knepley wrote: > >> On Fri, Oct 31, 2014 at 4:15 PM, Justin Chang wrote: >> Matt, >> >> Thanks for the response. One more question: >> >> If I go with approach 2, will manually setting the constraints/indices and its values be compatible with the DMPlexSNESComputeResidual/JacobianFEM routines? When I look at the source code of those routines it seems the constrained values are added to the local solution vectors via DMPlexInsertBoundaryValuesFEM. If I choose not to use DMPlexAddBoundary and wish to manually declare my boundary values, i assume I should call DMPlexProjectField with mode INSERT_BC_VALUES? > > Yes, exactly. I use DMPlexProjectFunctionLabelLocal(), > > https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plexfem.c?at=master#cl-576 > > Thanks, > > Matt > >> Thanks, >> Justin >> >> Sent from my iPhone >> >>> On Oct 30, 2014, at 4:24 PM, Matthew Knepley wrote: >>> >>>> On Thu, Oct 30, 2014 at 4:16 PM, Justin Chang wrote: >>>> Matt, thanks for the quick response. >>>> >>>> What about for (dirichlet) boundary conditions? Would it be possible to do something similar for those, like using those PetscSectionSetConstraint functions? >>> >>> Yes. There are generally two ways of handling Dirichlet conditions: >>> >>> 1) Replace those rows of the Jacobian with the identity and put the boundary value in the rhs. I find >>> this cumbersome for nonlinear solves. >>> >>> 2) Remove these unknowns from the system using PetscSectionSetConstraintDof() and ConstratinIndices(). >>> >>> The DMPlexAddBoundary() functions are automating this processing by marking boundaries using DMLabels, >>> and then constraining dofs on those boundaries. >>> >>> Thanks, >>> >>> Matt >>> >>>> Thanks, >>>> Justin >>>> >>>>> On Thu, Oct 30, 2014 at 3:35 PM, Matthew Knepley wrote: >>>>>> On Thu, Oct 30, 2014 at 3:29 PM, Justin Chang wrote: >>>>>> Hi all, >>>>>> >>>>>> So I am writing an FEM code where it reads input data (e.g., auxiliary coefficients, source terms, etc) from a text file. I have preprocessed the data text file so that each vertex point has its corresponding data. For instance, if my source term for a diffusion problem has a sin or cos function of the coordinates, then this data is already tabulated and simply needs to be fed into my PETSc program. The data text file containing both the auxiliary data and the coordinate/connectivities will also be used to provide the input arguments for the DMPlexCreateFromDAG() function. >>>>>> >>>>>> Normally, I would hard-code the sin/cos functions into the source code itself, but i want my program to be "universal" in that it can take as input any user-defined function for not only these auxiliaries but for other things like boundary conditions. I see that PETSc has a lot of examples on how to read data into vectors and matrices, but I guess my question is is there a way to project data from a text file into a vector that has already been created with a defined DM structure? >>>>> >>>>> If you have the functions available, I think it is far more universal to use the function itself, since then you can be independent >>>>> of mesh and discretization when specifying input and BC. >>>>> >>>>> However, if you want to read it in, and you guarantee that it matches the mesh and discretization, I think the easiest thing to do is >>>>> demand that it come in the same order as the vertices and use >>>>> >>>>> VecGetArray(V, &a); >>>>> for (v = vStart, i = 0; v < vEnd; ++v) { >>>>> PetscSectionGetDof(s, v, &dof); >>>>> PetscSectionGetOffset(s, v, &off); >>>>> for (d = 0; d < dof; ++d) a[off+d] = text_data[i++]; >>>>> } >>>>> VecRestoreArray(V, &a); >>>>> >>>>> Matt >>>>> >>>>>> Thanks, >>>>>> Justin >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Nov 1 13:30:24 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 1 Nov 2014 13:30:24 -0500 Subject: [petsc-users] Reading data from a file into a DM created vector In-Reply-To: References: <491840EA-7604-4E78-B152-4A9C96F3A691@gmail.com> Message-ID: On Sat, Nov 1, 2014 at 12:20 PM, Justin Chang wrote: > Thanks for the response. On a related note, > > In the source code of SNES ex12, where is the PetscSection actually being > created? I can't seem to find anywhere in the code or its routines where > the creation has been explicitly called. Because when the > discretization/problem is define, I imagine a PetscSection for the fields > and constraints has to be invoked somewhere. It seems to me as soon as > DMPlexAddBoundary() is called it creates the PetscSectionConstraints but I > can't seem to really confirm this within the source of that function. The > reason I want to know this is because if I really were to declare my own > constraints, I want to make sure that the PetscSection for the field > variables has already been set up. > Much of the mechanism has been moved into the library. The reason I did this was to make things like nonlinear multigrid automatic, rather than making the user define a different section on each level. Let me explain what is happening. When we call DMGetDefaultSection(), if the section is not present it will call the createdefaultsection() member function: https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/interface/dm.c?at=master#cl-2977 which for Plex is here https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plex.c?at=master#cl-5601 It uses the discretization information from PetscDS and the BC information from DMPlexAddBoundary() to build the inputs for DMPlexCreateSection() https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plex.c?at=master#cl-5739 I realize that call is still limited. For example, it constrains all dofs on a point, but sometimes you only want to constrain some. In the most general case, you would just build the Section manually, using the same tools I use in DMPlexCreateSection(): https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plex.c?at=master#cl-3204 Thanks, Matt > Thanks, > Justin > > > Sent from my iPhone > > On Nov 1, 2014, at 11:40 AM, Matthew Knepley wrote: > > On Fri, Oct 31, 2014 at 4:15 PM, Justin Chang wrote: > >> Matt, >> >> Thanks for the response. One more question: >> >> If I go with approach 2, will manually setting the constraints/indices >> and its values be compatible with the DMPlexSNESComputeResidual/JacobianFEM >> routines? When I look at the source code of those routines it seems the >> constrained values are added to the local solution vectors via >> DMPlexInsertBoundaryValuesFEM. If I choose not to use DMPlexAddBoundary and >> wish to manually declare my boundary values, i assume I should call >> DMPlexProjectField with mode INSERT_BC_VALUES? >> > > Yes, exactly. I use DMPlexProjectFunctionLabelLocal(), > > > https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plexfem.c?at=master#cl-576 > > Thanks, > > Matt > > >> Thanks, >> Justin >> >> Sent from my iPhone >> >> On Oct 30, 2014, at 4:24 PM, Matthew Knepley wrote: >> >> On Thu, Oct 30, 2014 at 4:16 PM, Justin Chang >> wrote: >> >>> Matt, thanks for the quick response. >>> >>> What about for (dirichlet) boundary conditions? Would it be possible to >>> do something similar for those, like using those PetscSectionSetConstraint >>> functions? >>> >> >> Yes. There are generally two ways of handling Dirichlet conditions: >> >> 1) Replace those rows of the Jacobian with the identity and put the >> boundary value in the rhs. I find >> this cumbersome for nonlinear solves. >> >> 2) Remove these unknowns from the system using >> PetscSectionSetConstraintDof() and ConstratinIndices(). >> >> The DMPlexAddBoundary() functions are automating this processing by >> marking boundaries using DMLabels, >> and then constraining dofs on those boundaries. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Justin >>> >>> On Thu, Oct 30, 2014 at 3:35 PM, Matthew Knepley >>> wrote: >>> >>>> On Thu, Oct 30, 2014 at 3:29 PM, Justin Chang >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> So I am writing an FEM code where it reads input data (e.g., auxiliary >>>>> coefficients, source terms, etc) from a text file. I have preprocessed the >>>>> data text file so that each vertex point has its corresponding data. For >>>>> instance, if my source term for a diffusion problem has a sin or cos >>>>> function of the coordinates, then this data is already tabulated and simply >>>>> needs to be fed into my PETSc program. The data text file containing both >>>>> the auxiliary data and the coordinate/connectivities will also be used to >>>>> provide the input arguments for the DMPlexCreateFromDAG() function. >>>>> >>>>> Normally, I would hard-code the sin/cos functions into the source code >>>>> itself, but i want my program to be "universal" in that it can take as >>>>> input any user-defined function for not only these auxiliaries but for >>>>> other things like boundary conditions. I see that PETSc has a lot of >>>>> examples on how to read data into vectors and matrices, but I guess my >>>>> question is is there a way to project data from a text file into a vector >>>>> that has already been created with a defined DM structure? >>>>> >>>> >>>> If you have the functions available, I think it is far more universal >>>> to use the function itself, since then you can be independent >>>> of mesh and discretization when specifying input and BC. >>>> >>>> However, if you want to read it in, and you guarantee that it matches >>>> the mesh and discretization, I think the easiest thing to do is >>>> demand that it come in the same order as the vertices and use >>>> >>>> VecGetArray(V, &a); >>>> for (v = vStart, i = 0; v < vEnd; ++v) { >>>> PetscSectionGetDof(s, v, &dof); >>>> PetscSectionGetOffset(s, v, &off); >>>> for (d = 0; d < dof; ++d) a[off+d] = text_data[i++]; >>>> } >>>> VecRestoreArray(V, &a); >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> Justin >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Sat Nov 1 19:22:50 2014 From: jychang48 at gmail.com (Justin Chang) Date: Sat, 1 Nov 2014 19:22:50 -0500 Subject: [petsc-users] Reading data from a file into a DM created vector In-Reply-To: References: <491840EA-7604-4E78-B152-4A9C96F3A691@gmail.com> Message-ID: Matt, Okay that makes a lot of sense now, thank you so much. Sorry I have one more question then I will leave you all alone :) >From what I understand about creating DMPlex, it seems that if there are multiple processes, only the root rank populates the DM structure. I see that DMPlexCreateBoxMesh as well as the GMSH/Exodus/CGNS readers have rank 0 read the mesh files and populate the sieve/mesh points within the DMPlex (assuming that the DM was created with PETSC_COMM_WORLD). However, when I look at the DMPlexCreateFromDAG and DMPlexCreateFromCellList functions, I don't see anywhere where only rank 0 handles the creation of the DM structure. I do notice that DMPlexCreateFromDAG already requires the DM structure to be created. So I guess my question is, is it possible to simply have rank 0 invoke the DMPlexCreateFromDAG function? Because that way only the root process reads the mesh data file and creates/distributes the DMPlex among the remaining processors. Thanks, Justin On Sat, Nov 1, 2014 at 1:30 PM, Matthew Knepley wrote: > On Sat, Nov 1, 2014 at 12:20 PM, Justin Chang wrote: > >> Thanks for the response. On a related note, >> >> In the source code of SNES ex12, where is the PetscSection actually being >> created? I can't seem to find anywhere in the code or its routines where >> the creation has been explicitly called. Because when the >> discretization/problem is define, I imagine a PetscSection for the fields >> and constraints has to be invoked somewhere. It seems to me as soon as >> DMPlexAddBoundary() is called it creates the PetscSectionConstraints but I >> can't seem to really confirm this within the source of that function. The >> reason I want to know this is because if I really were to declare my own >> constraints, I want to make sure that the PetscSection for the field >> variables has already been set up. >> > > Much of the mechanism has been moved into the library. The reason I did > this was to make things like > nonlinear multigrid automatic, rather than making the user define a > different section on each level. Let > me explain what is happening. When we call DMGetDefaultSection(), if the > section is not present it > will call the createdefaultsection() member function: > > > https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/interface/dm.c?at=master#cl-2977 > > which for Plex is here > > > https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plex.c?at=master#cl-5601 > > It uses the discretization information from PetscDS and the BC information > from DMPlexAddBoundary() to build the inputs for > DMPlexCreateSection() > > > https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plex.c?at=master#cl-5739 > > I realize that call is still limited. For example, it constrains all dofs > on a point, but sometimes you only want to constrain some. In > the most general case, you would just build the Section manually, using > the same tools I use in DMPlexCreateSection(): > > > https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plex.c?at=master#cl-3204 > > Thanks, > > Matt > > >> Thanks, >> Justin >> >> >> Sent from my iPhone >> >> On Nov 1, 2014, at 11:40 AM, Matthew Knepley wrote: >> >> On Fri, Oct 31, 2014 at 4:15 PM, Justin Chang >> wrote: >> >>> Matt, >>> >>> Thanks for the response. One more question: >>> >>> If I go with approach 2, will manually setting the constraints/indices >>> and its values be compatible with the DMPlexSNESComputeResidual/JacobianFEM >>> routines? When I look at the source code of those routines it seems the >>> constrained values are added to the local solution vectors via >>> DMPlexInsertBoundaryValuesFEM. If I choose not to use DMPlexAddBoundary and >>> wish to manually declare my boundary values, i assume I should call >>> DMPlexProjectField with mode INSERT_BC_VALUES? >>> >> >> Yes, exactly. I use DMPlexProjectFunctionLabelLocal(), >> >> >> https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plexfem.c?at=master#cl-576 >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Justin >>> >>> Sent from my iPhone >>> >>> On Oct 30, 2014, at 4:24 PM, Matthew Knepley wrote: >>> >>> On Thu, Oct 30, 2014 at 4:16 PM, Justin Chang >>> wrote: >>> >>>> Matt, thanks for the quick response. >>>> >>>> What about for (dirichlet) boundary conditions? Would it be possible to >>>> do something similar for those, like using those PetscSectionSetConstraint >>>> functions? >>>> >>> >>> Yes. There are generally two ways of handling Dirichlet conditions: >>> >>> 1) Replace those rows of the Jacobian with the identity and put the >>> boundary value in the rhs. I find >>> this cumbersome for nonlinear solves. >>> >>> 2) Remove these unknowns from the system using >>> PetscSectionSetConstraintDof() and ConstratinIndices(). >>> >>> The DMPlexAddBoundary() functions are automating this processing by >>> marking boundaries using DMLabels, >>> and then constraining dofs on those boundaries. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Justin >>>> >>>> On Thu, Oct 30, 2014 at 3:35 PM, Matthew Knepley >>>> wrote: >>>> >>>>> On Thu, Oct 30, 2014 at 3:29 PM, Justin Chang >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> So I am writing an FEM code where it reads input data (e.g., >>>>>> auxiliary coefficients, source terms, etc) from a text file. I have >>>>>> preprocessed the data text file so that each vertex point has its >>>>>> corresponding data. For instance, if my source term for a diffusion problem >>>>>> has a sin or cos function of the coordinates, then this data is already >>>>>> tabulated and simply needs to be fed into my PETSc program. The data text >>>>>> file containing both the auxiliary data and the coordinate/connectivities >>>>>> will also be used to provide the input arguments for the >>>>>> DMPlexCreateFromDAG() function. >>>>>> >>>>>> Normally, I would hard-code the sin/cos functions into the source >>>>>> code itself, but i want my program to be "universal" in that it can take as >>>>>> input any user-defined function for not only these auxiliaries but for >>>>>> other things like boundary conditions. I see that PETSc has a lot of >>>>>> examples on how to read data into vectors and matrices, but I guess my >>>>>> question is is there a way to project data from a text file into a vector >>>>>> that has already been created with a defined DM structure? >>>>>> >>>>> >>>>> If you have the functions available, I think it is far more universal >>>>> to use the function itself, since then you can be independent >>>>> of mesh and discretization when specifying input and BC. >>>>> >>>>> However, if you want to read it in, and you guarantee that it matches >>>>> the mesh and discretization, I think the easiest thing to do is >>>>> demand that it come in the same order as the vertices and use >>>>> >>>>> VecGetArray(V, &a); >>>>> for (v = vStart, i = 0; v < vEnd; ++v) { >>>>> PetscSectionGetDof(s, v, &dof); >>>>> PetscSectionGetOffset(s, v, &off); >>>>> for (d = 0; d < dof; ++d) a[off+d] = text_data[i++]; >>>>> } >>>>> VecRestoreArray(V, &a); >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks, >>>>>> Justin >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Sun Nov 2 00:24:47 2014 From: jychang48 at gmail.com (Justin Chang) Date: Sun, 2 Nov 2014 00:24:47 -0500 Subject: [petsc-users] Reading data from a file into a DM created vector In-Reply-To: References: <491840EA-7604-4E78-B152-4A9C96F3A691@gmail.com> Message-ID: Nevermind, I just looked at some of the DM test examples and it answered my question. Thanks On Sat, Nov 1, 2014 at 7:22 PM, Justin Chang wrote: > Matt, Okay that makes a lot of sense now, thank you so much. Sorry I have > one more question then I will leave you all alone :) > > From what I understand about creating DMPlex, it seems that if there are > multiple processes, only the root rank populates the DM structure. I see > that DMPlexCreateBoxMesh as well as the GMSH/Exodus/CGNS readers have rank > 0 read the mesh files and populate the sieve/mesh points within the DMPlex > (assuming that the DM was created with PETSC_COMM_WORLD). > > However, when I look at the DMPlexCreateFromDAG and > DMPlexCreateFromCellList functions, I don't see anywhere where only rank 0 > handles the creation of the DM structure. I do notice that > DMPlexCreateFromDAG already requires the DM structure to be created. So I > guess my question is, is it possible to simply have rank 0 invoke the > DMPlexCreateFromDAG function? Because that way only the root process reads > the mesh data file and creates/distributes the DMPlex among the remaining > processors. > > Thanks, > Justin > > On Sat, Nov 1, 2014 at 1:30 PM, Matthew Knepley wrote: > >> On Sat, Nov 1, 2014 at 12:20 PM, Justin Chang >> wrote: >> >>> Thanks for the response. On a related note, >>> >>> In the source code of SNES ex12, where is the PetscSection actually >>> being created? I can't seem to find anywhere in the code or its routines >>> where the creation has been explicitly called. Because when the >>> discretization/problem is define, I imagine a PetscSection for the fields >>> and constraints has to be invoked somewhere. It seems to me as soon as >>> DMPlexAddBoundary() is called it creates the PetscSectionConstraints but I >>> can't seem to really confirm this within the source of that function. The >>> reason I want to know this is because if I really were to declare my own >>> constraints, I want to make sure that the PetscSection for the field >>> variables has already been set up. >>> >> >> Much of the mechanism has been moved into the library. The reason I did >> this was to make things like >> nonlinear multigrid automatic, rather than making the user define a >> different section on each level. Let >> me explain what is happening. When we call DMGetDefaultSection(), if the >> section is not present it >> will call the createdefaultsection() member function: >> >> >> https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/interface/dm.c?at=master#cl-2977 >> >> which for Plex is here >> >> >> https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plex.c?at=master#cl-5601 >> >> It uses the discretization information from PetscDS and the BC >> information from DMPlexAddBoundary() to build the inputs for >> DMPlexCreateSection() >> >> >> https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plex.c?at=master#cl-5739 >> >> I realize that call is still limited. For example, it constrains all dofs >> on a point, but sometimes you only want to constrain some. In >> the most general case, you would just build the Section manually, using >> the same tools I use in DMPlexCreateSection(): >> >> >> https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plex.c?at=master#cl-3204 >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Justin >>> >>> >>> Sent from my iPhone >>> >>> On Nov 1, 2014, at 11:40 AM, Matthew Knepley wrote: >>> >>> On Fri, Oct 31, 2014 at 4:15 PM, Justin Chang >>> wrote: >>> >>>> Matt, >>>> >>>> Thanks for the response. One more question: >>>> >>>> If I go with approach 2, will manually setting the constraints/indices >>>> and its values be compatible with the DMPlexSNESComputeResidual/JacobianFEM >>>> routines? When I look at the source code of those routines it seems the >>>> constrained values are added to the local solution vectors via >>>> DMPlexInsertBoundaryValuesFEM. If I choose not to use DMPlexAddBoundary and >>>> wish to manually declare my boundary values, i assume I should call >>>> DMPlexProjectField with mode INSERT_BC_VALUES? >>>> >>> >>> Yes, exactly. I use DMPlexProjectFunctionLabelLocal(), >>> >>> >>> https://bitbucket.org/petsc/petsc/src/ea1a9a653238fa98fb68e87b49145608ab6e5301/src/dm/impls/plex/plexfem.c?at=master#cl-576 >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Justin >>>> >>>> Sent from my iPhone >>>> >>>> On Oct 30, 2014, at 4:24 PM, Matthew Knepley wrote: >>>> >>>> On Thu, Oct 30, 2014 at 4:16 PM, Justin Chang >>>> wrote: >>>> >>>>> Matt, thanks for the quick response. >>>>> >>>>> What about for (dirichlet) boundary conditions? Would it be possible >>>>> to do something similar for those, like using those >>>>> PetscSectionSetConstraint functions? >>>>> >>>> >>>> Yes. There are generally two ways of handling Dirichlet conditions: >>>> >>>> 1) Replace those rows of the Jacobian with the identity and put the >>>> boundary value in the rhs. I find >>>> this cumbersome for nonlinear solves. >>>> >>>> 2) Remove these unknowns from the system using >>>> PetscSectionSetConstraintDof() and ConstratinIndices(). >>>> >>>> The DMPlexAddBoundary() functions are automating this processing by >>>> marking boundaries using DMLabels, >>>> and then constraining dofs on those boundaries. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> Justin >>>>> >>>>> On Thu, Oct 30, 2014 at 3:35 PM, Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Thu, Oct 30, 2014 at 3:29 PM, Justin Chang >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> So I am writing an FEM code where it reads input data (e.g., >>>>>>> auxiliary coefficients, source terms, etc) from a text file. I have >>>>>>> preprocessed the data text file so that each vertex point has its >>>>>>> corresponding data. For instance, if my source term for a diffusion problem >>>>>>> has a sin or cos function of the coordinates, then this data is already >>>>>>> tabulated and simply needs to be fed into my PETSc program. The data text >>>>>>> file containing both the auxiliary data and the coordinate/connectivities >>>>>>> will also be used to provide the input arguments for the >>>>>>> DMPlexCreateFromDAG() function. >>>>>>> >>>>>>> Normally, I would hard-code the sin/cos functions into the source >>>>>>> code itself, but i want my program to be "universal" in that it can take as >>>>>>> input any user-defined function for not only these auxiliaries but for >>>>>>> other things like boundary conditions. I see that PETSc has a lot of >>>>>>> examples on how to read data into vectors and matrices, but I guess my >>>>>>> question is is there a way to project data from a text file into a vector >>>>>>> that has already been created with a defined DM structure? >>>>>>> >>>>>> >>>>>> If you have the functions available, I think it is far more universal >>>>>> to use the function itself, since then you can be independent >>>>>> of mesh and discretization when specifying input and BC. >>>>>> >>>>>> However, if you want to read it in, and you guarantee that it matches >>>>>> the mesh and discretization, I think the easiest thing to do is >>>>>> demand that it come in the same order as the vertices and use >>>>>> >>>>>> VecGetArray(V, &a); >>>>>> for (v = vStart, i = 0; v < vEnd; ++v) { >>>>>> PetscSectionGetDof(s, v, &dof); >>>>>> PetscSectionGetOffset(s, v, &off); >>>>>> for (d = 0; d < dof; ++d) a[off+d] = text_data[i++]; >>>>>> } >>>>>> VecRestoreArray(V, &a); >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> Justin >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leoni.massimiliano1 at gmail.com Sun Nov 2 07:19:10 2014 From: leoni.massimiliano1 at gmail.com (Massimiliano Leoni) Date: Sun, 02 Nov 2014 14:19:10 +0100 Subject: [petsc-users] Split vector with VecGetSubVector Message-ID: <5381830.AIzMAuthyv@debianxps> Hi everyone, this is my first email so I'll use this sentence to say hello from Milan, Italy :) I am very new to PETSc, I always "used" it through another library, FEniCS, but now I need to use its interface directly. Straight to my problem, I have this working code: // PETScVector is a FEniCS container for a PETSc Vec, which can be retrieved // with vec() method; similarly P->mat(). void solve(PETScVector& x, const PETScVector& b) { Mat Pr(P->mat()); KSP ksp; KSPCreate(PETSC_COMM_WORLD,&ksp); KSPSetOperators(ksp,Pr,Pr,DIFFERENT_NONZERO_PATTERN); KSPSolve(ksp,b.vec(),x.vec()); KSPDestroy(&ksp); } which simply solves with a linear system. Now I want to do the same, but separating the system [the matrix is block diagonal] into two parts [velocity and pressure, that's Stokes!]. My try, basing on my basic understanding of the documentation void solve(PETScVector& x, const PETScVector& b) { // I understand I need these Index Sets to split a vector IS isu; int Nu = Mv->size(0); // number of components of velocity ISCreateStride(PETSC_COMM_SELF,Nu,0,1,&isu); Vec buv; // to contain velocity rhs VecCreate(PETSC_COMM_WORLD,&buv); // I am not sure when to use VecSetSizes(buv,PETSC_DECIDE,Nu); // VecCreate and VecSetSizes VecGetSubVector(b.vec(),isu,&buv); Mat Mvp(Mv->mat()); // again, retrieve a Mat Vec xuv; // to contain velocity solution VecCreate(PETSC_COMM_WORLD,&xuv); VecSetSizes(xuv,PETSC_DECIDE,Nu); KSP ksp; KSPCreate(PETSC_COMM_WORLD,&ksp); KSPSetOperators(ksp,Mvp,Mvp,DIFFERENT_NONZERO_PATTERN); KSPSolve(ksp,buv,xuv); [** segfault! **] KSPDestroy(&ksp); [pressure part is analogous] Vec finalVec; // to re-assemble whole vector VecCreate(PETSC_COMM_WORLD,&finalVec); VecSetSizes(finalVec,PETSC_DECIDE,Nu+Np); IS iss[2] = {isu,isp}; Vec vecs[2] = {xuv,xpv}; VecCreateNest(PETSC_COMM_SELF, 2, iss, vecs, &finalVec); VecCopy(finalVec,x.vec()); } The problem is that it segfaults at KSPSolve. Since the structure is not too far from my working code above, I would guess I am missing something when extracting the subvectors, maybe some initializing procedure? I read the tutorial code http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex70.c.html in which this is used, but it's a little too complicated for me now. I am using petsc 3.4.2 [debian testing]. Thanks for help! Massimiliano From knepley at gmail.com Sun Nov 2 13:52:36 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 2 Nov 2014 13:52:36 -0600 Subject: [petsc-users] Split vector with VecGetSubVector In-Reply-To: <5381830.AIzMAuthyv@debianxps> References: <5381830.AIzMAuthyv@debianxps> Message-ID: On Sun, Nov 2, 2014 at 7:19 AM, Massimiliano Leoni < leoni.massimiliano1 at gmail.com> wrote: > Hi everyone, > this is my first email so I'll use this sentence to say hello from Milan, > Italy :) > > I am very new to PETSc, I always "used" it through another library, FEniCS, > but now I need to use its interface directly. > > Straight to my problem, I have this working code: > > // PETScVector is a FEniCS container for a PETSc Vec, which can be > retrieved > // with vec() method; similarly P->mat(). > void solve(PETScVector& x, const PETScVector& b) > { > Mat Pr(P->mat()); > > KSP ksp; > KSPCreate(PETSC_COMM_WORLD,&ksp); > KSPSetOperators(ksp,Pr,Pr,DIFFERENT_NONZERO_PATTERN); > KSPSolve(ksp,b.vec(),x.vec()); > > KSPDestroy(&ksp); > } > > which simply solves with a linear system. > > Now I want to do the same, but separating the system [the matrix is block > diagonal] into two parts [velocity and pressure, that's Stokes!]. > > My try, basing on my basic understanding of the documentation > > void solve(PETScVector& x, const PETScVector& b) > { > // I understand I need these Index Sets to split a vector > IS isu; > int Nu = Mv->size(0); // number of components of velocity > ISCreateStride(PETSC_COMM_SELF,Nu,0,1,&isu); > > Vec buv; // to contain velocity rhs > VecCreate(PETSC_COMM_WORLD,&buv); // I am not sure when to use > VecSetSizes(buv,PETSC_DECIDE,Nu); // VecCreate and > VecSetSizes > VecGetSubVector(b.vec(),isu,&buv); > > Mat Mvp(Mv->mat()); // again, retrieve a Mat > Vec xuv; // to contain velocity solution > VecCreate(PETSC_COMM_WORLD,&xuv); > VecSetSizes(xuv,PETSC_DECIDE,Nu); > > KSP ksp; > KSPCreate(PETSC_COMM_WORLD,&ksp); > KSPSetOperators(ksp,Mvp,Mvp,DIFFERENT_NONZERO_PATTERN); > KSPSolve(ksp,buv,xuv); [** segfault! **] > > KSPDestroy(&ksp); > > [pressure part is analogous] > > > Vec finalVec; // to re-assemble whole vector > VecCreate(PETSC_COMM_WORLD,&finalVec); > VecSetSizes(finalVec,PETSC_DECIDE,Nu+Np); > > IS iss[2] = {isu,isp}; > Vec vecs[2] = {xuv,xpv}; > VecCreateNest(PETSC_COMM_SELF, 2, iss, vecs, &finalVec); > > VecCopy(finalVec,x.vec()); > } > > > The problem is that it segfaults at KSPSolve. > Since the structure is not too far from my working code above, I would > guess I > am missing something when extracting the subvectors, maybe some > initializing > procedure? > Run in the debugger with -start_in_debugger, and when it stops type 'where' and send all output. Thanks, Matt > I read the tutorial code > http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex70.c.html > in which this is used, but > it's a little too complicated for me now. > > I am using petsc 3.4.2 [debian testing]. > > Thanks for help! > Massimiliano > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Nov 2 15:30:26 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 2 Nov 2014 15:30:26 -0600 Subject: [petsc-users] Split vector with VecGetSubVector In-Reply-To: <5381830.AIzMAuthyv@debianxps> References: <5381830.AIzMAuthyv@debianxps> Message-ID: You may also want to consider using the PETSc PCFIELDSPLIT preconditioner; this is designed for solving problems like Stokes and many others. With PCFIELDSPLIT you build one matrix that defines the entire problem over the velocities and pressures together and then it internally builds up solvers for the blocks so you do not need to monkey with all those details yourself. Barry > On Nov 2, 2014, at 7:19 AM, Massimiliano Leoni wrote: > > Hi everyone, > this is my first email so I'll use this sentence to say hello from Milan, > Italy :) > > I am very new to PETSc, I always "used" it through another library, FEniCS, > but now I need to use its interface directly. > > Straight to my problem, I have this working code: > > // PETScVector is a FEniCS container for a PETSc Vec, which can be retrieved > // with vec() method; similarly P->mat(). > void solve(PETScVector& x, const PETScVector& b) > { > Mat Pr(P->mat()); > > KSP ksp; > KSPCreate(PETSC_COMM_WORLD,&ksp); > KSPSetOperators(ksp,Pr,Pr,DIFFERENT_NONZERO_PATTERN); > KSPSolve(ksp,b.vec(),x.vec()); > > KSPDestroy(&ksp); > } > > which simply solves with a linear system. > > Now I want to do the same, but separating the system [the matrix is block > diagonal] into two parts [velocity and pressure, that's Stokes!]. > > My try, basing on my basic understanding of the documentation > > void solve(PETScVector& x, const PETScVector& b) > { > // I understand I need these Index Sets to split a vector > IS isu; > int Nu = Mv->size(0); // number of components of velocity > ISCreateStride(PETSC_COMM_SELF,Nu,0,1,&isu); > > Vec buv; // to contain velocity rhs > VecCreate(PETSC_COMM_WORLD,&buv); // I am not sure when to use > VecSetSizes(buv,PETSC_DECIDE,Nu); // VecCreate and VecSetSizes > VecGetSubVector(b.vec(),isu,&buv); > > Mat Mvp(Mv->mat()); // again, retrieve a Mat > Vec xuv; // to contain velocity solution > VecCreate(PETSC_COMM_WORLD,&xuv); > VecSetSizes(xuv,PETSC_DECIDE,Nu); > > KSP ksp; > KSPCreate(PETSC_COMM_WORLD,&ksp); > KSPSetOperators(ksp,Mvp,Mvp,DIFFERENT_NONZERO_PATTERN); > KSPSolve(ksp,buv,xuv); [** segfault! **] > > KSPDestroy(&ksp); > > [pressure part is analogous] > > > Vec finalVec; // to re-assemble whole vector > VecCreate(PETSC_COMM_WORLD,&finalVec); > VecSetSizes(finalVec,PETSC_DECIDE,Nu+Np); > > IS iss[2] = {isu,isp}; > Vec vecs[2] = {xuv,xpv}; > VecCreateNest(PETSC_COMM_SELF, 2, iss, vecs, &finalVec); > > VecCopy(finalVec,x.vec()); > } > > > The problem is that it segfaults at KSPSolve. > Since the structure is not too far from my working code above, I would guess I > am missing something when extracting the subvectors, maybe some initializing > procedure? > > I read the tutorial code http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex70.c.html in which this is used, but > it's a little too complicated for me now. > > I am using petsc 3.4.2 [debian testing]. > > Thanks for help! > Massimiliano From leoni.massimiliano1 at gmail.com Mon Nov 3 03:25:21 2014 From: leoni.massimiliano1 at gmail.com (Massimiliano Leoni) Date: Mon, 03 Nov 2014 10:25:21 +0100 Subject: [petsc-users] Split vector with VecGetSubVector In-Reply-To: References: <5381830.AIzMAuthyv@debianxps> Message-ID: <1575856.xZEEoKIuAc@debianxps> @Matt: I luckily managed to solve the problem myself, I started reading through the PETSc manual and re-wrote cleaner code. It runs, but yields a wrong result. @Barry: this definitely looks interesting! I am on a university assignment and I'll often have to solve this kind of problems, so this PCFIELDSPLIT looks just like what is needed. Also, people at FEniCS dev are very interested in it, so I want to go deeper. I read the documentation and the tutorial and I wrote this version, starting from my original working code and the tutorial: void StokesIdBlTrPrecSegregated::solve(PETScVector& x, const PETScVector& b) { // PCFIELDSPLIT version // "the whole world is talking about it, // and I've got to know why" [inits, bcs, and other stuff I'll move to the constructor asap] // KSP *subksp; // PetscInt n = 1; IS isu; int Nu = Mv->size(0); ISCreateStride(PETSC_COMM_WORLD,Nu,0,1,&isu); IS isp; int Np = Mp->size(0); ISCreateStride(PETSC_COMM_WORLD,Np,0,1,&isp); PC pc; KSP ksp; KSPCreate(PETSC_COMM_WORLD,&ksp); KSPGetPC(ksp, &pc); PCFieldSplitSetIS(pc, "0", isu); PCFieldSplitSetIS(pc, "1", isp); /* if (s->userPC) { //not yet clear what's this for PCFieldSplitSchurPrecondition(pc, PC_FIELDSPLIT_SCHUR_PRE_USER, s->myS); } if (s->userKSP) { PCSetUp(pc); PCFieldSplitGetSubKSP(pc, &n, &subksp); KSPSetOperators(subksp[1], s->myS, s->myS, SAME_PRECONDITIONER); PetscFree(subksp); } */ Mat Pr(P->mat()); KSPSetOperators(ksp,Pr,Pr,DIFFERENT_NONZERO_PATTERN); KSPSetFromOptions(ksp); KSPSolve(ksp,b.vec(),x.vec()); KSPDestroy(&ksp); } Now, this runs and yields the correct solution, but how can I be sure it is actually doing splitting blocks? This looks rather incomplete to me. What's missing in this code? More importantly, where can I read more on this PCFIELDSPLIT? Thanks for the precious help, Massimiliano P.s. Shall I change the thread subject to another? Or start another thread? In data domenica 2 novembre 2014 15:30:26, hai scritto: > You may also want to consider using the PETSc PCFIELDSPLIT preconditioner; > this is designed for solving problems like Stokes and many others. With > PCFIELDSPLIT you build one matrix that defines the entire problem over the > velocities and pressures together and then it internally builds up solvers > for the blocks so you do not need to monkey with all those details > yourself. > > Barry > > > On Nov 2, 2014, at 7:19 AM, Massimiliano Leoni > > wrote: > > > > Hi everyone, > > > > this is my first email so I'll use this sentence to say hello from Milan, > > > > Italy :) > > > > I am very new to PETSc, I always "used" it through another library, > > FEniCS, > > but now I need to use its interface directly. > > > > Straight to my problem, I have this working code: > > > > // PETScVector is a FEniCS container for a PETSc Vec, which can be > > retrieved // with vec() method; similarly P->mat(). > > void solve(PETScVector& x, const PETScVector& b) > > { > > > > Mat Pr(P->mat()); > > > > KSP ksp; > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > KSPSetOperators(ksp,Pr,Pr,DIFFERENT_NONZERO_PATTERN); > > KSPSolve(ksp,b.vec(),x.vec()); > > > > KSPDestroy(&ksp); > > > > } > > > > which simply solves with a linear system. > > > > Now I want to do the same, but separating the system [the matrix is block > > diagonal] into two parts [velocity and pressure, that's Stokes!]. > > > > My try, basing on my basic understanding of the documentation > > > > void solve(PETScVector& x, const PETScVector& b) > > { > > > > // I understand I need these Index Sets to split a vector > > IS isu; > > int Nu = Mv->size(0); // number of components of velocity > > ISCreateStride(PETSC_COMM_SELF,Nu,0,1,&isu); > > > > Vec buv; // to contain velocity rhs > > VecCreate(PETSC_COMM_WORLD,&buv); // I am not sure when to use > > VecSetSizes(buv,PETSC_DECIDE,Nu); // VecCreate and VecSetSizes > > VecGetSubVector(b.vec(),isu,&buv); > > > > Mat Mvp(Mv->mat()); // again, retrieve a Mat > > Vec xuv; // to contain velocity solution > > VecCreate(PETSC_COMM_WORLD,&xuv); > > VecSetSizes(xuv,PETSC_DECIDE,Nu); > > > > KSP ksp; > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > KSPSetOperators(ksp,Mvp,Mvp,DIFFERENT_NONZERO_PATTERN); > > KSPSolve(ksp,buv,xuv); [** segfault! **] > > > > KSPDestroy(&ksp); > > > > [pressure part is analogous] > > > > > > Vec finalVec; // to re-assemble whole vector > > VecCreate(PETSC_COMM_WORLD,&finalVec); > > VecSetSizes(finalVec,PETSC_DECIDE,Nu+Np); > > > > IS iss[2] = {isu,isp}; > > Vec vecs[2] = {xuv,xpv}; > > VecCreateNest(PETSC_COMM_SELF, 2, iss, vecs, &finalVec); > > > > VecCopy(finalVec,x.vec()); > > > > } > > > > > > The problem is that it segfaults at KSPSolve. > > Since the structure is not too far from my working code above, I would > > guess I am missing something when extracting the subvectors, maybe some > > initializing procedure? > > > > I read the tutorial code > > http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex > > 70.c.html in which this is used, but it's a little too complicated for me > > now. > > > > I am using petsc 3.4.2 [debian testing]. > > > > Thanks for help! > > Massimiliano From olivier.bonnefon at avignon.inra.fr Mon Nov 3 03:46:21 2014 From: olivier.bonnefon at avignon.inra.fr (Olivier Bonnefon) Date: Mon, 03 Nov 2014 10:46:21 +0100 Subject: [petsc-users] petsc-3.5.2: ex12 with Neumann BC In-Reply-To: References: <524430F2.2020507@avignon.inra.fr> <524D3D5D.5090301@avignon.inra.fr> <525804C2.3000101@avignon.inra.fr> <5453AE09.5090806@avignon.inra.fr> Message-ID: <54574EED.3060309@avignon.inra.fr> Hello, Thank for your answer, I'll explain my trouble: My problem is that the BC Neumann leads to wrong result. With Dirichlet BC, I get: ------------------------------- > ./ex12 -run_type full -refinement_limit 0.0 -bc_type dirichlet -interpolate 0 -petscspace_order 1 -show_initial -show_solution -dm_plex_print_fem 1 ... ... Solution Vec Object:potential 1 MPI processes type: seq 0.5 This result is correct. With Neuman BC, I get: -------------------------------- > ./ex12 -run_type full -refinement_limit 0.0 -bc_type neumann -interpolate 1 -petscspace_order 1 -show_initial -dm_plex_print_fem 1 -show_solution -bd_petscspace_order 1 -snes_linesearch_monitor -snes_monitor -ksp_monitor_true_residual -snes_converged_reason -ksp_converged_reason .... .... Solution Vec Object:potential 1 MPI processes type: seq -0.75 -0.583333 0.0833333 -0.583333 -0.333333 0.416667 0.0833333 0.416667 1.25 That is not the values of the solution x*x+y*y. I tried many ksp options. Moreover, the neumann BC with "-run_type full" is not cover in the list https://bitbucket.org/petsc/petsc/src/fced3c3f9e703542693913793d15321603e40fe6/config/builder.py?at=master#cl-257 Do you know what is wrong ? Thanks, Olivier Bonnefon On 10/31/2014 06:50 PM, Matthew Knepley wrote: > On Fri, Oct 31, 2014 at 10:43 AM, Olivier Bonnefon > > wrote: > > Hello, > > I'm working on the snes/examples/tutorial/ex12 version 3.5.2. > > I didn't succed to run the simplest case with Neumann BC: > > ./ex12 -run_type full -refinement_limit 0.0 -bc_type neumann > -interpolate 1 -petscspace_order 1 -show_initial > -dm_plex_print_fem 1 -show_solution -bd_petscspace_order 1 > -snes_linesearch_monitor -snes_monitor -ksp_monitor_true_residual > -snes_converged_reason -ksp_converged_reason > > This leads to dofs negatives values. > > > I do not understand what you mean. Please always mail the full error > message. > > Do you know the options to get a correct result with Neumann BC ? > > > There are some tests here: > > https://bitbucket.org/petsc/petsc/src/fced3c3f9e703542693913793d15321603e40fe6/config/builder.py?at=master#cl-257 > > Matt > > Regards, > Olivier Bonnefon > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -- Olivier Bonnefon INRA PACA-Avignon, Unit? BioSP Tel: +33 (0)4 32 72 21 58 -------------- next part -------------- An HTML attachment was scrubbed... URL: From gbisht at lbl.gov Tue Nov 4 00:56:58 2014 From: gbisht at lbl.gov (Gautam Bisht) Date: Mon, 3 Nov 2014 22:56:58 -0800 Subject: [petsc-users] sundials results do not agree with beuler/rosw/pseudo Message-ID: Hi, I'm solving subsurface flow equation in which the governing ODE is reformulated as a system of DAE. I'm using PETSc TS+DMComposite to solve the system with a LU preconditioner. I get comparable results for BEULER, ROSW and PSEUDO ts_type. But results with SUNDIALS for even a single TS step are significantly underestimated when compared to those obtained for the other ts_types. I would appreciate if folks would suggest ideas on how can I go about figuring out what is going wrong with SUNDIALS. I'm using following PETSc options: >/opt/local/bin/mpiexec -n 1 $EXEROOT/cesm.exe \ -ts_monitor \ -ts_view \ -snes_monitor \ -pc_type lu \ -ts_type sundials -ts_sundials_monitor_steps \ -ts_dt 1.0 -ts_final_time 1.0 0 TS dt 1 time 0 1 TS dt 1 time 1 TS Object: 1 MPI processes type: sundials maximum steps=100000 maximum time=1 total number of nonlinear solver iterations=0 total number of nonlinear solve failures=0 total number of linear solver iterations=0 total number of rejected steps=0 Sundials integrater does not use SNES! Sundials integrater type BDF: backward differentiation formula Sundials abs tol 1e-06 rel tol 1e-06 Sundials linear solver tolerance factor 0.05 Sundials max dimension of Krylov subspace 5 Sundials using unmodified (classical) Gram-Schmidt for orthogonalization in GMRES Sundials suggested factor for tolerance scaling 1 Sundials cumulative number of internal steps 1 Sundials no. of calls to rhs function 2 Sundials no. of calls to linear solver setup function 1 Sundials no. of error test failures 0 Sundials no. of nonlinear solver iterations 1 Sundials no. of nonlinear convergence failure 0 Sundials no. of linear iterations 0 Sundials no. of linear convergence failures 0 PC Object: 1 MPI processes type: lu PC has not been set up so information may be incomplete LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=200, cols=200 total: nonzeros=598, allocated nonzeros=3200 total number of mallocs used during MatSetValues calls =200 not using I-node routines Sundials no. of preconditioner evaluations 1 Sundials no. of preconditioner solves 0 Sundials no. of Jacobian-vector product evaluations 0 Sundials no. of rhs calls for finite diff. Jacobian-vector evals 0 Thanks, -Gautam. -------------- next part -------------- An HTML attachment was scrubbed... URL: From filippo.leonardi at sam.math.ethz.ch Tue Nov 4 01:55:22 2014 From: filippo.leonardi at sam.math.ethz.ch (Filippo Leonardi) Date: Tue, 4 Nov 2014 08:55:22 +0100 Subject: [petsc-users] Multiple solves with PCMG fail In-Reply-To: References: <9621705.dtimTkejkQ@besikovitch-ii> <87y4s0xp36.fsf@jedbrown.org> Message-ID: <2052102.bo3NGSHfll@besikovitch-iii> Sorry for the late update. I tested the code with the branch: ksp-zero-eig But the problem seems to still be there, also tried: -ksp_chebyshev_estimate_eigenvalues_random but does not change. Just tell me what test you need and I'll try to do them. Best, Filippo On Wednesday 29 October 2014 10.30:14 Mark Adams wrote: > > Yeah, that should work; an alternative is to defer the estimate until > > the next solve. > > Sounds better. As I said one of my users has a zero RHS on the first solve. -------------- next part -------------- A non-text attachment was scrubbed... Name: Filippo Leonardi.vcf Type: text/vcard Size: 322 bytes Desc: not available URL: From jed at jedbrown.org Tue Nov 4 02:02:00 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 04 Nov 2014 01:02:00 -0700 Subject: [petsc-users] Multiple solves with PCMG fail In-Reply-To: <2052102.bo3NGSHfll@besikovitch-iii> References: <9621705.dtimTkejkQ@besikovitch-ii> <87y4s0xp36.fsf@jedbrown.org> <2052102.bo3NGSHfll@besikovitch-iii> Message-ID: <87k33be6mf.fsf@jedbrown.org> Filippo Leonardi writes: > Sorry for the late update. > > I tested the code with the branch: > ksp-zero-eig > > But the problem seems to still be there, also tried: > -ksp_chebyshev_estimate_eigenvalues_random > but does not change. These are probably on levels, like -mg_levels_ksp_chebyshev_estimate_eigenvalues_random, but perhaps you set other prefixes or it's inside fieldsplit, etc. Just use the correct prefix for the solve that is failing. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From mailinglists at xgm.de Tue Nov 4 03:28:41 2014 From: mailinglists at xgm.de (Florian Lindner) Date: Tue, 04 Nov 2014 10:28:41 +0100 Subject: [petsc-users] Speed up KSPSolve of dense matrix Message-ID: <5550582.ftmzpoOxMT@asaru> Hello, I have a fulll matrix of size e.g. 603x603 of which I'm very disappointed with the runtime, compared to a naive LU / forward / backward solution. My petsc solution takes about 14s, while the old one takes just 0.5s. (when you're looking at sparse matrices the figures are almost inversed). I try to investigate what's the problem. Most of the time is spent in the KSPSolve. Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage MatMult 10334 1.0 6.3801e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 45 49 0 0 0 45 49 0 0 0 1177 MatSolve 10334 1.0 6.9187e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 49 49 0 0 0 49 49 0 0 0 1085 MatCholFctrNum 1 1.0 1.8968e-01 1.0 6.03e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 KSPGMRESOrthog 10000 1.0 2.1417e-01 1.0 3.73e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 2 0 0 0 2 2 0 0 0 1744 KSPSolve 1 1.0 1.3763e+01 1.0 1.54e+10 1.0 0.0e+00 0.0e+00 0.0e+00 97100 0 0 0 97100 0 0 0 1120 Prealloc Matrix C 1 1.0 5.7458e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 Filling Matrix C 1 1.0 6.7984e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecMDot 10000 1.0 8.8615e-02 1.0 1.87e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 2106 VecMAXPY 10334 1.0 1.2585e-01 1.0 1.99e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1580 Prealloc Matrix A 1 1.0 1.1071e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 Filling Matrix A 1 1.0 1.2574e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 PCSetUp 1 1.0 1.9073e-01 1.0 6.03e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 PCApply 10334 1.0 6.9233e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 49 49 0 0 0 49 49 0 0 0 1085 ------------------------------------------------------------------------------------------------------------------------ Removed all events with global %T == 0, but my own ones. Matrix type is sequential MATSBAIJ on PETSC_COMM_SELF. KSP is initalized once: KSPSetOperators(_solver, _matrixC.matrix, _matrixC.matrix); KSPSetTolerances(_solver, _solverRtol, PETSC_DEFAULT, PETSC_DEFAULT, PETSC_DEFAULT); KSPSetFromOptions(_solver); KSPSolve: ierr = KSPSolve(_solver, in.vector, p.vector); CHKERRV(ierr) KSPSolve may be executed multiple times with the same SetOperators acording to dimensionality of the data. Here it's just one. -ksp_view says: KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-09, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: icc 0 levels of fill tolerance for zero pivot 2.22045e-14 using Manteuffel shift [POSITIVE_DEFINITE] matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Mat Object: 1 MPI processes type: seqsbaij rows=603, cols=603 package used to perform factorization: petsc total: nonzeros=182103, allocated nonzeros=182103 total number of mallocs used during MatSetValues calls =0 block size is 1 linear system matrix = precond matrix: Mat Object: C 1 MPI processes type: seqsbaij rows=603, cols=603 total: nonzeros=182103, allocated nonzeros=182103 total number of mallocs used during MatSetValues calls =0 block size is 1 I've tried to use a direct solver like suggested on pp 72, but: ./petBench 600 1 -ksp_type preonly -pc_type lu Mesh of size: 600 Do mappings: 1 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Factor type not supported [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown [0]PETSC ERROR: ./petBench on a arch-linux2-c-opt named helium by lindnefn Tue Nov 4 10:19:28 2014 [0]PETSC ERROR: Configure options --with-debugging=0 [0]PETSC ERROR: #1 MatGetFactor_seqsbaij_petsc() line 1833 in /data2/scratch/lindner/petsc/src/mat/impls/sbaij/seq/sbaij.c [0]PETSC ERROR: #2 MatGetFactor() line 3963 in /data2/scratch/lindner/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #3 PCSetUp_LU() line 125 in /data2/scratch/lindner/petsc/src/ksp/pc/impls/factor/lu/lu.c [0]PETSC ERROR: #4 PCSetUp() line 902 in /data2/scratch/lindner/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #5 KSPSetUp() line 305 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #6 KSPSolve() line 417 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #7 map() line 391 in /data2/scratch/lindner/precice/src/mapping/PetRadialBasisFctMapping.hpp Any other idea about how to speed up that? Thanks for any suggestions! Florian From norihiro.w at gmail.com Tue Nov 4 05:21:51 2014 From: norihiro.w at gmail.com (Norihiro Watanabe) Date: Tue, 4 Nov 2014 12:21:51 +0100 Subject: [petsc-users] VecView in Ascii VTK mode Message-ID: Hi, I've updated PETSc version from 3.4 to 3.5 and found that object names are printed in VecView() even for ASCII VTK format. Is there any changes in the VTK output since the new version? Our code is something like below. == PetscViewerCreate(PETSC_COMM_WORLD, &viewer); PetscViewerSetType(viewer, PETSCVIEWERASCII); PetscViewerFileSetName(viewer, filename); PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_VTK); VecView(x, viewer); == and VTK files created from v3.5 include additional two lines before POINT_DATA as == Vec Object:Concentration 4 MPI processes type: mpi POINT_DATA 1915 SCALARS Concentration double 1 LOOKUP_TABLE default 0.007413 0.00741375 0.00741394 .... == Best, Nori -- Norihiro Watanabe -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Tue Nov 4 09:40:35 2014 From: abhyshr at mcs.anl.gov (Abhyankar, Shrirang G.) Date: Tue, 4 Nov 2014 15:40:35 +0000 Subject: [petsc-users] sundials results do not agree with beuler/rosw/pseudo In-Reply-To: Message-ID: PETSc only supports the ODE solver (CVODE) from Sundials. Shri From: Gautam Bisht > Date: Mon, 3 Nov 2014 22:56:58 -0800 To: > Subject: [petsc-users] sundials results do not agree with beuler/rosw/pseudo Hi, I'm solving subsurface flow equation in which the governing ODE is reformulated as a system of DAE. I'm using PETSc TS+DMComposite to solve the system with a LU preconditioner. I get comparable results for BEULER, ROSW and PSEUDO ts_type. But results with SUNDIALS for even a single TS step are significantly underestimated when compared to those obtained for the other ts_types. I would appreciate if folks would suggest ideas on how can I go about figuring out what is going wrong with SUNDIALS. I'm using following PETSc options: >/opt/local/bin/mpiexec -n 1 $EXEROOT/cesm.exe \ -ts_monitor \ -ts_view \ -snes_monitor \ -pc_type lu \ -ts_type sundials -ts_sundials_monitor_steps \ -ts_dt 1.0 -ts_final_time 1.0 0 TS dt 1 time 0 1 TS dt 1 time 1 TS Object: 1 MPI processes type: sundials maximum steps=100000 maximum time=1 total number of nonlinear solver iterations=0 total number of nonlinear solve failures=0 total number of linear solver iterations=0 total number of rejected steps=0 Sundials integrater does not use SNES! Sundials integrater type BDF: backward differentiation formula Sundials abs tol 1e-06 rel tol 1e-06 Sundials linear solver tolerance factor 0.05 Sundials max dimension of Krylov subspace 5 Sundials using unmodified (classical) Gram-Schmidt for orthogonalization in GMRES Sundials suggested factor for tolerance scaling 1 Sundials cumulative number of internal steps 1 Sundials no. of calls to rhs function 2 Sundials no. of calls to linear solver setup function 1 Sundials no. of error test failures 0 Sundials no. of nonlinear solver iterations 1 Sundials no. of nonlinear convergence failure 0 Sundials no. of linear iterations 0 Sundials no. of linear convergence failures 0 PC Object: 1 MPI processes type: lu PC has not been set up so information may be incomplete LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=200, cols=200 total: nonzeros=598, allocated nonzeros=3200 total number of mallocs used during MatSetValues calls =200 not using I-node routines Sundials no. of preconditioner evaluations 1 Sundials no. of preconditioner solves 0 Sundials no. of Jacobian-vector product evaluations 0 Sundials no. of rhs calls for finite diff. Jacobian-vector evals 0 Thanks, -Gautam. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Nov 4 10:08:27 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 4 Nov 2014 10:08:27 -0600 Subject: [petsc-users] Speed up KSPSolve of dense matrix In-Reply-To: <5550582.ftmzpoOxMT@asaru> References: <5550582.ftmzpoOxMT@asaru> Message-ID: <09B0443C-D63A-4173-A969-457188238E28@mcs.anl.gov> There are a lot of questions here. > On Nov 4, 2014, at 3:28 AM, Florian Lindner wrote: > > Hello, > > I have a fulll matrix of size e.g. 603x603 of which I'm very disappointed with the runtime, compared to a naive LU / forward / backward solution. My petsc solution takes about 14s, while the old one takes just 0.5s. (when you're looking at sparse matrices the figures are almost inversed). I try to investigate what's the problem. For small problems like this direct dense LU could easily be faster than iterative schemes. Especially if you have many right hand sides with the same matrix since the factorization only needs to be done one. > > Most of the time is spent in the KSPSolve. > > Event Count Time (sec) Flops --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > MatMult 10334 1.0 6.3801e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 45 49 0 0 0 45 49 0 0 0 1177 > MatSolve 10334 1.0 6.9187e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 49 49 0 0 0 49 49 0 0 0 1085 If this is for a single solver for a 603 by 603 matrix then this is absurd. 10000 plus iterations > MatCholFctrNum 1 1.0 1.8968e-01 1.0 6.03e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > KSPGMRESOrthog 10000 1.0 2.1417e-01 1.0 3.73e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 2 0 0 0 2 2 0 0 0 1744 > KSPSolve 1 1.0 1.3763e+01 1.0 1.54e+10 1.0 0.0e+00 0.0e+00 0.0e+00 97100 0 0 0 97100 0 0 0 1120 > Prealloc Matrix C 1 1.0 5.7458e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > Filling Matrix C 1 1.0 6.7984e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecMDot 10000 1.0 8.8615e-02 1.0 1.87e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 2106 > VecMAXPY 10334 1.0 1.2585e-01 1.0 1.99e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1580 > Prealloc Matrix A 1 1.0 1.1071e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > Filling Matrix A 1 1.0 1.2574e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > PCSetUp 1 1.0 1.9073e-01 1.0 6.03e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > PCApply 10334 1.0 6.9233e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 49 49 0 0 0 49 49 0 0 0 1085 > ------------------------------------------------------------------------------------------------------------------------ > > Removed all events with global %T == 0, but my own ones. > > Matrix type is sequential MATSBAIJ on PETSC_COMM_SELF. > > KSP is initalized once: > > KSPSetOperators(_solver, _matrixC.matrix, _matrixC.matrix); > KSPSetTolerances(_solver, _solverRtol, PETSC_DEFAULT, PETSC_DEFAULT, PETSC_DEFAULT); > KSPSetFromOptions(_solver); > > KSPSolve: > > ierr = KSPSolve(_solver, in.vector, p.vector); CHKERRV(ierr) > > KSPSolve may be executed multiple times with the same SetOperators acording to dimensionality of the data. Here it's just one. > > -ksp_view says: > > KSP Object: 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-09, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: icc > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > using Manteuffel shift [POSITIVE_DEFINITE] > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqsbaij > rows=603, cols=603 > package used to perform factorization: petsc > total: nonzeros=182103, allocated nonzeros=182103 > total number of mallocs used during MatSetValues calls =0 > block size is 1 > linear system matrix = precond matrix: > Mat Object: C 1 MPI processes > type: seqsbaij > rows=603, cols=603 > total: nonzeros=182103, allocated nonzeros=182103 > total number of mallocs used during MatSetValues calls =0 > block size is 1 > > > I've tried to use a direct solver like suggested on pp 72, but: > > ./petBench 600 1 -ksp_type preonly -pc_type lu You cannot use LU with SBAIJ format. Only Cholesky. So use -pc_type cholesky > Mesh of size: 600 > Do mappings: 1 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Factor type not supported > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown > [0]PETSC ERROR: ./petBench on a arch-linux2-c-opt named helium by lindnefn Tue Nov 4 10:19:28 2014 > [0]PETSC ERROR: Configure options --with-debugging=0 > [0]PETSC ERROR: #1 MatGetFactor_seqsbaij_petsc() line 1833 in /data2/scratch/lindner/petsc/src/mat/impls/sbaij/seq/sbaij.c > [0]PETSC ERROR: #2 MatGetFactor() line 3963 in /data2/scratch/lindner/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #3 PCSetUp_LU() line 125 in /data2/scratch/lindner/petsc/src/ksp/pc/impls/factor/lu/lu.c > [0]PETSC ERROR: #4 PCSetUp() line 902 in /data2/scratch/lindner/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSPSetUp() line 305 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #6 KSPSolve() line 417 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #7 map() line 391 in /data2/scratch/lindner/precice/src/mapping/PetRadialBasisFctMapping.hpp > > > Any other idea about how to speed up that? > > Thanks for any suggestions! > Florian > From bsmith at mcs.anl.gov Tue Nov 4 10:19:56 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 4 Nov 2014 10:19:56 -0600 Subject: [petsc-users] sundials results do not agree with beuler/rosw/pseudo In-Reply-To: References: Message-ID: <629E9E22-0D16-4BDF-9BA2-1B357A5BB1B0@mcs.anl.gov> Note: PSEUDO is a continuation method for solving a nonlinear system, it doesn?t make sense to use it to replace other methods. That time step of 1 seems very large for SUNDIALs to be using Barry > On Nov 4, 2014, at 9:40 AM, Abhyankar, Shrirang G. wrote: > > PETSc only supports the ODE solver (CVODE) from Sundials. > > Shri > > From: Gautam Bisht > Date: Mon, 3 Nov 2014 22:56:58 -0800 > To: > Subject: [petsc-users] sundials results do not agree with beuler/rosw/pseudo > > Hi, > > I'm solving subsurface flow equation in which the governing ODE is reformulated as a system of DAE. I'm using PETSc TS+DMComposite to solve the system with a LU preconditioner. I get comparable results for BEULER, ROSW and PSEUDO ts_type. But results with SUNDIALS for even a single TS step are significantly underestimated when compared to those obtained for the other ts_types. I would appreciate if folks would suggest ideas on how can I go about figuring out what is going wrong with SUNDIALS. > > I'm using following PETSc options: > > >/opt/local/bin/mpiexec -n 1 $EXEROOT/cesm.exe \ > -ts_monitor \ > -ts_view \ > -snes_monitor \ > -pc_type lu \ > -ts_type sundials -ts_sundials_monitor_steps \ > -ts_dt 1.0 -ts_final_time 1.0 > > 0 TS dt 1 time 0 > 1 TS dt 1 time 1 > TS Object: 1 MPI processes > type: sundials > maximum steps=100000 > maximum time=1 > total number of nonlinear solver iterations=0 > total number of nonlinear solve failures=0 > total number of linear solver iterations=0 > total number of rejected steps=0 > Sundials integrater does not use SNES! > Sundials integrater type BDF: backward differentiation formula > Sundials abs tol 1e-06 rel tol 1e-06 > Sundials linear solver tolerance factor 0.05 > Sundials max dimension of Krylov subspace 5 > Sundials using unmodified (classical) Gram-Schmidt for orthogonalization in GMRES > Sundials suggested factor for tolerance scaling 1 > Sundials cumulative number of internal steps 1 > Sundials no. of calls to rhs function 2 > Sundials no. of calls to linear solver setup function 1 > Sundials no. of error test failures 0 > Sundials no. of nonlinear solver iterations 1 > Sundials no. of nonlinear convergence failure 0 > Sundials no. of linear iterations 0 > Sundials no. of linear convergence failures 0 > PC Object: 1 MPI processes > type: lu > PC has not been set up so information may be incomplete > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=200, cols=200 > total: nonzeros=598, allocated nonzeros=3200 > total number of mallocs used during MatSetValues calls =200 > not using I-node routines > Sundials no. of preconditioner evaluations 1 > Sundials no. of preconditioner solves 0 > Sundials no. of Jacobian-vector product evaluations 0 > Sundials no. of rhs calls for finite diff. Jacobian-vector evals 0 > > Thanks, > -Gautam. > From bsmith at mcs.anl.gov Tue Nov 4 10:22:37 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 4 Nov 2014 10:22:37 -0600 Subject: [petsc-users] VecView in Ascii VTK mode In-Reply-To: References: Message-ID: This is my fault. Is there a comment marker for ASCII VTK format? Let me know what it is and I?ll fix the code to make those lines comments and hence ignored by VTK Barry > On Nov 4, 2014, at 5:21 AM, Norihiro Watanabe wrote: > > Hi, > > I've updated PETSc version from 3.4 to 3.5 and found that object names are printed in VecView() even for ASCII VTK format. Is there any changes in the VTK output since the new version? > > Our code is something like below. > > == > PetscViewerCreate(PETSC_COMM_WORLD, &viewer); > PetscViewerSetType(viewer, PETSCVIEWERASCII); > PetscViewerFileSetName(viewer, filename); > PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_VTK); > VecView(x, viewer); > == > > and VTK files created from v3.5 include additional two lines before POINT_DATA as > > == > Vec Object:Concentration 4 MPI processes > type: mpi > POINT_DATA 1915 > SCALARS Concentration double 1 > LOOKUP_TABLE default > 0.007413 > 0.00741375 > 0.00741394 > .... > == > > > Best, > Nori > > -- > Norihiro Watanabe From norihiro.w at gmail.com Tue Nov 4 10:37:38 2014 From: norihiro.w at gmail.com (Norihiro Watanabe) Date: Tue, 4 Nov 2014 17:37:38 +0100 Subject: [petsc-users] VecView in Ascii VTK mode In-Reply-To: References: Message-ID: as far as I read http://www.vtk.org/VTK/img/file-formats.pdf, there is no comment marker in ASCII VTK format. Best, nori On Tue, Nov 4, 2014 at 5:22 PM, Barry Smith wrote: > > This is my fault. Is there a comment marker for ASCII VTK format? Let me > know what it is and I?ll fix the code to make those lines comments and > hence ignored by VTK > > Barry > > > On Nov 4, 2014, at 5:21 AM, Norihiro Watanabe > wrote: > > > > Hi, > > > > I've updated PETSc version from 3.4 to 3.5 and found that object names > are printed in VecView() even for ASCII VTK format. Is there any changes in > the VTK output since the new version? > > > > Our code is something like below. > > > > == > > PetscViewerCreate(PETSC_COMM_WORLD, &viewer); > > PetscViewerSetType(viewer, PETSCVIEWERASCII); > > PetscViewerFileSetName(viewer, filename); > > PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_VTK); > > VecView(x, viewer); > > == > > > > and VTK files created from v3.5 include additional two lines before > POINT_DATA as > > > > == > > Vec Object:Concentration 4 MPI processes > > type: mpi > > POINT_DATA 1915 > > SCALARS Concentration double 1 > > LOOKUP_TABLE default > > 0.007413 > > 0.00741375 > > 0.00741394 > > .... > > == > > > > > > Best, > > Nori > > > > -- > > Norihiro Watanabe > > -- Norihiro Watanabe -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Nov 4 10:40:17 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 4 Nov 2014 10:40:17 -0600 Subject: [petsc-users] VecView in Ascii VTK mode In-Reply-To: References: Message-ID: <2F9CD9B4-3D8D-47EF-A6F9-19137DBBA414@mcs.anl.gov> Ok, then I?ll have it not print that information at all. Barry > On Nov 4, 2014, at 10:37 AM, Norihiro Watanabe wrote: > > as far as I read http://www.vtk.org/VTK/img/file-formats.pdf, there is no comment marker in ASCII VTK format. > > Best, > nori > > On Tue, Nov 4, 2014 at 5:22 PM, Barry Smith wrote: > > This is my fault. Is there a comment marker for ASCII VTK format? Let me know what it is and I?ll fix the code to make those lines comments and hence ignored by VTK > > Barry > > > On Nov 4, 2014, at 5:21 AM, Norihiro Watanabe wrote: > > > > Hi, > > > > I've updated PETSc version from 3.4 to 3.5 and found that object names are printed in VecView() even for ASCII VTK format. Is there any changes in the VTK output since the new version? > > > > Our code is something like below. > > > > == > > PetscViewerCreate(PETSC_COMM_WORLD, &viewer); > > PetscViewerSetType(viewer, PETSCVIEWERASCII); > > PetscViewerFileSetName(viewer, filename); > > PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_VTK); > > VecView(x, viewer); > > == > > > > and VTK files created from v3.5 include additional two lines before POINT_DATA as > > > > == > > Vec Object:Concentration 4 MPI processes > > type: mpi > > POINT_DATA 1915 > > SCALARS Concentration double 1 > > LOOKUP_TABLE default > > 0.007413 > > 0.00741375 > > 0.00741394 > > .... > > == > > > > > > Best, > > Nori > > > > -- > > Norihiro Watanabe > > > > > -- > Norihiro Watanabe From norihiro.w at gmail.com Tue Nov 4 10:41:25 2014 From: norihiro.w at gmail.com (Norihiro Watanabe) Date: Tue, 4 Nov 2014 17:41:25 +0100 Subject: [petsc-users] VecView in Ascii VTK mode In-Reply-To: <2F9CD9B4-3D8D-47EF-A6F9-19137DBBA414@mcs.anl.gov> References: <2F9CD9B4-3D8D-47EF-A6F9-19137DBBA414@mcs.anl.gov> Message-ID: thank you! On Tue, Nov 4, 2014 at 5:40 PM, Barry Smith wrote: > Ok, then I?ll have it not print that information at all. > > Barry > > > On Nov 4, 2014, at 10:37 AM, Norihiro Watanabe > wrote: > > > > as far as I read http://www.vtk.org/VTK/img/file-formats.pdf, there is > no comment marker in ASCII VTK format. > > > > Best, > > nori > > > > On Tue, Nov 4, 2014 at 5:22 PM, Barry Smith wrote: > > > > This is my fault. Is there a comment marker for ASCII VTK format? Let > me know what it is and I?ll fix the code to make those lines comments and > hence ignored by VTK > > > > Barry > > > > > On Nov 4, 2014, at 5:21 AM, Norihiro Watanabe > wrote: > > > > > > Hi, > > > > > > I've updated PETSc version from 3.4 to 3.5 and found that object names > are printed in VecView() even for ASCII VTK format. Is there any changes in > the VTK output since the new version? > > > > > > Our code is something like below. > > > > > > == > > > PetscViewerCreate(PETSC_COMM_WORLD, &viewer); > > > PetscViewerSetType(viewer, PETSCVIEWERASCII); > > > PetscViewerFileSetName(viewer, filename); > > > PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_VTK); > > > VecView(x, viewer); > > > == > > > > > > and VTK files created from v3.5 include additional two lines before > POINT_DATA as > > > > > > == > > > Vec Object:Concentration 4 MPI processes > > > type: mpi > > > POINT_DATA 1915 > > > SCALARS Concentration double 1 > > > LOOKUP_TABLE default > > > 0.007413 > > > 0.00741375 > > > 0.00741394 > > > .... > > > == > > > > > > > > > Best, > > > Nori > > > > > > -- > > > Norihiro Watanabe > > > > > > > > > > -- > > Norihiro Watanabe > > -- Norihiro Watanabe -------------- next part -------------- An HTML attachment was scrubbed... URL: From gbisht at lbl.gov Tue Nov 4 11:00:07 2014 From: gbisht at lbl.gov (Gautam Bisht) Date: Tue, 4 Nov 2014 09:00:07 -0800 Subject: [petsc-users] sundials results do not agree with beuler/rosw/pseudo In-Reply-To: <629E9E22-0D16-4BDF-9BA2-1B357A5BB1B0@mcs.anl.gov> References: <629E9E22-0D16-4BDF-9BA2-1B357A5BB1B0@mcs.anl.gov> Message-ID: Shri: Thanks for pointing out that I need IDA instead of CVODE from sundials, which isn't currently supported by PETSc. Are there any near term plans for supporting IDA via PETSc? Barry: I now realize that PSEUDO is applicable for steady-state problems. For the time being I will stick with BEULER and ROSW methods and look into PCFIELDSPLIT. -Gautam. On Tue, Nov 4, 2014 at 8:19 AM, Barry Smith wrote: > > Note: PSEUDO is a continuation method for solving a nonlinear system, it > doesn?t make sense to use it to replace other methods. > > That time step of 1 seems very large for SUNDIALs to be using > > Barry > > > On Nov 4, 2014, at 9:40 AM, Abhyankar, Shrirang G. > wrote: > > > > PETSc only supports the ODE solver (CVODE) from Sundials. > > > > Shri > > > > From: Gautam Bisht > > Date: Mon, 3 Nov 2014 22:56:58 -0800 > > To: > > Subject: [petsc-users] sundials results do not agree with > beuler/rosw/pseudo > > > > Hi, > > > > I'm solving subsurface flow equation in which the governing ODE is > reformulated as a system of DAE. I'm using PETSc TS+DMComposite to solve > the system with a LU preconditioner. I get comparable results for BEULER, > ROSW and PSEUDO ts_type. But results with SUNDIALS for even a single TS > step are significantly underestimated when compared to those obtained for > the other ts_types. I would appreciate if folks would suggest ideas on how > can I go about figuring out what is going wrong with SUNDIALS. > > > > I'm using following PETSc options: > > > > >/opt/local/bin/mpiexec -n 1 $EXEROOT/cesm.exe \ > > -ts_monitor \ > > -ts_view \ > > -snes_monitor \ > > -pc_type lu \ > > -ts_type sundials -ts_sundials_monitor_steps \ > > -ts_dt 1.0 -ts_final_time 1.0 > > > > 0 TS dt 1 time 0 > > 1 TS dt 1 time 1 > > TS Object: 1 MPI processes > > type: sundials > > maximum steps=100000 > > maximum time=1 > > total number of nonlinear solver iterations=0 > > total number of nonlinear solve failures=0 > > total number of linear solver iterations=0 > > total number of rejected steps=0 > > Sundials integrater does not use SNES! > > Sundials integrater type BDF: backward differentiation formula > > Sundials abs tol 1e-06 rel tol 1e-06 > > Sundials linear solver tolerance factor 0.05 > > Sundials max dimension of Krylov subspace 5 > > Sundials using unmodified (classical) Gram-Schmidt for > orthogonalization in GMRES > > Sundials suggested factor for tolerance scaling 1 > > Sundials cumulative number of internal steps 1 > > Sundials no. of calls to rhs function 2 > > Sundials no. of calls to linear solver setup function 1 > > Sundials no. of error test failures 0 > > Sundials no. of nonlinear solver iterations 1 > > Sundials no. of nonlinear convergence failure 0 > > Sundials no. of linear iterations 0 > > Sundials no. of linear convergence failures 0 > > PC Object: 1 MPI processes > > type: lu > > PC has not been set up so information may be incomplete > > LU: out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=200, cols=200 > > total: nonzeros=598, allocated nonzeros=3200 > > total number of mallocs used during MatSetValues calls =200 > > not using I-node routines > > Sundials no. of preconditioner evaluations 1 > > Sundials no. of preconditioner solves 0 > > Sundials no. of Jacobian-vector product evaluations 0 > > Sundials no. of rhs calls for finite diff. Jacobian-vector evals 0 > > > > Thanks, > > -Gautam. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Nov 4 11:21:29 2014 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 4 Nov 2014 12:21:29 -0500 Subject: [petsc-users] Multiple solves with PCMG fail In-Reply-To: <2052102.bo3NGSHfll@besikovitch-iii> References: <9621705.dtimTkejkQ@besikovitch-ii> <87y4s0xp36.fsf@jedbrown.org> <2052102.bo3NGSHfll@besikovitch-iii> Message-ID: On Tue, Nov 4, 2014 at 2:55 AM, Filippo Leonardi < filippo.leonardi at sam.math.ethz.ch> wrote: > Sorry for the late update. > > I tested the code with the branch: > ksp-zero-eig > > Did you make PETSc after the checking out the branch? Please send us the error message and run in debug mode if possible. > But the problem seems to still be there, also tried: > -ksp_chebyshev_estimate_eigenvalues_random > but does not change. > > Just tell me what test you need and I'll try to do them. > > Best, > Filippo > > On Wednesday 29 October 2014 10.30:14 Mark Adams wrote: > > > Yeah, that should work; an alternative is to defer the estimate until > > > the next solve. > > > > Sounds better. As I said one of my users has a zero RHS on the first > solve. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brianyang1106 at gmail.com Tue Nov 4 14:13:36 2014 From: brianyang1106 at gmail.com (Brian Yang) Date: Tue, 4 Nov 2014 14:13:36 -0600 Subject: [petsc-users] Does KSPSetInitialGuessNonzero mean that x with initial values and will be updated? Message-ID: Hi all, I am trying to solve Ax=0 with an initial values of x. I know KSPSetInitialGuessNonzero is able to disable the zero guess, so it will keep updating the initial x right? Thanks. -- Brian Yang U of Houston -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Nov 4 14:20:47 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 4 Nov 2014 14:20:47 -0600 Subject: [petsc-users] Does KSPSetInitialGuessNonzero mean that x with initial values and will be updated? In-Reply-To: References: Message-ID: > On Nov 4, 2014, at 2:13 PM, Brian Yang wrote: > > Hi all, > > I am trying to solve Ax=0 with an initial values of x. I know KSPSetInitialGuessNonzero is > able to disable the zero guess, so it will keep updating the initial x right? What do you mean by ?it will keep updating the initial x right?? What is ?it? and what do you mean by ?keep updating?. If you have not called KSPSetInitialGuessNonzero() then KSPSolve() will automatically zero x before starting the iteration. If you do call KSPSetInitialGuessNonzero() then KSPSolve will not automatically zero x before starting the iteration. That is all it does. Barry > > Thanks. > > -- > Brian Yang > U of Houston > > > From brianyang1106 at gmail.com Tue Nov 4 14:23:14 2014 From: brianyang1106 at gmail.com (Brian Yang) Date: Tue, 4 Nov 2014 14:23:14 -0600 Subject: [petsc-users] Does KSPSetInitialGuessNonzero mean that x with initial values and will be updated? In-Reply-To: References: Message-ID: Sorry, yeah, I mean KSPSolve using iterative ways like LSQR, CG and etc. Thanks. On Tue, Nov 4, 2014 at 2:20 PM, Barry Smith wrote: > > > On Nov 4, 2014, at 2:13 PM, Brian Yang wrote: > > > > Hi all, > > > > I am trying to solve Ax=0 with an initial values of x. I know > KSPSetInitialGuessNonzero is > > able to disable the zero guess, so it will keep updating the initial x > right? > > What do you mean by "it will keep updating the initial x right"? What > is "it" and what do you mean by "keep updating". > > If you have not called KSPSetInitialGuessNonzero() then KSPSolve() will > automatically zero x before starting the iteration. If you do call > KSPSetInitialGuessNonzero() then KSPSolve will not automatically zero x > before starting the iteration. That is all it does. > > Barry > > > > > Thanks. > > > > -- > > Brian Yang > > U of Houston > > > > > > > > -- Brian Yang U of Houston -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Nov 4 15:17:52 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 4 Nov 2014 15:17:52 -0600 Subject: [petsc-users] VecView in Ascii VTK mode In-Reply-To: References: <2F9CD9B4-3D8D-47EF-A6F9-19137DBBA414@mcs.anl.gov> Message-ID: Here is the patch you can apply to fix the problem. -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-ascii-vtk-output.patch Type: application/octet-stream Size: 713 bytes Desc: not available URL: -------------- next part -------------- After applying it run make gnumake in the PETSc root directory and then relink your application code. Sorry for the inconvenience and thanks for reporting the problem. Barry > On Nov 4, 2014, at 10:41 AM, Norihiro Watanabe wrote: > > thank you! > > On Tue, Nov 4, 2014 at 5:40 PM, Barry Smith wrote: > Ok, then I?ll have it not print that information at all. > > Barry > > > On Nov 4, 2014, at 10:37 AM, Norihiro Watanabe wrote: > > > > as far as I read http://www.vtk.org/VTK/img/file-formats.pdf, there is no comment marker in ASCII VTK format. > > > > Best, > > nori > > > > On Tue, Nov 4, 2014 at 5:22 PM, Barry Smith wrote: > > > > This is my fault. Is there a comment marker for ASCII VTK format? Let me know what it is and I?ll fix the code to make those lines comments and hence ignored by VTK > > > > Barry > > > > > On Nov 4, 2014, at 5:21 AM, Norihiro Watanabe wrote: > > > > > > Hi, > > > > > > I've updated PETSc version from 3.4 to 3.5 and found that object names are printed in VecView() even for ASCII VTK format. Is there any changes in the VTK output since the new version? > > > > > > Our code is something like below. > > > > > > == > > > PetscViewerCreate(PETSC_COMM_WORLD, &viewer); > > > PetscViewerSetType(viewer, PETSCVIEWERASCII); > > > PetscViewerFileSetName(viewer, filename); > > > PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_VTK); > > > VecView(x, viewer); > > > == > > > > > > and VTK files created from v3.5 include additional two lines before POINT_DATA as > > > > > > == > > > Vec Object:Concentration 4 MPI processes > > > type: mpi > > > POINT_DATA 1915 > > > SCALARS Concentration double 1 > > > LOOKUP_TABLE default > > > 0.007413 > > > 0.00741375 > > > 0.00741394 > > > .... > > > == > > > > > > > > > Best, > > > Nori > > > > > > -- > > > Norihiro Watanabe > > > > > > > > > > -- > > Norihiro Watanabe > > > > > -- > Norihiro Watanabe From jed at jedbrown.org Tue Nov 4 17:34:10 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 04 Nov 2014 18:34:10 -0500 Subject: [petsc-users] VecView in Ascii VTK mode In-Reply-To: References: Message-ID: <8761euee19.fsf@jedbrown.org> Norihiro Watanabe writes: > as far as I read http://www.vtk.org/VTK/img/file-formats.pdf, there is no > comment marker in ASCII VTK format. Note that for any use beyond the simplest debugging, you should use the VTK XML formats of PETSCVIEWERVTK. Write to *.vtr or *.vts for structured grids, or *.vtu for unstructured. These formats are much faster and more robust. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Tue Nov 4 18:14:58 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 04 Nov 2014 19:14:58 -0500 Subject: [petsc-users] Speed up KSPSolve of dense matrix In-Reply-To: <09B0443C-D63A-4173-A969-457188238E28@mcs.anl.gov> References: <5550582.ftmzpoOxMT@asaru> <09B0443C-D63A-4173-A969-457188238E28@mcs.anl.gov> Message-ID: <87tx2ecxkt.fsf@jedbrown.org> Barry Smith writes: >> I've tried to use a direct solver like suggested on pp 72, but: >> >> ./petBench 600 1 -ksp_type preonly -pc_type lu > > You cannot use LU with SBAIJ format. Only Cholesky. So use -pc_type cholesky And using a sparse matrix format for a dense matrix is a recipe for poor performance. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From filippo.leonardi at sam.math.ethz.ch Wed Nov 5 01:37:50 2014 From: filippo.leonardi at sam.math.ethz.ch (Filippo Leonardi) Date: Wed, 5 Nov 2014 08:37:50 +0100 Subject: [petsc-users] Multiple solves with PCMG fail In-Reply-To: References: <9621705.dtimTkejkQ@besikovitch-ii> <2052102.bo3NGSHfll@besikovitch-iii> Message-ID: <58968237.1VxD0XNj1Y@besikovitch-iii> @ Jed Can't figure out what solver is causing the problem actually, I tried evrerything: mg_levels, mg_coarse and nothing. On Tuesday 04 November 2014 12.21:29 Mark Adams wrote: > On Tue, Nov 4, 2014 at 2:55 AM, Filippo Leonardi < > > filippo.leonardi at sam.math.ethz.ch> wrote: > > Sorry for the late update. > > > > I tested the code with the branch: > > ksp-zero-eig > > Did you make PETSc after the checking out the branch? Yes. > Please send us the > error message and run in debug mode if possible. See below. > > > But the problem seems to still be there, also tried: > > -ksp_chebyshev_estimate_eigenvalues_random > > but does not change. > > > > Just tell me what test you need and I'll try to do them. > > > > Best, > > Filippo > > > > On Wednesday 29 October 2014 10.30:14 Mark Adams wrote: > > > > Yeah, that should work; an alternative is to defer the estimate until > > > > the next solve. > > > > > > Sounds better. As I said one of my users has a zero RHS on the first > > > > solve. ( Despite ARCH being arch-linux2-cxx-opt it's compiled in debug mode ) mpirun -np 4 ./test1 -mg_levels_ksp_chebyshev_estimate_eigenvalues_random - ksp_chebyshev_estimate_eigenvalues_random -pc_type mg -pc_mg_levels 5 - draw_pause -1 -ksp_view [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Null argument, when expecting valid pointer [1]PETSC ERROR: Null Pointer: Parameter # 3 [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Development GIT revision: v3.5.2-925-g233c798 GIT Date: 2014-11-03 17:44:15 -0700 [1]PETSC ERROR: ./test1 on a arch-linux2-cxx-opt named Besikovitch-III by filippo Wed Nov 5 08:32:52 2014 [1]PETSC ERROR: Configure options [1]PETSC ERROR: #1 KSPComputeEigenvalues() line 119 in /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: #2 KSPChebyshevComputeExtremeEigenvalues_Private() line 328 in /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/impls/cheby/cheby.c [1]PETSC ERROR: #3 KSPSolve_Chebyshev() line 373 in /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/impls/cheby/cheby.c [1]PETSC ERROR: #4 KSPSolve() line 460 in /home/filippo/Workspace/petsc- git/ksp-zero-eig/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: #5 PCMGMCycle_Private() line 19 in /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/pc/impls/mg/mg.c [1]PETSC ERROR: #6 PCApply_MG() line 337 in /home/filippo/Workspace/petsc- git/ksp-zero-eig/src/ksp/pc/impls/mg/mg.c -------------- next part -------------- A non-text attachment was scrubbed... Name: Filippo Leonardi.vcf Type: text/vcard Size: 322 bytes Desc: not available URL: From mailinglists at xgm.de Wed Nov 5 03:40:17 2014 From: mailinglists at xgm.de (Florian Lindner) Date: Wed, 05 Nov 2014 10:40:17 +0100 Subject: [petsc-users] Speed up KSPSolve of dense matrix In-Reply-To: <87tx2ecxkt.fsf@jedbrown.org> References: <5550582.ftmzpoOxMT@asaru> <09B0443C-D63A-4173-A969-457188238E28@mcs.anl.gov> <87tx2ecxkt.fsf@jedbrown.org> Message-ID: <6263118.r4tAurIbQI@asaru> Am Dienstag, 4. November 2014, 19:14:58 schrieb Jed Brown: > Barry Smith writes: > >> I've tried to use a direct solver like suggested on pp 72, but: > >> > >> ./petBench 600 1 -ksp_type preonly -pc_type lu > > > > You cannot use LU with SBAIJ format. Only Cholesky. So use -pc_type cholesky > > And using a sparse matrix format for a dense matrix is a recipe for poor performance. I'm more or less aware of that, but I'm surprised that the performance is such poor. I changed the format of matrixC to dense now, but the runtime is still poor. Overall time is now around 9s with 95% spent in KSPSolve and exact the same number of MatMult executions, taking 65% of time. Thx, Florian From mailinglists at xgm.de Wed Nov 5 03:41:41 2014 From: mailinglists at xgm.de (Florian Lindner) Date: Wed, 05 Nov 2014 10:41:41 +0100 Subject: [petsc-users] Speed up KSPSolve of dense matrix In-Reply-To: <09B0443C-D63A-4173-A969-457188238E28@mcs.anl.gov> References: <5550582.ftmzpoOxMT@asaru> <09B0443C-D63A-4173-A969-457188238E28@mcs.anl.gov> Message-ID: <1498803.iTGvHRNR5e@asaru> Am Dienstag, 4. November 2014, 10:08:27 schrieb Barry Smith: > > There are a lot of questions here. Yes, thanks for replying! > > On Nov 4, 2014, at 328 AM, Florian Lindner wrote: > > > > Hello, > > > > I have a fulll matrix of size e.g. 603x603 of which I'm very disappointed with the runtime, compared to a naive LU / forward / backward solution. My petsc solution takes about 14s, while the old one takes just 0.5s. (when you're looking at sparse matrices the figures are almost inversed). I try to investigate what's the problem. > > For small problems like this direct dense LU could easily be faster than iterative schemes. Especially if you have many right hand sides with the same matrix since the factorization only needs to be done one. > > > > > Most of the time is spent in the KSPSolve. > > > > Event Count Time (sec) Flops --- Global --- --- Stage --- Total > > Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > > ------------------------------------------------------------------------------------------------------------------------ > > > > --- Event Stage 0: Main Stage > > > > MatMult 10334 1.0 6.3801e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 45 49 0 0 0 45 49 0 0 0 1177 > > MatSolve 10334 1.0 6.9187e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 49 49 0 0 0 49 49 0 0 0 1085 > > If this is for a single solver for a 603 by 603 matrix then this is absurd. 10000 plus iterations But that's the output I got. When I run with -info I get a lot of messages: [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged though doing just on KSPSolve / KSPSetOperators. Is that ok? > > MatCholFctrNum 1 1.0 1.8968e-01 1.0 6.03e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > > KSPGMRESOrthog 10000 1.0 2.1417e-01 1.0 3.73e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 2 0 0 0 2 2 0 0 0 1744 > > KSPSolve 1 1.0 1.3763e+01 1.0 1.54e+10 1.0 0.0e+00 0.0e+00 0.0e+00 97100 0 0 0 97100 0 0 0 1120 > > Prealloc Matrix C 1 1.0 5.7458e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > Filling Matrix C 1 1.0 6.7984e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecMDot 10000 1.0 8.8615e-02 1.0 1.87e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 2106 > > VecMAXPY 10334 1.0 1.2585e-01 1.0 1.99e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1580 > > Prealloc Matrix A 1 1.0 1.1071e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > > Filling Matrix A 1 1.0 1.2574e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > > PCSetUp 1 1.0 1.9073e-01 1.0 6.03e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > > PCApply 10334 1.0 6.9233e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 49 49 0 0 0 49 49 0 0 0 1085 > > ------------------------------------------------------------------------------------------------------------------------ > > > > Removed all events with global %T == 0, but my own ones. > > > > Matrix type is sequential MATSBAIJ on PETSC_COMM_SELF. > > > > KSP is initalized once: > > > > KSPSetOperators(_solver, _matrixC.matrix, _matrixC.matrix); > > KSPSetTolerances(_solver, _solverRtol, PETSC_DEFAULT, PETSC_DEFAULT, PETSC_DEFAULT); > > KSPSetFromOptions(_solver); > > > > KSPSolve: > > > > ierr = KSPSolve(_solver, in.vector, p.vector); CHKERRV(ierr) > > > > KSPSolve may be executed multiple times with the same SetOperators acording to dimensionality of the data. Here it's just one. > > > > -ksp_view says: > > > > KSP Object: 1 MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-09, absolute=1e-50, divergence=10000 > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: icc > > 0 levels of fill > > tolerance for zero pivot 2.22045e-14 > > using Manteuffel shift [POSITIVE_DEFINITE] > > matrix ordering: natural > > factor fill ratio given 1, needed 1 > > Factored matrix follows: > > Mat Object: 1 MPI processes > > type: seqsbaij > > rows=603, cols=603 > > package used to perform factorization: petsc > > total: nonzeros=182103, allocated nonzeros=182103 > > total number of mallocs used during MatSetValues calls =0 > > block size is 1 > > linear system matrix = precond matrix: > > Mat Object: C 1 MPI processes > > type: seqsbaij > > rows=603, cols=603 > > total: nonzeros=182103, allocated nonzeros=182103 > > total number of mallocs used during MatSetValues calls =0 > > block size is 1 > > > > > > I've tried to use a direct solver like suggested on pp 72, but: > > > > ./petBench 600 1 -ksp_type preonly -pc_type lu > > You cannot use LU with SBAIJ format. Only Cholesky. So use -pc_type cholesky I tried that too, it gives me an error message: % ./petBench 100 1 -pc_type cholesky Mesh of size: 100 Do mappings: 1 KSP Init Do KSPSolve consistent [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot [0]PETSC ERROR: Zero pivot row 0 value 0 tolerance 2.22045e-14 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown [0]PETSC ERROR: ./petBench on a arch-linux2-c-debug named helium by lindnefn Wed Nov 5 10:22:07 2014 [0]PETSC ERROR: Configure options --with-debugging=1 [0]PETSC ERROR: #1 MatPivotCheck_none() line 622 in /data2/scratch/lindner/petsc/include/petsc-private/matimpl.h [0]PETSC ERROR: #2 MatPivotCheck() line 641 in /data2/scratch/lindner/petsc/include/petsc-private/matimpl.h [0]PETSC ERROR: #3 MatCholeskyFactorNumeric_SeqSBAIJ_1_NaturalOrdering() line 1435 in /data2/scratch/lindner/petsc/src/mat/impls/sbaij/seq/sbaijfact.c [0]PETSC ERROR: #4 MatCholeskyFactorNumeric() line 3063 in /data2/scratch/lindner/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #5 PCSetUp_Cholesky() line 150 in /data2/scratch/lindner/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c [0]PETSC ERROR: #6 PCSetUp() line 902 in /data2/scratch/lindner/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #7 KSPSetUp() line 305 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #8 KSPSolve() line 417 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 map() line 401 in /data2/scratch/lindner/precice/src/mapping/PetRadialBasisFctMapping.hpp FAQ says I likely have an error in my matrix generation when it occurs in the zeroth row. But for RBF the main diagonal is always BasisFunction( | x_n - x_n | ). Since I test with ThinPlateSplines as basis function here it's TPS(0) == 0. > > Mesh of size: 600 > > Do mappings: 1 > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: No support for this operation for this object type > > [0]PETSC ERROR: Factor type not supported > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown > > [0]PETSC ERROR: ./petBench on a arch-linux2-c-opt named helium by lindnefn Tue Nov 4 10:19:28 2014 > > [0]PETSC ERROR: Configure options --with-debugging=0 > > [0]PETSC ERROR: #1 MatGetFactor_seqsbaij_petsc() line 1833 in /data2/scratch/lindner/petsc/src/mat/impls/sbaij/seq/sbaij.c > > [0]PETSC ERROR: #2 MatGetFactor() line 3963 in /data2/scratch/lindner/petsc/src/mat/interface/matrix.c > > [0]PETSC ERROR: #3 PCSetUp_LU() line 125 in /data2/scratch/lindner/petsc/src/ksp/pc/impls/factor/lu/lu.c > > [0]PETSC ERROR: #4 PCSetUp() line 902 in /data2/scratch/lindner/petsc/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #5 KSPSetUp() line 305 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #6 KSPSolve() line 417 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #7 map() line 391 in /data2/scratch/lindner/precice/src/mapping/PetRadialBasisFctMapping.hpp Thanks, Florian From norihiro.w at gmail.com Wed Nov 5 08:33:02 2014 From: norihiro.w at gmail.com (Norihiro Watanabe) Date: Wed, 5 Nov 2014 15:33:02 +0100 Subject: [petsc-users] VecView in Ascii VTK mode In-Reply-To: <8761euee19.fsf@jedbrown.org> References: <8761euee19.fsf@jedbrown.org> Message-ID: Dear Barry, Many thanks for the patch. It works on my computer Dear Jed, Thanks for the advice. We are mainly using it for testing purpose but some of our colleagues prefer it because it can be easily processed with awk. Best regards, Nori On Wed, Nov 5, 2014 at 12:34 AM, Jed Brown wrote: > Norihiro Watanabe writes: > > > as far as I read http://www.vtk.org/VTK/img/file-formats.pdf, there is > no > > comment marker in ASCII VTK format. > > Note that for any use beyond the simplest debugging, you should use the > VTK XML formats of PETSCVIEWERVTK. Write to *.vtr or *.vts for > structured grids, or *.vtu for unstructured. These formats are much > faster and more robust. > -- Norihiro Watanabe -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Nov 5 08:36:26 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 5 Nov 2014 08:36:26 -0600 Subject: [petsc-users] Speed up KSPSolve of dense matrix In-Reply-To: <1498803.iTGvHRNR5e@asaru> References: <5550582.ftmzpoOxMT@asaru> <09B0443C-D63A-4173-A969-457188238E28@mcs.anl.gov> <1498803.iTGvHRNR5e@asaru> Message-ID: <365491BE-1733-4AEA-BF66-B4E9DDE7002F@mcs.anl.gov> So you have a dense, very ill-conditioned matrix. Seems to me you should just be using the dense LU/Cholesky factorization unless you know how to preconditioner these types of matrices. http://scicomp.stackexchange.com/questions/7561/do-rbf-kernel-matrices-tend-to-be-ill-conditioned You can?t expect to take monstrously ill-conditioned matrices and expect to use iterative methods unless you really really really understand the source and and control of the ill-conditioning. What is your end goal in trying to use iterative methods? Barry > On Nov 5, 2014, at 3:41 AM, Florian Lindner wrote: > > Am Dienstag, 4. November 2014, 10:08:27 schrieb Barry Smith: >> >> There are a lot of questions here. > > Yes, thanks for replying! > >>> On Nov 4, 2014, at 328 AM, Florian Lindner wrote: >>> >>> Hello, >>> >>> I have a fulll matrix of size e.g. 603x603 of which I'm very disappointed with the runtime, compared to a naive LU / forward / backward solution. My petsc solution takes about 14s, while the old one takes just 0.5s. (when you're looking at sparse matrices the figures are almost inversed). I try to investigate what's the problem. >> >> For small problems like this direct dense LU could easily be faster than iterative schemes. Especially if you have many right hand sides with the same matrix since the factorization only needs to be done one. >> >>> >>> Most of the time is spent in the KSPSolve. >>> >>> Event Count Time (sec) Flops --- Global --- --- Stage --- Total >>> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>> ------------------------------------------------------------------------------------------------------------------------ >>> >>> --- Event Stage 0: Main Stage >>> >>> MatMult 10334 1.0 6.3801e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 45 49 0 0 0 45 49 0 0 0 1177 >>> MatSolve 10334 1.0 6.9187e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 49 49 0 0 0 49 49 0 0 0 1085 >> >> If this is for a single solver for a 603 by 603 matrix then this is absurd. 10000 plus iterations > > But that's the output I got. When I run with -info I get a lot of messages: > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged > > though doing just on KSPSolve / KSPSetOperators. Is that ok? > >>> MatCholFctrNum 1 1.0 1.8968e-01 1.0 6.03e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>> KSPGMRESOrthog 10000 1.0 2.1417e-01 1.0 3.73e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 2 0 0 0 2 2 0 0 0 1744 >>> KSPSolve 1 1.0 1.3763e+01 1.0 1.54e+10 1.0 0.0e+00 0.0e+00 0.0e+00 97100 0 0 0 97100 0 0 0 1120 >>> Prealloc Matrix C 1 1.0 5.7458e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> Filling Matrix C 1 1.0 6.7984e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecMDot 10000 1.0 8.8615e-02 1.0 1.87e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 2106 >>> VecMAXPY 10334 1.0 1.2585e-01 1.0 1.99e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1580 >>> Prealloc Matrix A 1 1.0 1.1071e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>> Filling Matrix A 1 1.0 1.2574e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>> PCSetUp 1 1.0 1.9073e-01 1.0 6.03e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>> PCApply 10334 1.0 6.9233e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 49 49 0 0 0 49 49 0 0 0 1085 >>> ------------------------------------------------------------------------------------------------------------------------ >>> >>> Removed all events with global %T == 0, but my own ones. >>> >>> Matrix type is sequential MATSBAIJ on PETSC_COMM_SELF. >>> >>> KSP is initalized once: >>> >>> KSPSetOperators(_solver, _matrixC.matrix, _matrixC.matrix); >>> KSPSetTolerances(_solver, _solverRtol, PETSC_DEFAULT, PETSC_DEFAULT, PETSC_DEFAULT); >>> KSPSetFromOptions(_solver); >>> >>> KSPSolve: >>> >>> ierr = KSPSolve(_solver, in.vector, p.vector); CHKERRV(ierr) >>> >>> KSPSolve may be executed multiple times with the same SetOperators acording to dimensionality of the data. Here it's just one. >>> >>> -ksp_view says: >>> >>> KSP Object: 1 MPI processes >>> type: gmres >>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >>> GMRES: happy breakdown tolerance 1e-30 >>> maximum iterations=10000, initial guess is zero >>> tolerances: relative=1e-09, absolute=1e-50, divergence=10000 >>> left preconditioning >>> using PRECONDITIONED norm type for convergence test >>> PC Object: 1 MPI processes >>> type: icc >>> 0 levels of fill >>> tolerance for zero pivot 2.22045e-14 >>> using Manteuffel shift [POSITIVE_DEFINITE] >>> matrix ordering: natural >>> factor fill ratio given 1, needed 1 >>> Factored matrix follows: >>> Mat Object: 1 MPI processes >>> type: seqsbaij >>> rows=603, cols=603 >>> package used to perform factorization: petsc >>> total: nonzeros=182103, allocated nonzeros=182103 >>> total number of mallocs used during MatSetValues calls =0 >>> block size is 1 >>> linear system matrix = precond matrix: >>> Mat Object: C 1 MPI processes >>> type: seqsbaij >>> rows=603, cols=603 >>> total: nonzeros=182103, allocated nonzeros=182103 >>> total number of mallocs used during MatSetValues calls =0 >>> block size is 1 >>> >>> >>> I've tried to use a direct solver like suggested on pp 72, but: >>> >>> ./petBench 600 1 -ksp_type preonly -pc_type lu >> >> You cannot use LU with SBAIJ format. Only Cholesky. So use -pc_type cholesky > > I tried that too, it gives me an error message: > > % ./petBench 100 1 -pc_type cholesky > Mesh of size: 100 > Do mappings: 1 > KSP Init > Do KSPSolve consistent > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot > [0]PETSC ERROR: Zero pivot row 0 value 0 tolerance 2.22045e-14 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown > [0]PETSC ERROR: ./petBench on a arch-linux2-c-debug named helium by lindnefn Wed Nov 5 10:22:07 2014 > [0]PETSC ERROR: Configure options --with-debugging=1 > [0]PETSC ERROR: #1 MatPivotCheck_none() line 622 in /data2/scratch/lindner/petsc/include/petsc-private/matimpl.h > [0]PETSC ERROR: #2 MatPivotCheck() line 641 in /data2/scratch/lindner/petsc/include/petsc-private/matimpl.h > [0]PETSC ERROR: #3 MatCholeskyFactorNumeric_SeqSBAIJ_1_NaturalOrdering() line 1435 in /data2/scratch/lindner/petsc/src/mat/impls/sbaij/seq/sbaijfact.c > [0]PETSC ERROR: #4 MatCholeskyFactorNumeric() line 3063 in /data2/scratch/lindner/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #5 PCSetUp_Cholesky() line 150 in /data2/scratch/lindner/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c > [0]PETSC ERROR: #6 PCSetUp() line 902 in /data2/scratch/lindner/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #7 KSPSetUp() line 305 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #8 KSPSolve() line 417 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #9 map() line 401 in /data2/scratch/lindner/precice/src/mapping/PetRadialBasisFctMapping.hpp > > FAQ says I likely have an error in my matrix generation when it occurs in the zeroth row. But for RBF the main diagonal is always BasisFunction( | x_n - x_n | ). Since I test with ThinPlateSplines as basis function here it's TPS(0) == 0. > >>> Mesh of size: 600 >>> Do mappings: 1 >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Factor type not supported >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown >>> [0]PETSC ERROR: ./petBench on a arch-linux2-c-opt named helium by lindnefn Tue Nov 4 10:19:28 2014 >>> [0]PETSC ERROR: Configure options --with-debugging=0 >>> [0]PETSC ERROR: #1 MatGetFactor_seqsbaij_petsc() line 1833 in /data2/scratch/lindner/petsc/src/mat/impls/sbaij/seq/sbaij.c >>> [0]PETSC ERROR: #2 MatGetFactor() line 3963 in /data2/scratch/lindner/petsc/src/mat/interface/matrix.c >>> [0]PETSC ERROR: #3 PCSetUp_LU() line 125 in /data2/scratch/lindner/petsc/src/ksp/pc/impls/factor/lu/lu.c >>> [0]PETSC ERROR: #4 PCSetUp() line 902 in /data2/scratch/lindner/petsc/src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: #5 KSPSetUp() line 305 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: #6 KSPSolve() line 417 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: #7 map() line 391 in /data2/scratch/lindner/precice/src/mapping/PetRadialBasisFctMapping.hpp > > Thanks, > Florian From knepley at gmail.com Thu Nov 6 00:14:16 2014 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 6 Nov 2014 06:14:16 +0000 Subject: [petsc-users] petsc-3.5.2: ex12 with Neumann BC In-Reply-To: <54574EED.3060309@avignon.inra.fr> References: <524430F2.2020507@avignon.inra.fr> <524D3D5D.5090301@avignon.inra.fr> <525804C2.3000101@avignon.inra.fr> <5453AE09.5090806@avignon.inra.fr> <54574EED.3060309@avignon.inra.fr> Message-ID: On Mon, Nov 3, 2014 at 9:46 AM, Olivier Bonnefon < olivier.bonnefon at avignon.inra.fr> wrote: > Hello, > > Thank for your answer, I'll explain my trouble: > > My problem is that the BC Neumann leads to wrong result. > > With Dirichlet BC, I get: > ------------------------------- > > > ./ex12 -run_type full -refinement_limit 0.0 -bc_type dirichlet > -interpolate 0 -petscspace_order 1 -show_initial -show_solution > -dm_plex_print_fem 1 > ... > ... > Solution > Vec Object:potential 1 MPI processes > type: seq > 0.5 > > This result is correct. > > With Neuman BC, I get: > -------------------------------- > > > ./ex12 -run_type full -refinement_limit 0.0 -bc_type neumann > -interpolate 1 -petscspace_order 1 -show_initial -dm_plex_print_fem 1 > -show_solution -bd_petscspace_order 1 -snes_linesearch_monitor > -snes_monitor -ksp_monitor_true_residual -snes_converged_reason > -ksp_converged_reason > .... > .... > > > Solution > Vec Object:potential 1 MPI processes > type: seq > -0.75 > -0.583333 > 0.0833333 > -0.583333 > -0.333333 > 0.416667 > 0.0833333 > 0.416667 > 1.25 > > > That is not the values of the solution x*x+y*y. > Sorry, this is poor documentation on my part. I will fix it in the example. The Naumann solution explicitly discards the constant part, meaning \int_\Omega u = 0 If we look at my solution \int^1_0 dx \int^1_0 dy x^2 + y^2 = 2/3 However, this is for the continuum result, whereas we are enforcing this in the discrete case. Thus, what we really get is 0.75, since we are integrating linear patches instead of quadratic patches. After shifting by that, you still do not get exactly x^2 + y^2 since there is some discretization error. If you run with P2 you should get the exact answer, shifted down to eliminate the DC component. Thanks, Matt > > I tried many ksp options. > Moreover, the neumann BC with "-run_type full" is not cover in the list > https://bitbucket.org/petsc/petsc/src/fced3c3f9e703542693913793d15321603e40fe6/config/builder.py?at=master#cl-257 > > Do you know what is wrong ? > > Thanks, > > Olivier Bonnefon > > On 10/31/2014 06:50 PM, Matthew Knepley wrote: > > On Fri, Oct 31, 2014 at 10:43 AM, Olivier Bonnefon < > olivier.bonnefon at avignon.inra.fr> wrote: > >> Hello, >> >> I'm working on the snes/examples/tutorial/ex12 version 3.5.2. >> >> I didn't succed to run the simplest case with Neumann BC: >> >> ./ex12 -run_type full -refinement_limit 0.0 -bc_type neumann >> -interpolate 1 -petscspace_order 1 -show_initial -dm_plex_print_fem 1 >> -show_solution -bd_petscspace_order 1 -snes_linesearch_monitor >> -snes_monitor -ksp_monitor_true_residual -snes_converged_reason >> -ksp_converged_reason >> >> This leads to dofs negatives values. >> > > I do not understand what you mean. Please always mail the full error > message. > > >> Do you know the options to get a correct result with Neumann BC ? >> > > There are some tests here: > > > https://bitbucket.org/petsc/petsc/src/fced3c3f9e703542693913793d15321603e40fe6/config/builder.py?at=master#cl-257 > > Matt > > >> Regards, >> Olivier Bonnefon >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Olivier Bonnefon > INRA PACA-Avignon, Unit? BioSP > Tel: +33 (0)4 32 72 21 58 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From haakon at hakostra.net Thu Nov 6 02:24:37 2014 From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=) Date: Thu, 06 Nov 2014 09:24:37 +0100 Subject: [petsc-users] PETSc VecLoad not obeying HDF5 group? Message-ID: <545B3045.5060001@hakostra.net> Hi, I am trying to load a dataset from a HDF5 file and into a vector in PETSc. For that purpose I use VecView etc in the following way: PetscViewerHDF5Open(PETSC_COMM_WORLD, "input.h5", FILE_MODE_READ, &H5viewer); PetscViewerHDF5PushGroup(H5viewer, "/FIELDS"); VecLoad(gSol, H5viewer); PetscViewerDestroy(&H5viewer); And yes, I also check the return variables for errors, I have just omitted them from this example to make it easier to read. My problem is: If I place a dataset, with the correct name and dimensions, in the group "/FIELDS" in my input.h5, PETSc can't find it and won't load it. If I place the same dataset in the root of my HDF5 file, the dataset is loaded and everything works, even with PetscViewerHDF5PushGroup set to "/FIELDS"... Does actually VecLoad obey the group set by PetscViewerHDF5PushGroup? Best regards, H?kon Strandenes From jed at jedbrown.org Thu Nov 6 02:52:37 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 06 Nov 2014 11:52:37 +0300 Subject: [petsc-users] Speed up KSPSolve of dense matrix In-Reply-To: <6263118.r4tAurIbQI@asaru> References: <5550582.ftmzpoOxMT@asaru> <09B0443C-D63A-4173-A969-457188238E28@mcs.anl.gov> <87tx2ecxkt.fsf@jedbrown.org> <6263118.r4tAurIbQI@asaru> Message-ID: <87r3xg90dm.fsf@jedbrown.org> Florian Lindner writes: > I'm more or less aware of that, but I'm surprised that the performance is such poor. If you've ever written a naive triply-nested loop dgemm, you'd know that it gives up orders of magnitude. > I changed the format of matrixC to dense now, but the runtime is still poor. Overall time is now around 9s with 95% spent in KSPSolve and exact the same number of MatMult executions, taking 65% of time. Your solver is broken. It takes 10000 iterations and presumably returns KSP_DIVERGED_ITS (use -ksp_converged_reason to check). This could be due to GMRES restarts. You might have better results with -ksp_type cg (if your matrix is symmetric) or -ksp_type bcgs for nonsymmetric. But the bottom line is that you need a good preconditioner (or well-conditioned system) for iterative methods to make sense for dense matrices. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From mfadams at lbl.gov Thu Nov 6 10:27:59 2014 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 6 Nov 2014 11:27:59 -0500 Subject: [petsc-users] Multiple solves with PCMG fail In-Reply-To: <58968237.1VxD0XNj1Y@besikovitch-iii> References: <9621705.dtimTkejkQ@besikovitch-ii> <2052102.bo3NGSHfll@besikovitch-iii> <58968237.1VxD0XNj1Y@besikovitch-iii> Message-ID: Humm, not sure what the problem is. I used your test code to debug this branch. You might go to this line: [1]PETSC ERROR: #1 KSPComputeEigenvalues() line 119 in /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/interface/itfunc.c if (n) PetscValidScalarPointer(r,3); and put a print statement. n==0 in your case when it fails. On Wed, Nov 5, 2014 at 2:37 AM, Filippo Leonardi < filippo.leonardi at sam.math.ethz.ch> wrote: > @ Jed > > Can't figure out what solver is causing the problem actually, I tried > evrerything: mg_levels, mg_coarse and nothing. > > On Tuesday 04 November 2014 12.21:29 Mark Adams wrote: > > On Tue, Nov 4, 2014 at 2:55 AM, Filippo Leonardi < > > > > filippo.leonardi at sam.math.ethz.ch> wrote: > > > Sorry for the late update. > > > > > > I tested the code with the branch: > > > ksp-zero-eig > > > > Did you make PETSc after the checking out the branch? > > Yes. > > > Please send us the > > error message and run in debug mode if possible. > > See below. > > > > > > But the problem seems to still be there, also tried: > > > -ksp_chebyshev_estimate_eigenvalues_random > > > but does not change. > > > > > > Just tell me what test you need and I'll try to do them. > > > > > > Best, > > > Filippo > > > > > > On Wednesday 29 October 2014 10.30:14 Mark Adams wrote: > > > > > Yeah, that should work; an alternative is to defer the estimate > until > > > > > the next solve. > > > > > > > > Sounds better. As I said one of my users has a zero RHS on the first > > > > > > solve. > > ( Despite ARCH being arch-linux2-cxx-opt it's compiled in debug mode ) > > mpirun -np 4 ./test1 -mg_levels_ksp_chebyshev_estimate_eigenvalues_random - > ksp_chebyshev_estimate_eigenvalues_random -pc_type mg -pc_mg_levels 5 - > draw_pause -1 -ksp_view > > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Null argument, when expecting valid pointer > [1]PETSC ERROR: Null Pointer: Parameter # 3 > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for > trouble shooting. > [1]PETSC ERROR: Petsc Development GIT revision: v3.5.2-925-g233c798 GIT > Date: > 2014-11-03 17:44:15 -0700 > [1]PETSC ERROR: ./test1 on a arch-linux2-cxx-opt named Besikovitch-III by > filippo Wed Nov 5 08:32:52 2014 > [1]PETSC ERROR: Configure options > [1]PETSC ERROR: #1 KSPComputeEigenvalues() line 119 in > > /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: #2 KSPChebyshevComputeExtremeEigenvalues_Private() line > 328 in > > /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/impls/cheby/cheby.c > [1]PETSC ERROR: #3 KSPSolve_Chebyshev() line 373 in > > /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/impls/cheby/cheby.c > [1]PETSC ERROR: #4 KSPSolve() line 460 in /home/filippo/Workspace/petsc- > git/ksp-zero-eig/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: #5 PCMGMCycle_Private() line 19 in > /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/pc/impls/mg/mg.c > [1]PETSC ERROR: #6 PCApply_MG() line 337 in /home/filippo/Workspace/petsc- > git/ksp-zero-eig/src/ksp/pc/impls/mg/mg.c -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Thu Nov 6 10:48:19 2014 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 06 Nov 2014 17:48:19 +0100 Subject: [petsc-users] [SLEPc] Get convergence metrics Message-ID: An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Thu Nov 6 11:23:58 2014 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 6 Nov 2014 18:23:58 +0100 Subject: [petsc-users] [SLEPc] Get convergence metrics In-Reply-To: References: Message-ID: El 06/11/2014, a las 17:48, Alexander Grayver escribi?: > Dear all, > > I'm wondering if there is a way in SLEPc to get different convergence metrics for EPS/SVD. > In particular, I was thinking about plotting "# matrix-vector products" versus "# converged eigenpairs". But how to get this information? > > I want to test different methods and what interests me is how many matrix-vector products each method requires to get N eigen/singular pairs (this is because each matrix-vector product entails some costly computations in my case). > > Thanks, > Alexander > You could consider writing a custom EPS monitor with EPSMonitorSet. The monitor subroutine would receive as an argument the number of eigenvalues converged so far. You would also need to query the event counts for the "MatMult" event, but I am not sure if this can be done programmatically with the PetscLog* functions. Jose From evanum at gmail.com Thu Nov 6 13:54:39 2014 From: evanum at gmail.com (Evan Um) Date: Thu, 6 Nov 2014 11:54:39 -0800 Subject: [petsc-users] Error message of nnz cannot be greater than block row length from MatSetValues() Message-ID: Dear PETSC users, I hope that I can have a comment about errors I got during sparse symmetric matrix construction. In this example, I used three processes. The size of a test matrix is 52105-by-52105. The length of array d_nnz and o_nnz is 17461 at rank 0, 17111 at rank 1 and 17535 at rank 2. The two arrays exactly describe nnz of the diagonal and off-diagonal part. At rank 0, I tried to add 148767 element for the diagonal part and 5821 for the off-diagonal part. At rank 1, I tried to add 135826 and 19541 for the diagonal and off-diagonal parts, respectively. At rank 2, I tried to add 138155 and 0 for the diagonal and off-diagonal parts. Rank 0 has matrix rows from 0 to 17460; rank 1 from 17460 to 34570 and rank 2 from 34570 to 52104. During the run, I got an error message: nnz cannot be greater than block row length: local row 17124 value 38762560 rowlength 17368. My d_nnz and o_nnz arrays never have a value such as 38762560. Could anyone help me decode this error message? In advance, I appreciate your help. Evan // CODE MatCreate(PETSC_COMM_WORLD,&B); MatSetSizes(B,PETSC_DECIDE,PETSC_DECIDE,ngedges_internal,ngedges_internal); MatSetType(B,MATSBAIJ); MatMPISBAIJSetPreallocation(B,1,0,d_nnz,0,o_nnz); delete [] d_nnz; delete [] o_nnz; MatSetValues(B, nentries_mat_b_partitioned, mat_b_i_partitioned, nentries_mat_b_partitioned, mat_b_j_partitioned, mat_b_val_partitioned, ADD_VALUES); delete [] mat_b_i_partitioned; delete [] mat_b_j_partitioned; delete [] mat_b_val_partitioned; MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY); MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY); //Errors: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Argument out of range [1]PETSC ERROR: nnz cannot be greater than block row length: local row 17124 value 38762560 rowlength 17368 [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [1]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Thu Nov 6 11:34:23 2014 [1]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [1]PETSC ERROR: #1 MatSeqSBAIJSetPreallocation_SeqSBAIJ() line 1599 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/seq/sbaij.c [1]PETSC ERROR: #2 MatSeqSBAIJSetPreallocation() line 2054 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/seq/sbaij.c [1]PETSC ERROR: #3 MatMPISBAIJSetPreallocation_MPISBAIJ() line 1626 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/mpi/mpisbaij.c [1]PETSC ERROR: #4 MatMPISBAIJSetPreallocation() line 1930 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/mpi/mpisbaij.c [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Object is in wrong state [1]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 1 "mat" before MatSetValues() [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [1]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Thu Nov 6 11:34:23 2014 [1]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [1]PETSC ERROR: #5 MatSetValues() line 1110 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c PNT 7 Rank:1 [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Object is in wrong state [1]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 1 "mat" before MatAssemblyBegin() [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [1]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Thu Nov 6 11:34:23 2014 [1]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [1]PETSC ERROR: #6 MatAssemblyBegin() line 4799 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. [1]PETSC ERROR: [1] MatStashScatterEnd_Private line 112 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/utils/matstash.c [1]PETSC ERROR: [1] MatAssemblyEnd_MPISBAIJ line 517 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/mpi/mpisbaij.c [1]PETSC ERROR: [1] MatAssemblyEnd line 4892 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c [1]PETSC ERROR: [1] MatAssemblyBegin line 4796 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c [1]PETSC ERROR: [1] MatSetValues line 1103 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c [1]PETSC ERROR: [1] MatSeqSBAIJSetPreallocation_SeqSBAIJ line 1575 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/seq/sbaij.c [1]PETSC ERROR: [1] MatSeqSBAIJSetPreallocation line 2050 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/seq/sbaij.c [1]PETSC ERROR: [1] MatMPISBAIJSetPreallocation_MPISBAIJ line 1587 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/mpi/mpisbaij.c [1]PETSC ERROR: [1] MatMPISBAIJSetPreallocation line 1926 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/impls/sbaij/mpi/mpisbaij.c [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Signal received [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [1]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Thu Nov 6 11:34:23 2014 [1]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [1]PETSC ERROR: #7 User provided function() line 0 in unknown file PNT 5 Rank:2 PNT 5 Rank:0 [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [2]PETSC ERROR: likely location of problem given in stack below [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [2]PETSC ERROR: INSTEAD the line number of the start of the function [2]PETSC ERROR: is given. [2]PETSC ERROR: [2] MatSetValues line 1103 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Signal received [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [2]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Thu Nov 6 11:34:23 2014 [2]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [2]PETSC ERROR: #1 User provided function() line 0 in unknown file [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] MatSetValues line 1103 /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/mat/interface/matrix.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [0]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Thu Nov 6 11:34:23 2014 [0]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [0]PETSC ERROR: #1 User provided function() line 0 in unknown file -------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Nov 6 14:23:42 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 06 Nov 2014 23:23:42 +0300 Subject: [petsc-users] Error message of nnz cannot be greater than block row length from MatSetValues() In-Reply-To: References: Message-ID: <87a94484dt.fsf@jedbrown.org> Evan Um writes: > Dear PETSC users, > > I hope that I can have a comment about errors I got during sparse symmetric > matrix construction. In this example, I used three processes. The size of a > test matrix is 52105-by-52105. The length of array d_nnz and o_nnz is 17461 > at rank 0, 17111 at rank 1 and 17535 at rank 2. The two arrays exactly > describe nnz of the diagonal and off-diagonal part. At rank 0, I tried > to add 148767 element for the diagonal part and 5821 for the off-diagonal > part. At rank 1, I tried to add 135826 and 19541 for the diagonal and > off-diagonal parts, respectively. Do these numbers mean that the sum of all the entries in the the arrays are 148767, 135826, etc? > At rank 2, I tried to add 138155 and 0 for the diagonal and > off-diagonal parts. Rank 0 has matrix rows from 0 to 17460; rank 1 > from 17460 to 34570 and rank 2 from 34570 to 52104. During the run, I > got an error message: nnz cannot be greater than block row length: > local row 17124 value 38762560 rowlength 17368. My d_nnz and o_nnz > arrays never have a value such as 38762560. Use MatGetOwnershipRange() and/or pass the local sizes to MatSetSizes. Your arrays are the wrong length, which is why the error refers to local row 17124 or rank 1 (which is larger than the array length of 17111). Valgrind would have told you this. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From evanum at gmail.com Thu Nov 6 18:40:31 2014 From: evanum at gmail.com (Evan Um) Date: Thu, 6 Nov 2014 16:40:31 -0800 Subject: [petsc-users] Error message of nnz cannot be greater than block row length from MatSetValues() In-Reply-To: <87a94484dt.fsf@jedbrown.org> References: <87a94484dt.fsf@jedbrown.org> Message-ID: Dear Jed, Thanks for your help many times. These numbers mean the total number of elements to be added using MatSetValues(). For example, at process 0, 148767+5821 elements are added to matrix B. In other words, the length of arrays (i.e. mat_b_i_partitioned, mat_b_j_partitioned and mat_b_val_partitioned) is 148767+5821. I used "ADD_VALUES" but "INSERT_VALUES" should be the same for my work. As you suggested, I explicitly passed the number of local rows to MatSetSizes as shown below. The number of local rows agrees with a range of continuous row indices stored at array mat_b_i_partitioned. I also explicitly set up the number of local column=the matrix size. In this case, I get error message: "Sum of local lengths 156315 does not equal global length 52105, my local length 52105". I used 3 processes. I guess that number 156315 seems to be related with 3*size of my global column size. In my problem, the matrix is partitioned by rows but not by columns. In this case, how can I specify the local column size in MatSetSizes()? Evan // CODE MatCreate(PETSC_COMM_WORLD,&B); MatSetSizes(B, partitioned_row_size, ngedges_internal, ngedges_internal, ngedges_internal); MatSetType(B,MATSBAIJ); MatMPISBAIJSetPreallocation(B,1,0,d_nnz,0,o_nnz); MatSetValues(B, nentries_mat_b_partitioned, mat_b_i_partitioned, nentries_mat_b_partitioned, mat_b_j_partitioned, mat_b_val_partitioned, ADD_VALUES); MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY); MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY); //ERROR: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Sum of local lengths 156315 does not equal global length 52105, my local length 52105 likely a call to VecSetSizes() or MatSetSizes() is wrong. On Thu, Nov 6, 2014 at 12:23 PM, Jed Brown wrote: > Evan Um writes: > > > Dear PETSC users, > > > > I hope that I can have a comment about errors I got during sparse > symmetric > > matrix construction. In this example, I used three processes. The size > of a > > test matrix is 52105-by-52105. The length of array d_nnz and o_nnz is > 17461 > > at rank 0, 17111 at rank 1 and 17535 at rank 2. The two arrays exactly > > describe nnz of the diagonal and off-diagonal part. At rank 0, I tried > > to add 148767 element for the diagonal part and 5821 for the off-diagonal > > part. At rank 1, I tried to add 135826 and 19541 for the diagonal and > > off-diagonal parts, respectively. > > Do these numbers mean that the sum of all the entries in the the arrays > are 148767, 135826, etc? > > > At rank 2, I tried to add 138155 and 0 for the diagonal and > > off-diagonal parts. Rank 0 has matrix rows from 0 to 17460; rank 1 > > from 17460 to 34570 and rank 2 from 34570 to 52104. During the run, I > > got an error message: nnz cannot be greater than block row length: > > local row 17124 value 38762560 rowlength 17368. My d_nnz and o_nnz > > arrays never have a value such as 38762560. > > Use MatGetOwnershipRange() and/or pass the local sizes to MatSetSizes. > Your arrays are the wrong length, which is why the error refers to local > row 17124 or rank 1 (which is larger than the array length of 17111). > Valgrind would have told you this. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Nov 6 18:48:59 2014 From: jed at jedbrown.org (Jed Brown) Date: Fri, 07 Nov 2014 03:48:59 +0300 Subject: [petsc-users] Error message of nnz cannot be greater than block row length from MatSetValues() In-Reply-To: References: <87a94484dt.fsf@jedbrown.org> Message-ID: <877fz77s3o.fsf@jedbrown.org> Evan Um writes: > Dear Jed, > > Thanks for your help many times. These numbers mean the total number of > elements to be added using MatSetValues(). For example, at process 0, > 148767+5821 elements are added to matrix B. In other words, the length of > arrays (i.e. mat_b_i_partitioned, mat_b_j_partitioned and > mat_b_val_partitioned) > is 148767+5821. I used "ADD_VALUES" but "INSERT_VALUES" should be the same > for my work. > > As you suggested, I explicitly passed the number of local rows to > MatSetSizes as shown below. The number of local rows agrees with a range of > continuous row indices stored at array mat_b_i_partitioned. I also > explicitly set up the number of local column=the matrix size. In this case, > I get error message: "Sum of local lengths 156315 does not equal global > length 52105, my local length 52105". I used 3 processes. I guess > that number 156315 seems to be related with 3*size of my global column > size. In my problem, the matrix is partitioned by rows but not by columns. Column ownership there refers to the distributiov of a vector that your matrix can be multiplied with, not with the ownership of entries (that is row-based for all of these formats, though others like MATELEMENTAL use totally different distributions for the entries). Since you have a square matrix, you probably want MatSetSizes(B, partitioned_row_size, partitioned_row_size, ngedges_internal, ngedges_internal); -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From evanum at gmail.com Thu Nov 6 19:04:04 2014 From: evanum at gmail.com (Evan Um) Date: Thu, 6 Nov 2014 17:04:04 -0800 Subject: [petsc-users] Error message of nnz cannot be greater than block row length from MatSetValues() In-Reply-To: <877fz77s3o.fsf@jedbrown.org> References: <87a94484dt.fsf@jedbrown.org> <877fz77s3o.fsf@jedbrown.org> Message-ID: Dear Jed, In my problem, each process has a part of contiguous rows of matrix B. In this partitioned matrix, locations of non-zero elements are irregular because it is a part of unstructured matrix. It is hard to define the distribution of a vector that the partitioned matrix B is multiplied with. Both MatSetSizes(B, partitioned_row_size, partitioned_row_size, ngedges_internal, ngedges_internal) and MatSetSizes(B, partitioned_row_size, PETSC_DECIDE, ngedges_internal, ngedges_internal) produced segmentation fault errors. How should MatSetSizes() be used for row-based-partitioned unstructured matrices? Evan On Thu, Nov 6, 2014 at 4:48 PM, Jed Brown wrote: > Evan Um writes: > > > Dear Jed, > > > > Thanks for your help many times. These numbers mean the total number of > > elements to be added using MatSetValues(). For example, at process 0, > > 148767+5821 elements are added to matrix B. In other words, the length of > > arrays (i.e. mat_b_i_partitioned, mat_b_j_partitioned and > > mat_b_val_partitioned) > > is 148767+5821. I used "ADD_VALUES" but "INSERT_VALUES" should be the > same > > for my work. > > > > As you suggested, I explicitly passed the number of local rows to > > MatSetSizes as shown below. The number of local rows agrees with a range > of > > continuous row indices stored at array mat_b_i_partitioned. I also > > explicitly set up the number of local column=the matrix size. In this > case, > > I get error message: "Sum of local lengths 156315 does not equal global > > length 52105, my local length 52105". I used 3 processes. I guess > > that number 156315 seems to be related with 3*size of my global column > > size. In my problem, the matrix is partitioned by rows but not by > columns. > > Column ownership there refers to the distributiov of a vector that your > matrix can be multiplied with, not with the ownership of entries (that > is row-based for all of these formats, though others like MATELEMENTAL > use totally different distributions for the entries). Since you have a > square matrix, you probably want > > MatSetSizes(B, partitioned_row_size, partitioned_row_size, > ngedges_internal, ngedges_internal); > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Nov 6 19:14:30 2014 From: jed at jedbrown.org (Jed Brown) Date: Fri, 07 Nov 2014 04:14:30 +0300 Subject: [petsc-users] Error message of nnz cannot be greater than block row length from MatSetValues() In-Reply-To: References: <87a94484dt.fsf@jedbrown.org> <877fz77s3o.fsf@jedbrown.org> Message-ID: <874mub7qx5.fsf@jedbrown.org> Evan Um writes: > Dear Jed, > > In my problem, each process has a part of contiguous rows of matrix B. In > this partitioned matrix, locations of non-zero elements are irregular > because it is a part of unstructured matrix. It is hard to define the > distribution of a vector that the partitioned matrix B is multiplied > with. You have a square matrix, so the row and column distribution is the same. In MatMult(A,x,y), both x and y will have the same distribution. > Both MatSetSizes(B, partitioned_row_size, partitioned_row_size, > ngedges_internal, ngedges_internal) We need error messages. Always. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jychang48 at gmail.com Thu Nov 6 22:38:44 2014 From: jychang48 at gmail.com (Justin Chang) Date: Thu, 6 Nov 2014 22:38:44 -0600 Subject: [petsc-users] Parallel IO for VecLoad? Message-ID: Hi all, So I have a binary datafile with auxiliary data for a vector in my FEM code and I want it to be read in parallel. If I have a vector created from DMCreateLocalVector and assuming that the binary auxiliary datafile has the correct global number of entries, can VecLoad read the file in parallel? I don't see any examples illustrating this, but if there are any can someone point me to them? Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Nov 6 22:55:55 2014 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 7 Nov 2014 04:55:55 +0000 Subject: [petsc-users] Parallel IO for VecLoad? In-Reply-To: References: Message-ID: On Fri, Nov 7, 2014 at 4:38 AM, Justin Chang wrote: > Hi all, > > So I have a binary datafile with auxiliary data for a vector in my FEM > code and I want it to be read in parallel. If I have a vector created from > DMCreateLocalVector and assuming that the binary auxiliary datafile has the > correct global number of entries, can VecLoad read the file in parallel? I > don't see any examples illustrating this, but if there are any can someone > point me to them? > VecLoad() works with files generated by VecView(). Otherwise, you have to load the data yourself. I recommend writing a small program to load your data in serial and save it as a PETSc binary file. Then loading in parallel will work using VecLoad. Thanks, Matt > Thanks, > Justin > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From luchao at mail.iggcas.ac.cn Fri Nov 7 03:03:13 2014 From: luchao at mail.iggcas.ac.cn (=?GBK?B?wsCzrA==?=) Date: Fri, 7 Nov 2014 17:03:13 +0800 (GMT+08:00) Subject: [petsc-users] Matrix Create Message-ID: <45d5c1.21327.149897e42cd.Coremail.luchao@mail.iggcas.ac.cn> Dear all, Now, I need create a big matrix, but only some rows have values, other rows are all 0. The spy of the matrix looks like the figure below. Can you tell me how to create this kind of matrix? Thank you for your help LV Chao 2014/11/7 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: kk.png Type: image/png Size: 7382 bytes Desc: not available URL: From lawrence.mitchell at imperial.ac.uk Fri Nov 7 05:04:59 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Fri, 7 Nov 2014 11:04:59 +0000 Subject: [petsc-users] Nullspaces for schur complement PCs Message-ID: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> Hi petsc-dev, I'm solving a pure Neumann mixed Poisson-like problem, preconditioning with a schur complement. The pressure space has a nullspace of the constant functions and so I attach the appropriate nullspace to the krylov solver, and compose the constant nullspace with the IS defining the pressure space. My RHS is consistent. When I precondition with: -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type full \ -pc_fieldsplit_schur_precondition selfp I notice that the nullspace is not transferred over to the preconditioning matrix for S. Is this a deliberate choice? ksp_view output below, note that the schurcomplement mat has an attached nullspace, but the generated pmat does not. Cheers, Lawrence KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=30, initial guess is zero tolerances: relative=1e-07, absolute=1e-50, divergence=10000 left preconditioning has attached null space using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: fieldsplit FieldSplit with Schur preconditioner, factorization FULL Preconditioner for the Schur complement formed from Sp, an assembled approximation to S, which uses (the lumped) A00's diagonal's inverse Split info: Split number 0 Defined by IS Split number 1 Defined by IS KSP solver for A00 block KSP Object: (fieldsplit_0_) 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: ilu ILU: out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=72, cols=72 package used to perform factorization: petsc total: nonzeros=1080, allocated nonzeros=1080 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 23 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=72, cols=72 total: nonzeros=1080, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 23 nodes, limit used is 5 KSP solver for S = A11 - A10 inv(A00) A01 KSP Object: (fieldsplit_1_) 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (fieldsplit_1_) 1 MPI processes type: ilu ILU: out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=24, cols=24 package used to perform factorization: petsc total: nonzeros=216, allocated nonzeros=216 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 8 nodes, limit used is 5 linear system matrix followed by preconditioner matrix: Mat Object: (fieldsplit_1_) 1 MPI processes type: schurcomplement rows=24, cols=24 has attached null space Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_1_) 1 MPI processes type: seqaij rows=24, cols=24 total: nonzeros=72, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 has attached null space using I-node routines: found 8 nodes, limit used is 5 A10 Mat Object: 1 MPI processes type: seqaij rows=24, cols=72 total: nonzeros=288, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 8 nodes, limit used is 5 KSP of A00 KSP Object: (fieldsplit_0_) 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: ilu ILU: out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=72, cols=72 package used to perform factorization: petsc total: nonzeros=1080, allocated nonzeros=1080 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 23 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=72, cols=72 total: nonzeros=1080, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 23 nodes, limit used is 5 A01 Mat Object: 1 MPI processes type: seqaij rows=72, cols=24 total: nonzeros=288, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 23 nodes, limit used is 5 Mat Object: 1 MPI processes type: seqaij rows=24, cols=24 total: nonzeros=216, allocated nonzeros=216 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 8 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: nest rows=96, cols=96 has attached null space Matrix object: type=nest, rows=2, cols=2 MatNest structure: (0,0) : prefix="fieldsplit_0_", type=seqaij, rows=72, cols=72 (0,1) : type=seqaij, rows=72, cols=24 (1,0) : type=seqaij, rows=24, cols=72 (1,1) : prefix="fieldsplit_1_", type=seqaij, rows=24, cols=24 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jed at jedbrown.org Fri Nov 7 05:20:11 2014 From: jed at jedbrown.org (Jed Brown) Date: Fri, 07 Nov 2014 14:20:11 +0300 Subject: [petsc-users] Matrix Create In-Reply-To: <45d5c1.21327.149897e42cd.Coremail.luchao@mail.iggcas.ac.cn> References: <45d5c1.21327.149897e42cd.Coremail.luchao@mail.iggcas.ac.cn> Message-ID: <87wq775kb8.fsf@jedbrown.org> ?? writes: > Dear all, > > Now, I need create a big matrix, but only some rows have values, > other rows are all 0. The spy of the matrix looks like the > figure below. You can create a matrix like this, but the default distribution is for balancing rows. Pass local sizes to MatSetSizes if you want something different. You'll get better performance if you scatter to the reduced space, then scatter back). > Can you tell me how to create this kind of matrix? > > Thank you for your help > > LV Chao > > 2014/11/7 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From mailinglists at xgm.de Fri Nov 7 07:03:28 2014 From: mailinglists at xgm.de (Florian Lindner) Date: Fri, 07 Nov 2014 14:03:28 +0100 Subject: [petsc-users] Speed up KSPSolve of dense matrix In-Reply-To: <365491BE-1733-4AEA-BF66-B4E9DDE7002F@mcs.anl.gov> References: <5550582.ftmzpoOxMT@asaru> <1498803.iTGvHRNR5e@asaru> <365491BE-1733-4AEA-BF66-B4E9DDE7002F@mcs.anl.gov> Message-ID: <1712746.eHjgV37Em5@asaru> Hey, I've modified my code that it counts the total number of non-zeros (trivial to put in the preallocation code) and decides upon that wheather to create a MATSBAIJ or MATDENSE matrix. Using sparse matrices (for one that is really dense) I've made good experiences with -pc_factor_shift_type NONZERO -pc_type cholesky -ksp_type preonly. But when I use that on a MATDENSE I again get a a Zero pivot in LU factorization though using pc_factor_shift_type. I suspect that is related to the MATDENSE matrix intended to be symmetric, but actually is not. I have MAT_SYMMETRY_ETERNAL set but when I view the matrix only the upper right triangular is set, how I did in my matrix assembly algorithm. Do the algorithm treat the matrix as symmetric when MAT_SYMMETRY_ETERNAL is set or is just a little hint? Any way to actually set a MATDENSE matrix to be symmetric or do I need to copy the entries to the lower triangular? Thanks, Florian Am Mittwoch, 5. November 2014, 08:36:26 schrieb Barry Smith: > > So you have a dense, very ill-conditioned matrix. Seems to me you should just be using the dense LU/Cholesky factorization unless you know how to preconditioner these types of matrices. http://scicomp.stackexchange.com/questions/7561/do-rbf-kernel-matrices-tend-to-be-ill-conditioned > > You can?t expect to take monstrously ill-conditioned matrices and expect to use iterative methods unless you really really really understand the source and and control of the ill-conditioning. > > What is your end goal in trying to use iterative methods? > > Barry > > > > On Nov 5, 2014, at 3:41 AM, Florian Lindner wrote: > > > > Am Dienstag, 4. November 2014, 10:08:27 schrieb Barry Smith: > >> > >> There are a lot of questions here. > > > > Yes, thanks for replying! > > > >>> On Nov 4, 2014, at 328 AM, Florian Lindner wrote: > >>> > >>> Hello, > >>> > >>> I have a fulll matrix of size e.g. 603x603 of which I'm very disappointed with the runtime, compared to a naive LU / forward / backward solution. My petsc solution takes about 14s, while the old one takes just 0.5s. (when you're looking at sparse matrices the figures are almost inversed). I try to investigate what's the problem. > >> > >> For small problems like this direct dense LU could easily be faster than iterative schemes. Especially if you have many right hand sides with the same matrix since the factorization only needs to be done one. > >> > >>> > >>> Most of the time is spent in the KSPSolve. > >>> > >>> Event Count Time (sec) Flops --- Global --- --- Stage --- Total > >>> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > >>> ------------------------------------------------------------------------------------------------------------------------ > >>> > >>> --- Event Stage 0: Main Stage > >>> > >>> MatMult 10334 1.0 6.3801e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 45 49 0 0 0 45 49 0 0 0 1177 > >>> MatSolve 10334 1.0 6.9187e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 49 49 0 0 0 49 49 0 0 0 1085 > >> > >> If this is for a single solver for a 603 by 603 matrix then this is absurd. 10000 plus iterations > > > > But that's the output I got. When I run with -info I get a lot of messages: > > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged > > > > though doing just on KSPSolve / KSPSetOperators. Is that ok? > > > >>> MatCholFctrNum 1 1.0 1.8968e-01 1.0 6.03e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > >>> KSPGMRESOrthog 10000 1.0 2.1417e-01 1.0 3.73e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 2 0 0 0 2 2 0 0 0 1744 > >>> KSPSolve 1 1.0 1.3763e+01 1.0 1.54e+10 1.0 0.0e+00 0.0e+00 0.0e+00 97100 0 0 0 97100 0 0 0 1120 > >>> Prealloc Matrix C 1 1.0 5.7458e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > >>> Filling Matrix C 1 1.0 6.7984e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > >>> VecMDot 10000 1.0 8.8615e-02 1.0 1.87e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 2106 > >>> VecMAXPY 10334 1.0 1.2585e-01 1.0 1.99e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1580 > >>> Prealloc Matrix A 1 1.0 1.1071e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > >>> Filling Matrix A 1 1.0 1.2574e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > >>> PCSetUp 1 1.0 1.9073e-01 1.0 6.03e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > >>> PCApply 10334 1.0 6.9233e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 49 49 0 0 0 49 49 0 0 0 1085 > >>> ------------------------------------------------------------------------------------------------------------------------ > >>> > >>> Removed all events with global %T == 0, but my own ones. > >>> > >>> Matrix type is sequential MATSBAIJ on PETSC_COMM_SELF. > >>> > >>> KSP is initalized once: > >>> > >>> KSPSetOperators(_solver, _matrixC.matrix, _matrixC.matrix); > >>> KSPSetTolerances(_solver, _solverRtol, PETSC_DEFAULT, PETSC_DEFAULT, PETSC_DEFAULT); > >>> KSPSetFromOptions(_solver); > >>> > >>> KSPSolve: > >>> > >>> ierr = KSPSolve(_solver, in.vector, p.vector); CHKERRV(ierr) > >>> > >>> KSPSolve may be executed multiple times with the same SetOperators acording to dimensionality of the data. Here it's just one. > >>> > >>> -ksp_view says: > >>> > >>> KSP Object: 1 MPI processes > >>> type: gmres > >>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > >>> GMRES: happy breakdown tolerance 1e-30 > >>> maximum iterations=10000, initial guess is zero > >>> tolerances: relative=1e-09, absolute=1e-50, divergence=10000 > >>> left preconditioning > >>> using PRECONDITIONED norm type for convergence test > >>> PC Object: 1 MPI processes > >>> type: icc > >>> 0 levels of fill > >>> tolerance for zero pivot 2.22045e-14 > >>> using Manteuffel shift [POSITIVE_DEFINITE] > >>> matrix ordering: natural > >>> factor fill ratio given 1, needed 1 > >>> Factored matrix follows: > >>> Mat Object: 1 MPI processes > >>> type: seqsbaij > >>> rows=603, cols=603 > >>> package used to perform factorization: petsc > >>> total: nonzeros=182103, allocated nonzeros=182103 > >>> total number of mallocs used during MatSetValues calls =0 > >>> block size is 1 > >>> linear system matrix = precond matrix: > >>> Mat Object: C 1 MPI processes > >>> type: seqsbaij > >>> rows=603, cols=603 > >>> total: nonzeros=182103, allocated nonzeros=182103 > >>> total number of mallocs used during MatSetValues calls =0 > >>> block size is 1 > >>> > >>> > >>> I've tried to use a direct solver like suggested on pp 72, but: > >>> > >>> ./petBench 600 1 -ksp_type preonly -pc_type lu > >> > >> You cannot use LU with SBAIJ format. Only Cholesky. So use -pc_type cholesky > > > > I tried that too, it gives me an error message: > > > > % ./petBench 100 1 -pc_type cholesky > > Mesh of size: 100 > > Do mappings: 1 > > KSP Init > > Do KSPSolve consistent > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot > > [0]PETSC ERROR: Zero pivot row 0 value 0 tolerance 2.22045e-14 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown > > [0]PETSC ERROR: ./petBench on a arch-linux2-c-debug named helium by lindnefn Wed Nov 5 10:22:07 2014 > > [0]PETSC ERROR: Configure options --with-debugging=1 > > [0]PETSC ERROR: #1 MatPivotCheck_none() line 622 in /data2/scratch/lindner/petsc/include/petsc-private/matimpl.h > > [0]PETSC ERROR: #2 MatPivotCheck() line 641 in /data2/scratch/lindner/petsc/include/petsc-private/matimpl.h > > [0]PETSC ERROR: #3 MatCholeskyFactorNumeric_SeqSBAIJ_1_NaturalOrdering() line 1435 in /data2/scratch/lindner/petsc/src/mat/impls/sbaij/seq/sbaijfact.c > > [0]PETSC ERROR: #4 MatCholeskyFactorNumeric() line 3063 in /data2/scratch/lindner/petsc/src/mat/interface/matrix.c > > [0]PETSC ERROR: #5 PCSetUp_Cholesky() line 150 in /data2/scratch/lindner/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c > > [0]PETSC ERROR: #6 PCSetUp() line 902 in /data2/scratch/lindner/petsc/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #7 KSPSetUp() line 305 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #8 KSPSolve() line 417 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #9 map() line 401 in /data2/scratch/lindner/precice/src/mapping/PetRadialBasisFctMapping.hpp > > > > FAQ says I likely have an error in my matrix generation when it occurs in the zeroth row. But for RBF the main diagonal is always BasisFunction( | x_n - x_n | ). Since I test with ThinPlateSplines as basis function here it's TPS(0) == 0. > > > >>> Mesh of size: 600 > >>> Do mappings: 1 > >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > >>> [0]PETSC ERROR: No support for this operation for this object type > >>> [0]PETSC ERROR: Factor type not supported > >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > >>> [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown > >>> [0]PETSC ERROR: ./petBench on a arch-linux2-c-opt named helium by lindnefn Tue Nov 4 10:19:28 2014 > >>> [0]PETSC ERROR: Configure options --with-debugging=0 > >>> [0]PETSC ERROR: #1 MatGetFactor_seqsbaij_petsc() line 1833 in /data2/scratch/lindner/petsc/src/mat/impls/sbaij/seq/sbaij.c > >>> [0]PETSC ERROR: #2 MatGetFactor() line 3963 in /data2/scratch/lindner/petsc/src/mat/interface/matrix.c > >>> [0]PETSC ERROR: #3 PCSetUp_LU() line 125 in /data2/scratch/lindner/petsc/src/ksp/pc/impls/factor/lu/lu.c > >>> [0]PETSC ERROR: #4 PCSetUp() line 902 in /data2/scratch/lindner/petsc/src/ksp/pc/interface/precon.c > >>> [0]PETSC ERROR: #5 KSPSetUp() line 305 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c > >>> [0]PETSC ERROR: #6 KSPSolve() line 417 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c > >>> [0]PETSC ERROR: #7 map() line 391 in /data2/scratch/lindner/precice/src/mapping/PetRadialBasisFctMapping.hpp > > > > Thanks, > > Florian > From bsmith at mcs.anl.gov Fri Nov 7 07:40:48 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 7 Nov 2014 07:40:48 -0600 Subject: [petsc-users] Speed up KSPSolve of dense matrix In-Reply-To: <1712746.eHjgV37Em5@asaru> References: <5550582.ftmzpoOxMT@asaru> <1498803.iTGvHRNR5e@asaru> <365491BE-1733-4AEA-BF66-B4E9DDE7002F@mcs.anl.gov> <1712746.eHjgV37Em5@asaru> Message-ID: <5659F8C7-9F99-471B-AA80-D171BFAA9EBC@mcs.anl.gov> > On Nov 7, 2014, at 7:03 AM, Florian Lindner wrote: > > Hey, > > I've modified my code that it counts the total number of non-zeros (trivial to put in the preallocation code) and decides upon that wheather to create a MATSBAIJ or MATDENSE matrix. > > Using sparse matrices (for one that is really dense) I've made good experiences with -pc_factor_shift_type NONZERO -pc_type cholesky -ksp_type preonly. But when I use that on a MATDENSE I again get a a Zero pivot in LU factorization though using pc_factor_shift_type. If you use Cholesky for sbaij why are you using LU for dense? Why not Cholesky? The pc_factor_shift_type stuff is not supported for dense. Is your matrix singular? > > I suspect that is related to the MATDENSE matrix intended to be symmetric, but actually is not. I have MAT_SYMMETRY_ETERNAL set but when I view the matrix only the upper right triangular is set, how I did in my matrix assembly algorithm. Do the algorithm treat the matrix as symmetric when MAT_SYMMETRY_ETERNAL is set or is just a little hint? Any way to actually set a MATDENSE matrix to be symmetric or do I need to copy the entries to the lower triangular? If you want to use LU or MatMult with the dense matrix then you need to set all the values in the matrix, you cannot set just 1/2 of them. Even if the matrix is symmetric. Note you can set all the values in the SBAIJ matrix and just tell it to ignore the lower triangular entries you pass in. To do this after you create the matrix and preallocate then call MatSetOption(mat,MAT_IGNORE_LOWER_TRIANGULAR,PETSC_TRUE); > > Thanks, > > Florian > > Am Mittwoch, 5. November 2014, 08:36:26 schrieb Barry Smith: >> >> So you have a dense, very ill-conditioned matrix. Seems to me you should just be using the dense LU/Cholesky factorization unless you know how to preconditioner these types of matrices. http://scicomp.stackexchange.com/questions/7561/do-rbf-kernel-matrices-tend-to-be-ill-conditioned >> >> You can?t expect to take monstrously ill-conditioned matrices and expect to use iterative methods unless you really really really understand the source and and control of the ill-conditioning. >> >> What is your end goal in trying to use iterative methods? >> >> Barry >> >> >>> On Nov 5, 2014, at 3:41 AM, Florian Lindner wrote: >>> >>> Am Dienstag, 4. November 2014, 10:08:27 schrieb Barry Smith: >>>> >>>> There are a lot of questions here. >>> >>> Yes, thanks for replying! >>> >>>>> On Nov 4, 2014, at 328 AM, Florian Lindner wrote: >>>>> >>>>> Hello, >>>>> >>>>> I have a fulll matrix of size e.g. 603x603 of which I'm very disappointed with the runtime, compared to a naive LU / forward / backward solution. My petsc solution takes about 14s, while the old one takes just 0.5s. (when you're looking at sparse matrices the figures are almost inversed). I try to investigate what's the problem. >>>> >>>> For small problems like this direct dense LU could easily be faster than iterative schemes. Especially if you have many right hand sides with the same matrix since the factorization only needs to be done one. >>>> >>>>> >>>>> Most of the time is spent in the KSPSolve. >>>>> >>>>> Event Count Time (sec) Flops --- Global --- --- Stage --- Total >>>>> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>>> ------------------------------------------------------------------------------------------------------------------------ >>>>> >>>>> --- Event Stage 0: Main Stage >>>>> >>>>> MatMult 10334 1.0 6.3801e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 45 49 0 0 0 45 49 0 0 0 1177 >>>>> MatSolve 10334 1.0 6.9187e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 49 49 0 0 0 49 49 0 0 0 1085 >>>> >>>> If this is for a single solver for a 603 by 603 matrix then this is absurd. 10000 plus iterations >>> >>> But that's the output I got. When I run with -info I get a lot of messages: >>> >>> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged >>> >>> though doing just on KSPSolve / KSPSetOperators. Is that ok? >>> >>>>> MatCholFctrNum 1 1.0 1.8968e-01 1.0 6.03e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>>>> KSPGMRESOrthog 10000 1.0 2.1417e-01 1.0 3.73e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 2 0 0 0 2 2 0 0 0 1744 >>>>> KSPSolve 1 1.0 1.3763e+01 1.0 1.54e+10 1.0 0.0e+00 0.0e+00 0.0e+00 97100 0 0 0 97100 0 0 0 1120 >>>>> Prealloc Matrix C 1 1.0 5.7458e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> Filling Matrix C 1 1.0 6.7984e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> VecMDot 10000 1.0 8.8615e-02 1.0 1.87e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 2106 >>>>> VecMAXPY 10334 1.0 1.2585e-01 1.0 1.99e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1580 >>>>> Prealloc Matrix A 1 1.0 1.1071e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>>>> Filling Matrix A 1 1.0 1.2574e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>>>> PCSetUp 1 1.0 1.9073e-01 1.0 6.03e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>>>> PCApply 10334 1.0 6.9233e+00 1.0 7.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 49 49 0 0 0 49 49 0 0 0 1085 >>>>> ------------------------------------------------------------------------------------------------------------------------ >>>>> >>>>> Removed all events with global %T == 0, but my own ones. >>>>> >>>>> Matrix type is sequential MATSBAIJ on PETSC_COMM_SELF. >>>>> >>>>> KSP is initalized once: >>>>> >>>>> KSPSetOperators(_solver, _matrixC.matrix, _matrixC.matrix); >>>>> KSPSetTolerances(_solver, _solverRtol, PETSC_DEFAULT, PETSC_DEFAULT, PETSC_DEFAULT); >>>>> KSPSetFromOptions(_solver); >>>>> >>>>> KSPSolve: >>>>> >>>>> ierr = KSPSolve(_solver, in.vector, p.vector); CHKERRV(ierr) >>>>> >>>>> KSPSolve may be executed multiple times with the same SetOperators acording to dimensionality of the data. Here it's just one. >>>>> >>>>> -ksp_view says: >>>>> >>>>> KSP Object: 1 MPI processes >>>>> type: gmres >>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >>>>> GMRES: happy breakdown tolerance 1e-30 >>>>> maximum iterations=10000, initial guess is zero >>>>> tolerances: relative=1e-09, absolute=1e-50, divergence=10000 >>>>> left preconditioning >>>>> using PRECONDITIONED norm type for convergence test >>>>> PC Object: 1 MPI processes >>>>> type: icc >>>>> 0 levels of fill >>>>> tolerance for zero pivot 2.22045e-14 >>>>> using Manteuffel shift [POSITIVE_DEFINITE] >>>>> matrix ordering: natural >>>>> factor fill ratio given 1, needed 1 >>>>> Factored matrix follows: >>>>> Mat Object: 1 MPI processes >>>>> type: seqsbaij >>>>> rows=603, cols=603 >>>>> package used to perform factorization: petsc >>>>> total: nonzeros=182103, allocated nonzeros=182103 >>>>> total number of mallocs used during MatSetValues calls =0 >>>>> block size is 1 >>>>> linear system matrix = precond matrix: >>>>> Mat Object: C 1 MPI processes >>>>> type: seqsbaij >>>>> rows=603, cols=603 >>>>> total: nonzeros=182103, allocated nonzeros=182103 >>>>> total number of mallocs used during MatSetValues calls =0 >>>>> block size is 1 >>>>> >>>>> >>>>> I've tried to use a direct solver like suggested on pp 72, but: >>>>> >>>>> ./petBench 600 1 -ksp_type preonly -pc_type lu >>>> >>>> You cannot use LU with SBAIJ format. Only Cholesky. So use -pc_type cholesky >>> >>> I tried that too, it gives me an error message: >>> >>> % ./petBench 100 1 -pc_type cholesky >>> Mesh of size: 100 >>> Do mappings: 1 >>> KSP Init >>> Do KSPSolve consistent >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot >>> [0]PETSC ERROR: Zero pivot row 0 value 0 tolerance 2.22045e-14 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown >>> [0]PETSC ERROR: ./petBench on a arch-linux2-c-debug named helium by lindnefn Wed Nov 5 10:22:07 2014 >>> [0]PETSC ERROR: Configure options --with-debugging=1 >>> [0]PETSC ERROR: #1 MatPivotCheck_none() line 622 in /data2/scratch/lindner/petsc/include/petsc-private/matimpl.h >>> [0]PETSC ERROR: #2 MatPivotCheck() line 641 in /data2/scratch/lindner/petsc/include/petsc-private/matimpl.h >>> [0]PETSC ERROR: #3 MatCholeskyFactorNumeric_SeqSBAIJ_1_NaturalOrdering() line 1435 in /data2/scratch/lindner/petsc/src/mat/impls/sbaij/seq/sbaijfact.c >>> [0]PETSC ERROR: #4 MatCholeskyFactorNumeric() line 3063 in /data2/scratch/lindner/petsc/src/mat/interface/matrix.c >>> [0]PETSC ERROR: #5 PCSetUp_Cholesky() line 150 in /data2/scratch/lindner/petsc/src/ksp/pc/impls/factor/cholesky/cholesky.c >>> [0]PETSC ERROR: #6 PCSetUp() line 902 in /data2/scratch/lindner/petsc/src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: #7 KSPSetUp() line 305 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: #8 KSPSolve() line 417 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: #9 map() line 401 in /data2/scratch/lindner/precice/src/mapping/PetRadialBasisFctMapping.hpp >>> >>> FAQ says I likely have an error in my matrix generation when it occurs in the zeroth row. But for RBF the main diagonal is always BasisFunction( | x_n - x_n | ). Since I test with ThinPlateSplines as basis function here it's TPS(0) == 0. >>> >>>>> Mesh of size: 600 >>>>> Do mappings: 1 >>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>> [0]PETSC ERROR: No support for this operation for this object type >>>>> [0]PETSC ERROR: Factor type not supported >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown >>>>> [0]PETSC ERROR: ./petBench on a arch-linux2-c-opt named helium by lindnefn Tue Nov 4 10:19:28 2014 >>>>> [0]PETSC ERROR: Configure options --with-debugging=0 >>>>> [0]PETSC ERROR: #1 MatGetFactor_seqsbaij_petsc() line 1833 in /data2/scratch/lindner/petsc/src/mat/impls/sbaij/seq/sbaij.c >>>>> [0]PETSC ERROR: #2 MatGetFactor() line 3963 in /data2/scratch/lindner/petsc/src/mat/interface/matrix.c >>>>> [0]PETSC ERROR: #3 PCSetUp_LU() line 125 in /data2/scratch/lindner/petsc/src/ksp/pc/impls/factor/lu/lu.c >>>>> [0]PETSC ERROR: #4 PCSetUp() line 902 in /data2/scratch/lindner/petsc/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: #5 KSPSetUp() line 305 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #6 KSPSolve() line 417 in /data2/scratch/lindner/petsc/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: #7 map() line 391 in /data2/scratch/lindner/precice/src/mapping/PetRadialBasisFctMapping.hpp >>> >>> Thanks, >>> Florian >> From davydden at gmail.com Fri Nov 7 08:10:06 2014 From: davydden at gmail.com (Denis Davydov) Date: Fri, 7 Nov 2014 15:10:06 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP Message-ID: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> Dear all, I was trying to set the problem type to EPS_GHEP as the matrices in the considered generalised eigenvalue problem are real and symmetric. However, this breaks the solver's convergence as compared to the default, non-symmetric case. The residual is quite small (~1e-9). I am puzzled why this happens. The output with -eps_view -esp_monitor_all flags is below. Thank you in advance for any advices on the issue. Kind regards, Denis === EPS Object: 1 MPI processes type: krylovschur Krylov-Schur: 50% of basis vectors kept after restart problem type: generalized symmetric eigenvalue problem selected portion of the spectrum: smallest real parts number of eigenvalues (nev): 5 number of column vectors (ncv): 20 maximum dimension of projected problem (mpd): 20 maximum number of iterations: 1089 tolerance: 1e-09 convergence test: absolute BV Object: 1 MPI processes type: svec 21 columns of global length 1089 orthogonalization method: classical Gram-Schmidt orthogonalization refinement: if needed (eta: 0.7071) non-standard inner product Mat Object: 1 MPI processes type: seqaij rows=1089, cols=1089 total: nonzeros=8409, allocated nonzeros=20691 total number of mallocs used during MatSetValues calls =0 not using I-node routines DS Object: 1 MPI processes type: hep solving the problem with: Implicit QR method (_steqr) ST Object: 1 MPI processes type: shift shift: 0 number of matrices: 2 all matrices have different nonzero pattern KSP Object: (st_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (st_) 1 MPI processes type: redundant Redundant preconditioner: First (color=0) of 1 PCs follows KSP Object: (st_redundant_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (st_redundant_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5, needed 4.41539 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=1089, cols=1089 package used to perform factorization: petsc total: nonzeros=37129, allocated nonzeros=37129 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=1089, cols=1089 total: nonzeros=8409, allocated nonzeros=20691 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=1089, cols=1089 total: nonzeros=8409, allocated nonzeros=20691 total number of mallocs used during MatSetValues calls =0 not using I-node routines From jroman at dsic.upv.es Fri Nov 7 08:14:56 2014 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 7 Nov 2014 15:14:56 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> Message-ID: <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> El 07/11/2014, a las 15:10, Denis Davydov escribi?: > Dear all, > > I was trying to set the problem type to EPS_GHEP as > the matrices in the considered generalised eigenvalue problem are real and symmetric. > However, this breaks the solver's convergence as compared to the default, non-symmetric case. > The residual is quite small (~1e-9). I am puzzled why this happens. > > The output with -eps_view -esp_monitor_all flags is below. > > Thank you in advance for any advices on the issue. > > Kind regards, > Denis > What do you mean by 'breaks convergence'? Are you sure your matrices are symmetric? Jose From davydden at gmail.com Fri Nov 7 08:19:27 2014 From: davydden at gmail.com (Denis Davydov) Date: Fri, 7 Nov 2014 15:19:27 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> Message-ID: <2DC016E3-DBC4-4FE6-8D15-FA6398A9336E@gmail.com> > What do you mean by 'breaks convergence'? Are you sure your matrices are symmetric? > Jose I mean the solver does not converge. Whereas if I do not specify that the matrices are symmetric and generalised non-symmetric eigenvalue problem is considered, everything is all right. However, i am sure the matrices are symmetric. I double check it with MatIsSymmetric. Regards, Denis. From jroman at dsic.upv.es Fri Nov 7 08:41:31 2014 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 7 Nov 2014 15:41:31 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: <2DC016E3-DBC4-4FE6-8D15-FA6398A9336E@gmail.com> References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> <2DC016E3-DBC4-4FE6-8D15-FA6398A9336E@gmail.com> Message-ID: El 07/11/2014, a las 15:19, Denis Davydov escribi?: > >> What do you mean by 'breaks convergence'? Are you sure your matrices are symmetric? >> Jose > > I mean the solver does not converge. > Whereas if I do not specify that the matrices are symmetric and > generalised non-symmetric eigenvalue problem is considered, everything is all right. > However, i am sure the matrices are symmetric. > I double check it with MatIsSymmetric. > > Regards, > Denis. > When solving as a symmetric problem, the solver uses B-inner products, so lengths are measured differently. Does your B-matrix have a large norm? Why are you using the absolute convergence criterion? Try with -eps_conv_eig or -eps_conv_norm If this does not solve the problem, send me a small matrix that I can try. Jose From davydden at gmail.com Fri Nov 7 08:45:09 2014 From: davydden at gmail.com (Denis Davydov) Date: Fri, 7 Nov 2014 15:45:09 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> Message-ID: <98D02121-2E62-47F8-8841-F9D89D1B87DE@gmail.com> > What do you mean by 'breaks convergence'? Are you sure your matrices are symmetric? > Jose I shall clarify that the calculated by EPSComputeResidualNorm residual norm is bigger than the required tolerance, set by EPSSetTolerances. In light of your latest answer about B-norms, supposedly this is a wrong approach. Sorry for being un-precise initially. Regards, Denis. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Fri Nov 7 09:03:31 2014 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 7 Nov 2014 16:03:31 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: <98D02121-2E62-47F8-8841-F9D89D1B87DE@gmail.com> References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> <98D02121-2E62-47F8-8841-F9D89D1B87DE@gmail.com> Message-ID: <396D16B5-D8C1-4180-A8B5-BC1F65A9C9D5@dsic.upv.es> El 07/11/2014, a las 15:45, Denis Davydov escribi?: > >> What do you mean by 'breaks convergence'? Are you sure your matrices are symmetric? >> Jose > > I shall clarify that the calculated by EPSComputeResidualNorm residual norm > is bigger than the required tolerance, set by EPSSetTolerances. > > In light of your latest answer about B-norms, > supposedly this is a wrong approach. > > Sorry for being un-precise initially. > > Regards, > Denis. Yes, EPSComputeResidualNorm uses the 2-norm. Maybe it would be better if the stopping criterion in symmetric problems would be more similar to the non-symmetric case. I will think about it. It would be good if you could send me sample matrices (to my personal email). Jose From davydden at gmail.com Fri Nov 7 09:05:35 2014 From: davydden at gmail.com (Denis Davydov) Date: Fri, 7 Nov 2014 16:05:35 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: <396D16B5-D8C1-4180-A8B5-BC1F65A9C9D5@dsic.upv.es> References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> <98D02121-2E62-47F8-8841-F9D89D1B87DE@gmail.com> <396D16B5-D8C1-4180-A8B5-BC1F65A9C9D5@dsic.upv.es> Message-ID: <5118D9F0-8399-4137-8013-D54F2BC0A20F@gmail.com> > Yes, EPSComputeResidualNorm uses the 2-norm. it is clear now, thanks for your prompt answers. > Maybe it would be better if the stopping criterion in symmetric problems would be more similar to the non-symmetric case. I will think about it. It would be good if you could send me sample matrices (to my personal email). will do. Regards, Denis. From davydden at gmail.com Fri Nov 7 10:31:40 2014 From: davydden at gmail.com (Denis Davydov) Date: Fri, 7 Nov 2014 17:31:40 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: <396D16B5-D8C1-4180-A8B5-BC1F65A9C9D5@dsic.upv.es> References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> <98D02121-2E62-47F8-8841-F9D89D1B87DE@gmail.com> <396D16B5-D8C1-4180-A8B5-BC1F65A9C9D5@dsic.upv.es> Message-ID: <386CA68B-2577-462C-A16D-F0A1F4F445B0@gmail.com> > Yes, EPSComputeResidualNorm uses the 2-norm. One last thing, if I force EPSSetTrueResidual(eps, PETSC_TRUE) will that guarantee that EPSComputeRelativeError() will give the norm consistent to that, used internally by all SLEPc solvers? Thanks. Regards, Denis. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Nov 7 10:36:57 2014 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 07 Nov 2014 17:36:57 +0100 Subject: [petsc-users] [SLEPc] Get convergence metrics In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Fri Nov 7 10:57:54 2014 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 7 Nov 2014 17:57:54 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: <386CA68B-2577-462C-A16D-F0A1F4F445B0@gmail.com> References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> <98D02121-2E62-47F8-8841-F9D89D1B87DE@gmail.com> <396D16B5-D8C1-4180-A8B5-BC1F65A9C9D5@dsic.upv.es> <386CA68B-2577-462C-A16D-F0A1F4F445B0@gmail.com> Message-ID: <3BE6DE10-465D-4D53-8D81-207C426F5502@dsic.upv.es> El 07/11/2014, a las 17:31, Denis Davydov escribi?: > >> Yes, EPSComputeResidualNorm uses the 2-norm. > > One last thing, if I force EPSSetTrueResidual(eps, PETSC_TRUE) > will that guarantee that EPSComputeRelativeError() > will give the norm consistent to that, used internally by all SLEPc solvers? > > Thanks. > Regards, > Denis. I would not recommend that, since it is not implemented in all solvers. I tried with your matrices. In your case, it is much better to compute eigenvalues close to the origin with shift-and-invert, rather than computing smallest_real eigenvalues. That is, replace -eps_smallest_real with -eps_target 0 -st_type sinvert Jose From davydden at gmail.com Fri Nov 7 11:33:06 2014 From: davydden at gmail.com (Denis Davydov) Date: Fri, 7 Nov 2014 18:33:06 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: <3BE6DE10-465D-4D53-8D81-207C426F5502@dsic.upv.es> References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> <98D02121-2E62-47F8-8841-F9D89D1B87DE@gmail.com> <396D16B5-D8C1-4180-A8B5-BC1F65A9C9D5@dsic.upv.es> <386CA68B-2577-462C-A16D-F0A1F4F445B0@gmail.com> <3BE6DE10-465D-4D53-8D81-207C426F5502@dsic.upv.es> Message-ID: > > I would not recommend that, since it is not implemented in all solvers. I see, thanks. Regards, Denis. From miguelarriagaecunha at gmail.com Fri Nov 7 16:39:19 2014 From: miguelarriagaecunha at gmail.com (Miguel Arriaga) Date: Fri, 7 Nov 2014 17:39:19 -0500 Subject: [petsc-users] Using a Matrix Shell with SLEPc Message-ID: Hi there, I have a matrix shell where the Mult operation looks like this: y=1/2(K+K*) x with K=M^-1 A and K* its transpose. 1- For setting-up the operation MatMult, since I need to do a KSPSolve on M, should I set-up the KSP on the operation MATOP_CREATE and destroy it on MATOP_DESTROY and store it as a global variable? 2- For "inverting" M, considering that M is symmetric, Should I (a) use LU, or should I (b) use an iterative procedure like CG with ILU, since M is symmetric positive definite? Is there a faster iterative method for a SPD matrix? 3- I want to use this matrix shell in SLEPc to compute the largest positive eigenvalue, which will be smaller in magnitude than the smallest negative real (possibly very close to zero). Since it is a matrix shell I assume that I can't really use Shift-and-invert without choosing an iterative procedure that only requires MatMult. What would be the best Method+Preconditioner for this case? Note that this is a Real Symmetric matrix (but not Positive definite). Should I set all of these as true? MATOP_IS_HERMITIAN MATOP_IS_STRUCTURALLY_SYMMETRIC MATOP_IS_SYMMETRIC Thank you, Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Nov 7 16:56:08 2014 From: jed at jedbrown.org (Jed Brown) Date: Sat, 08 Nov 2014 01:56:08 +0300 Subject: [petsc-users] Using a Matrix Shell with SLEPc In-Reply-To: References: Message-ID: <87vbmq4o3b.fsf@jedbrown.org> Miguel Arriaga writes: > Hi there, > I have a matrix shell where the Mult operation looks like this: > y=1/2(K+K*) x > with K=M^-1 A and K* its transpose. > > 1- For setting-up the operation MatMult, since I need to do a KSPSolve on > M, should I set-up the KSP on the operation MATOP_CREATE and destroy it on > MATOP_DESTROY and store it as a global variable? Store it in the context. That's why callbacks always have a context. Global variables are terrible. > 2- For "inverting" M, considering that M is symmetric, Should I (a) use LU, > or should I (b) use an iterative procedure like CG with ILU, since M is > symmetric positive definite? Is there a faster iterative method for a SPD > matrix? CG/Jacobi is usually fine for the mass matrix. > 3- I want to use this matrix shell in SLEPc to compute the largest positive > eigenvalue, which will be smaller in magnitude than the smallest negative > real (possibly very close to zero). Since it is a matrix shell I assume > that I can't really use Shift-and-invert without choosing an iterative > procedure that only requires MatMult. Start with a simple shift (not shift-and-invert). > What would be the best Method+Preconditioner for this case? Note that > this is a Real Symmetric matrix (but not Positive definite). Should I > set all of these as true? > > MATOP_IS_HERMITIAN > MATOP_IS_STRUCTURALLY_SYMMETRIC > MATOP_IS_SYMMETRIC > > Thank you, > Miguel -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From miguelarriagaecunha at gmail.com Fri Nov 7 18:45:11 2014 From: miguelarriagaecunha at gmail.com (Miguel Arriaga) Date: Fri, 7 Nov 2014 19:45:11 -0500 Subject: [petsc-users] Using a Matrix Shell with SLEPc In-Reply-To: <87vbmq4o3b.fsf@jedbrown.org> References: <87vbmq4o3b.fsf@jedbrown.org> Message-ID: Hi, Thank you so much for your reply. 1 One note that I forgot to mention is that I'm using FORTRAN (sorry!). Is it possible to set up the context to pass M and the ksp for M? I also try to avoid common blocks but in this case it seemed inevitable. Since M and A are already global in the code that I'm using maybe I could just use the kspM in the context. The other side of my question was more on whether doing MatCreateShell(...) and then MatShellSetOperation(...,MATOP_CREATE,...) would work because it seems like a chicken-vs-egg kind of situation. Is the CREATE operation executed after this? I couldn't find an example that uses this operation and if I google MATOP_CREATE the only thing I get is the source code of the file petscmat.h. 2 Thanks! I'm sorry if this was already included in your answer but I just wanted to make sure. For a moderately large sparse problem (100K equations) considering that I'm using two KSPSolve in each call to MatMult, and I'm going to use this Shell in an eigensolver (that will probably call MatMult several times), wouldn't it end up being faster to just do a Sparse Direct to get the LU decomposition of M and then just do the forward and back substitution? 3 My colleague suggested that I do a first run without shift for EPS_LARGEST_MAGNITUDE and then, if this value is negative, I would use the eigenvalue as a shift and compute EPS_LARGEST_MAGNITUDE again. This seems like a fine solution to me but I was wondering if just doing EPS_LARGEST_REAL without shift would already do something like this but more efficiently since, for example, you can detect very soon if your eigenvalue is converging to a negative number, allowing with just a few iterations to get the magnitude of the necessary shift. Should I just add a very large shift to begin with? Or is EPS_LARGEST_REAL fine? Thank for your time, Miguel On Fri, Nov 7, 2014 at 5:56 PM, Jed Brown wrote: > Miguel Arriaga writes: > > > Hi there, > > I have a matrix shell where the Mult operation looks like this: > > y=1/2(K+K*) x > > with K=M^-1 A and K* its transpose. > > > > 1- For setting-up the operation MatMult, since I need to do a KSPSolve on > > M, should I set-up the KSP on the operation MATOP_CREATE and destroy it > on > > MATOP_DESTROY and store it as a global variable? > > Store it in the context. That's why callbacks always have a context. > Global variables are terrible. > > > 2- For "inverting" M, considering that M is symmetric, Should I (a) use > LU, > > or should I (b) use an iterative procedure like CG with ILU, since M is > > symmetric positive definite? Is there a faster iterative method for a SPD > > matrix? > > CG/Jacobi is usually fine for the mass matrix. > > > 3- I want to use this matrix shell in SLEPc to compute the largest > positive > > eigenvalue, which will be smaller in magnitude than the smallest negative > > real (possibly very close to zero). Since it is a matrix shell I assume > > that I can't really use Shift-and-invert without choosing an iterative > > procedure that only requires MatMult. > > Start with a simple shift (not shift-and-invert). > > > What would be the best Method+Preconditioner for this case? Note that > > this is a Real Symmetric matrix (but not Positive definite). Should I > > set all of these as true? > > > > MATOP_IS_HERMITIAN > > MATOP_IS_STRUCTURALLY_SYMMETRIC > > MATOP_IS_SYMMETRIC > > > > Thank you, > > Miguel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Nov 7 19:04:33 2014 From: jed at jedbrown.org (Jed Brown) Date: Sat, 08 Nov 2014 04:04:33 +0300 Subject: [petsc-users] Using a Matrix Shell with SLEPc In-Reply-To: References: <87vbmq4o3b.fsf@jedbrown.org> Message-ID: <8761eq4i5a.fsf@jedbrown.org> Miguel Arriaga writes: > Hi, > Thank you so much for your reply. > > 1 > One note that I forgot to mention is that I'm using FORTRAN (sorry!). > Is it possible to set up the context to pass M and the ksp for M? I also > try to avoid common blocks but in this case it seemed inevitable. Since M > and A are already global in the code that I'm using maybe I could just use > the kspM in the context. Fortran 90 has "derived types" that you can use for this. If you insist on using Fortran 77, bad software design is inevitable (you can use magic arrays or globals). > The other side of my question was more on whether doing MatCreateShell(...) > and then MatShellSetOperation(...,MATOP_CREATE,...) would work because it > seems like a chicken-vs-egg kind of situation. Is the CREATE operation > executed after this? I couldn't find an example that uses this operation > and if I google MATOP_CREATE the only thing I get is the source code of the > file petscmat.h. MATOP_CREATE was removed (it existed for something different anyway and is unused now). It should have been deleted so that people like you wouldn't stumble upon it. I recommend doing that setup up-front and set the context when you set MATOP_MULT. > 2 > Thanks! > I'm sorry if this was already included in your answer but I just wanted to > make sure. For a moderately large sparse problem (100K equations) > considering that I'm using two KSPSolve in each call to MatMult, and I'm > going to use this Shell in an eigensolver (that will probably call MatMult > several times), wouldn't it end up being faster to just do a Sparse Direct > to get the LU decomposition of M and then just do the forward and back > substitution? M is pretty well conditioned and direct solves are can be mighty expensive in 3D. So try it. For a 2D 5-point stencil, direct might win. For a 3D 13-point stencil (second neighbors, a class of fourth order methods), iterative is probably vastly better. > 3 > My colleague suggested that I do a first run without shift for > EPS_LARGEST_MAGNITUDE and then, if this value is negative, I would use the > eigenvalue as a shift and compute EPS_LARGEST_MAGNITUDE again. This seems > like a fine solution to me but I was wondering if just doing > EPS_LARGEST_REAL without shift would already do something like this but > more efficiently since, for example, you can detect very soon if your > eigenvalue is converging to a negative number, allowing with just a few > iterations to get the magnitude of the necessary shift. Should I just add a > very large shift to begin with? Or is EPS_LARGEST_REAL fine? Start simple, EPS_LARGEST_REAL. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jychang48 at gmail.com Sun Nov 9 04:10:00 2014 From: jychang48 at gmail.com (Justin Chang) Date: Sun, 9 Nov 2014 04:10:00 -0600 Subject: [petsc-users] Local/global numbering to natural numbering? Message-ID: Hi all, Given the local/global numbering of vector/array entries generated after DMPlexDistribute(), is it possible to obtain the "natural numbering"? That is, the numbering of the vertex points before the mesh was distributed. I see that this has been discussed thoroughly for DMDA's but is it possible to obtain some sort of natural numbering for DMPlex? For instance, when I call DMPlexCreateFromDAG() or DMPlexCreateFromCellList() I would consider vertexCoords[] to be of "natural numbering" since it's the original ordering specified by the mesh generator. Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Sun Nov 9 04:33:42 2014 From: jychang48 at gmail.com (Justin Chang) Date: Sun, 9 Nov 2014 04:33:42 -0600 Subject: [petsc-users] Local/global numbering to natural numbering? In-Reply-To: References: Message-ID: Or would PetscSF sort of give you that information? Say if I had auxiliary data associated for each vertex point, I would first distribute the mesh via DMPlexDistribute, obtain the PetscSF from that routine, and then call DMPlexDistributeField for the vector containing said auxiliary data? Thanks, Justin On Sun, Nov 9, 2014 at 4:10 AM, Justin Chang wrote: > Hi all, > > Given the local/global numbering of vector/array entries generated after > DMPlexDistribute(), is it possible to obtain the "natural numbering"? That > is, the numbering of the vertex points before the mesh was distributed. I > see that this has been discussed thoroughly for DMDA's but is it possible > to obtain some sort of natural numbering for DMPlex? > > For instance, when I call DMPlexCreateFromDAG() or > DMPlexCreateFromCellList() I would consider vertexCoords[] to be of > "natural numbering" since it's the original ordering specified by the mesh > generator. > > Thanks, > Justin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gpentafragkas at gmail.com Sun Nov 9 20:24:22 2014 From: gpentafragkas at gmail.com (Giorgos Pentafragkas) Date: Mon, 10 Nov 2014 04:24:22 +0200 Subject: [petsc-users] GPU support: ODE's Message-ID: Hi guys, quick question: The GPU support page says that "algebraic solvers now run on GPU". I'm interested in ODE solvers. Can someone clarify if they are supported as well, or is it just the algebraic ones? (DAE's I presume, which could be used for implicit ODE's) Many thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Nov 9 20:34:30 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 9 Nov 2014 20:34:30 -0600 Subject: [petsc-users] GPU support: ODE's In-Reply-To: References: Message-ID: On Sun, Nov 9, 2014 at 8:24 PM, Giorgos Pentafragkas < gpentafragkas at gmail.com> wrote: > Hi guys, quick question: > > The GPU support page says that "algebraic solvers now run on GPU". > I'm interested in ODE solvers. > Can someone clarify if they are supported as well, or is it just the > algebraic ones? (DAE's I presume, which could be used for implicit ODE's) > The linear algebra making up the ODE solvers can run on the GPU, but all logic is on the CPU, and we have not fused kernels for these operations. Kernel fusion is important for GPUs, but (I think) there is no clear path for its implementation in libraries. Thus, you can run it. Your mileage may vary. I would be really surprised to see it beat a modern CPU. Thanks, Matt > Many thanks! > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From filippo.leonardi at sam.math.ethz.ch Mon Nov 10 08:18:40 2014 From: filippo.leonardi at sam.math.ethz.ch (Filippo Leonardi) Date: Mon, 10 Nov 2014 15:18:40 +0100 Subject: [petsc-users] Multiple solves with PCMG fail In-Reply-To: References: <9621705.dtimTkejkQ@besikovitch-ii> <58968237.1VxD0XNj1Y@besikovitch-iii> Message-ID: <7794796.iyiicWNzpQ@besikovitch-iii> Super late update, sorry. Not sure if I understood you correctly: if(n) { PetscValidScalarPointer(r,3); ierr = PetscPrintf(PETSC_COMM_WORLD, " ---> n = %d nonzero if.\n", n);CHKERRQ(ierr); } else { ierr = PetscPrintf(PETSC_COMM_WORLD, " ---> n = 0 else.\n");CHKERRQ(ierr); } I get (btw this is also the case with a single process): ./test1 -pc_type mg -pc_mg_levels 3 ---> n = 0 else. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Pointer: Parameter # 4 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5.2-925-g233c798 GIT Date: 2014-11-03 17:44:15 -0700 [0]PETSC ERROR: ./test1 on a arch-linux2-cxx-opt named Besikovitch-III by filippo Mon Nov 10 15:15:47 2014 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-cxx-opt [0]PETSC ERROR: #1 KSPComputeEigenvalues() line 125 in /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #2 KSPChebyshevComputeExtremeEigenvalues_Private() line 328 in /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/impls/cheby/cheby.c [0]PETSC ERROR: #3 KSPSolve_Chebyshev() line 373 in /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/impls/cheby/cheby.c [0]PETSC ERROR: #4 KSPSolve() line 465 in /home/filippo/Workspace/petsc- git/ksp-zero-eig/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #5 PCMGMCycle_Private() line 19 in /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #6 PCApply_MG() line 337 in /home/filippo/Workspace/petsc- git/ksp-zero-eig/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #7 PCApply() line 440 in /home/filippo/Workspace/petsc- git/ksp-zero-eig/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #8 KSP_PCApply() line 230 in /home/filippo/Workspace/petsc- git/ksp-zero-eig/include/petsc-private/kspimpl.h [0]PETSC ERROR: #9 KSPInitialResidual() line 63 in /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: #10 KSPSolve_GMRES() line 234 in /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #11 KSPSolve() line 465 in /home/filippo/Workspace/petsc- git/ksp-zero-eig/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #12 main() line 149 in /home/filippo/Workspace/C++/MLMCTest/test1.cpp [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- On Thursday 06 November 2014 11.27:59 you wrote: > if (n) PetscValidScalarPointer(r,3); > and put a print statement. n==0 in your case when it fails. -------------- next part -------------- A non-text attachment was scrubbed... Name: Filippo Leonardi.vcf Type: text/vcard Size: 322 bytes Desc: not available URL: From mfadams at lbl.gov Mon Nov 10 08:26:45 2014 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 10 Nov 2014 09:26:45 -0500 Subject: [petsc-users] Multiple solves with PCMG fail In-Reply-To: <7794796.iyiicWNzpQ@besikovitch-iii> References: <9621705.dtimTkejkQ@besikovitch-ii> <58968237.1VxD0XNj1Y@besikovitch-iii> <7794796.iyiicWNzpQ@besikovitch-iii> Message-ID: It does not look like you are in fact using the branch with this fix. The branch code looks like: if (n) PetscValidScalarPointer(r,3); *and* if (n) PetscValidScalarPointer(r,4); So if you now put the check for argument 4 inside of your if statement then you will have reimplemented my branch, or at least most of it, and I think it might work or at least not give these errors. I don't think we ever fixed the issue that Cheby does will not have an eigen estimate now and so you will probably see terrible performance. I'm not sure if came up with a fix for this. On Mon, Nov 10, 2014 at 9:18 AM, Filippo Leonardi < filippo.leonardi at sam.math.ethz.ch> wrote: > Super late update, sorry. > > Not sure > if I understood you correctly: > if(n) { > PetscValidScalarPointer(r,3); > ierr = PetscPrintf(PETSC_COMM_WORLD, " ---> n = %d nonzero if.\n", > n);CHKERRQ(ierr); > } else { > ierr = PetscPrintf(PETSC_COMM_WORLD, " ---> n = 0 > else.\n");CHKERRQ(ierr); > } > > I get (btw this is also the case with a single process): > ./test1 -pc_type mg -pc_mg_levels 3 > > ---> n = 0 else. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Null Pointer: Parameter # 4 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for > trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.5.2-925-g233c798 GIT > Date: > 2014-11-03 17:44:15 -0700 > [0]PETSC ERROR: ./test1 on a arch-linux2-cxx-opt named Besikovitch-III by > filippo Mon Nov 10 15:15:47 2014 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-cxx-opt > [0]PETSC ERROR: #1 KSPComputeEigenvalues() line 125 in > > /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #2 KSPChebyshevComputeExtremeEigenvalues_Private() line > 328 in > > /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/impls/cheby/cheby.c > [0]PETSC ERROR: #3 KSPSolve_Chebyshev() line 373 in > > /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/impls/cheby/cheby.c > [0]PETSC ERROR: #4 KSPSolve() line 465 in /home/filippo/Workspace/petsc- > git/ksp-zero-eig/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #5 PCMGMCycle_Private() line 19 in > /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #6 PCApply_MG() line 337 in /home/filippo/Workspace/petsc- > git/ksp-zero-eig/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #7 PCApply() line 440 in /home/filippo/Workspace/petsc- > git/ksp-zero-eig/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #8 KSP_PCApply() line 230 in /home/filippo/Workspace/petsc- > git/ksp-zero-eig/include/petsc-private/kspimpl.h > [0]PETSC ERROR: #9 KSPInitialResidual() line 63 in > > /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #10 KSPSolve_GMRES() line 234 in > > /home/filippo/Workspace/petsc-git/ksp-zero-eig/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #11 KSPSolve() line 465 in /home/filippo/Workspace/petsc- > git/ksp-zero-eig/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #12 main() line 149 in > /home/filippo/Workspace/C++/MLMCTest/test1.cpp > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error > message to petsc-maint at mcs.anl.gov---------- > > On Thursday 06 November 2014 11.27:59 you wrote: > > if (n) PetscValidScalarPointer(r,3); > > and put a print statement. n==0 in your case when it fails. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From filippo.leonardi at sam.math.ethz.ch Mon Nov 10 08:46:56 2014 From: filippo.leonardi at sam.math.ethz.ch (Filippo Leonardi) Date: Mon, 10 Nov 2014 15:46:56 +0100 Subject: [petsc-users] Multiple solves with PCMG fail In-Reply-To: References: <9621705.dtimTkejkQ@besikovitch-ii> <7794796.iyiicWNzpQ@besikovitch-iii> Message-ID: <2242449.GqaeaLRqZi@besikovitch-iii> Switched branch and now it works. No error and the solution is correct. I guess i not that more inefficient since the times when the rhs is zero is not that often, otherwise you wouldn't solve the system that way. I'm stupid: somehow I was in master and didn't notice that. Sorry for the confusion. On Monday 10 November 2014 09.26:45 you wrote: > It does not look like you are in fact using the branch with this fix. The > branch code looks like: -------------- next part -------------- A non-text attachment was scrubbed... Name: Filippo Leonardi.vcf Type: text/vcard Size: 322 bytes Desc: not available URL: From Vincent.De-Groof at uibk.ac.at Mon Nov 10 10:57:55 2014 From: Vincent.De-Groof at uibk.ac.at (De Groof, Vincent Frans Maria) Date: Mon, 10 Nov 2014 16:57:55 +0000 Subject: [petsc-users] MatSolve in CG Message-ID: <17A78B9D13564547AC894B88C1596747203B9455@XMBX4.uibk.ac.at> Dear all, I am writing a few variations on the PCG algorithm. Starting with my own PCG implementation, I noticed that my routine is quite a bit slower (~20%). I think I tracked it down to the MatSolve calls in PCApply. I am using BDDC as the preconditioner. In the Petsc PCG function, each PCApply calls the MatSolve 3 times. It also calls MatSolve once somewhere between calling KSPSolve and KSPSolve_CG (KSP_SetUp has been called before). So MatSolve is called 1+3*(# iterations) per KSPSolve. In my own PCG variation, the initial MatSolve is not called. But MatSolve is called 4 times in each iteration, so in total 4*(# iterations). Where does this MatSolve in between KSPSolve and KSPSolve_CG come from? And how can I make my function use less MatSolve's? I hope that I explained the problem clearly, and maybe this is a very trivial question. kind regards, Vincent -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Nov 10 15:48:27 2014 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 10 Nov 2014 15:48:27 -0600 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> Message-ID: On Fri, Nov 7, 2014 at 11:04 AM, Lawrence Mitchell < lawrence.mitchell at imperial.ac.uk> wrote: > Hi petsc-dev, > > I'm solving a pure Neumann mixed Poisson-like problem, preconditioning > with a schur complement. The pressure space has a nullspace of the > constant functions and so I attach the appropriate nullspace to the krylov > solver, and compose the constant nullspace with the IS defining the > pressure space. My RHS is consistent. > That is supposed to work, and I think it does in my tests. The code is here https://bitbucket.org/petsc/petsc/src/1f0d623c8336219eb98f7ded6f95c151ca603fe7/src/ksp/pc/impls/fieldsplit/fieldsplit.c?at=master#cl-562 so maybe we can track this down in the debugger. Thanks, Matt > When I precondition with: > > -pc_type fieldsplit -pc_fieldsplit_type schur > -pc_fieldsplit_schur_fact_type full \ > -pc_fieldsplit_schur_precondition selfp > > I notice that the nullspace is not transferred over to the preconditioning > matrix for S. Is this a deliberate choice? > > ksp_view output below, note that the schurcomplement mat has an attached > nullspace, but the generated pmat does not. > > Cheers, > > Lawrence > > KSP Object: 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=30, initial guess is zero > tolerances: relative=1e-07, absolute=1e-50, divergence=10000 > left preconditioning > has attached null space > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, factorization FULL > Preconditioner for the Schur complement formed from Sp, an assembled > approximation to S, which uses (the lumped) A00's diagonal's inverse > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (fieldsplit_0_) 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=72, cols=72 > package used to perform factorization: petsc > total: nonzeros=1080, allocated nonzeros=1080 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 23 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=72, cols=72 > total: nonzeros=1080, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 23 nodes, limit used is 5 > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (fieldsplit_1_) 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (fieldsplit_1_) 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=24, cols=24 > package used to perform factorization: petsc > total: nonzeros=216, allocated nonzeros=216 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 8 nodes, limit used is 5 > linear system matrix followed by preconditioner matrix: > Mat Object: (fieldsplit_1_) 1 MPI processes > type: schurcomplement > rows=24, cols=24 > has attached null space > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (fieldsplit_1_) 1 MPI > processes > type: seqaij > rows=24, cols=24 > total: nonzeros=72, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls =0 > has attached null space > using I-node routines: found 8 nodes, limit used is 5 > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=24, cols=72 > total: nonzeros=288, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 8 nodes, limit used is 5 > KSP of A00 > KSP Object: (fieldsplit_0_) 1 MPI > processes > type: gmres > GMRES: restart=30, using Classical (unmodified) > Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI > processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=72, cols=72 > package used to perform factorization: petsc > total: nonzeros=1080, allocated nonzeros=1080 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 23 nodes, limit > used is 5 > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) > 1 MPI processes > type: seqaij > rows=72, cols=72 > total: nonzeros=1080, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 23 nodes, limit used is 5 > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=72, cols=24 > total: nonzeros=288, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 23 nodes, limit used is 5 > Mat Object: 1 MPI processes > type: seqaij > rows=24, cols=24 > total: nonzeros=216, allocated nonzeros=216 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 8 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: nest > rows=96, cols=96 > has attached null space > Matrix object: > type=nest, rows=2, cols=2 > MatNest structure: > (0,0) : prefix="fieldsplit_0_", type=seqaij, rows=72, cols=72 > (0,1) : type=seqaij, rows=72, cols=24 > (1,0) : type=seqaij, rows=24, cols=72 > (1,1) : prefix="fieldsplit_1_", type=seqaij, rows=24, cols=24 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Mon Nov 10 17:34:38 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Mon, 10 Nov 2014 23:34:38 +0000 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> Message-ID: On 10 Nov 2014, at 21:48, Matthew Knepley wrote: > On Fri, Nov 7, 2014 at 11:04 AM, Lawrence Mitchell wrote: > Hi petsc-dev, > > I'm solving a pure Neumann mixed Poisson-like problem, preconditioning with a schur complement. The pressure space has a nullspace of the constant functions and so I attach the appropriate nullspace to the krylov solver, and compose the constant nullspace with the IS defining the pressure space. My RHS is consistent. > > That is supposed to work, and I think it does in my tests. The code is here > > https://bitbucket.org/petsc/petsc/src/1f0d623c8336219eb98f7ded6f95c151ca603fe7/src/ksp/pc/impls/fieldsplit/fieldsplit.c?at=master#cl-562 > That code attaches the nullspace attached to the IS to jac->pmat if appropriate. However, if I'm using -pc_fieldsplit_schur_precondition selfp then the schur complement is preconditioned with jac->schurp (rather than jac->pmat[1]) which is not setup until here https://bitbucket.org/petsc/petsc/src/1f0d623c8336219eb98f7ded6f95c151ca603fe7/src/ksp/pc/impls/fieldsplit/fieldsplit.c?at=master#cl-656 at which point the nullspace is not attached, no? Cheers, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From knepley at gmail.com Mon Nov 10 17:50:16 2014 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 10 Nov 2014 17:50:16 -0600 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> Message-ID: On Mon, Nov 10, 2014 at 5:34 PM, Lawrence Mitchell < lawrence.mitchell at imperial.ac.uk> wrote: > > On 10 Nov 2014, at 21:48, Matthew Knepley wrote: > > > On Fri, Nov 7, 2014 at 11:04 AM, Lawrence Mitchell < > lawrence.mitchell at imperial.ac.uk>wrote: > > Hi petsc-dev, > > > > I'm solving a pure Neumann mixed Poisson-like problem, preconditioning > with a schur complement. The pressure space has a nullspace of the > constant functions and so I attach the appropriate nullspace to the krylov > solver, and compose the constant nullspace with the IS defining the > pressure space. My RHS is consistent. > > > > That is supposed to work, and I think it does in my tests. The code is > here > > > > > https://bitbucket.org/petsc/petsc/src/1f0d623c8336219eb98f7ded6f95c151ca603fe7/src/ksp/pc/impls/fieldsplit/fieldsplit.c?at=master#cl-562 > > > > That code attaches the nullspace attached to the IS to jac->pmat if > appropriate. However, if I'm using -pc_fieldsplit_schur_precondition selfp > then the schur complement is preconditioned with jac->schurp (rather than > jac->pmat[1]) which is not setup until here > https://bitbucket.org/petsc/petsc/src/1f0d623c8336219eb98f7ded6f95c151ca603fe7/src/ksp/pc/impls/fieldsplit/fieldsplit.c?at=master#cl-656 > > at which point the nullspace is not attached, no? > Crap, that is right. I think we should propagate the nullspace from pmat[1]. The next 2 weeks are insane for me. If you do it, I promise to look at the pull request and merge to next. Thanks, Matt > Cheers, > > Lawrence > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Nov 10 22:28:09 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 11 Nov 2014 07:28:09 +0300 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> Message-ID: <87ppcus6na.fsf@jedbrown.org> Matthew Knepley writes: >> at which point the nullspace is not attached, no? >> > > Crap, that is right. I think we should propagate the nullspace from pmat[1]. Uh, there is no reason pmat[1] can be expected to have the same null space. pmat[1] is the zero matrix in some important cases, and the null space of the Schur complement (or its SIMPLE-type approximation) depends on boundary conditions in a different block (thus problematic for segregated assembly). I've been unsuccessful so far in my search for an elegant solution, but passing the null space along in this way is encouraging overt lying and can be expected to break other algorithms as well as any consistency tests we might add in the future. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From atmmachado at gmail.com Mon Nov 10 23:44:51 2014 From: atmmachado at gmail.com (=?UTF-8?B?QW5kcsOpIFRpbcOzdGhlbw==?=) Date: Tue, 11 Nov 2014 03:44:51 -0200 Subject: [petsc-users] petsc4py 3.5 BOUND CONSTRAINED optimization example Message-ID: How can I model this kind of problem with petsc 3.5.2 and petsc4py? The demo examples are unconstrained optimization ones in all the petsc versions... Please, I need any python scrip to deal with problems like: minimize sum(x) subject to. x >= 0 and Ax = b It will be great. Sorry about the elementar question. I need to couple optimization features to FEM python projects like FEniCS. Andre Machado -------------- next part -------------- An HTML attachment was scrubbed... URL: From kvoronin at labchem.sscc.ru Tue Nov 11 02:02:24 2014 From: kvoronin at labchem.sscc.ru (Kirill Voronin) Date: Tue, 11 Nov 2014 15:02:24 +0700 Subject: [petsc-users] PETSC for symmetric SLAE Message-ID: <63dd84852409b753b072170cbd892396.squirrel@mx.sscc.ru> Hello everyone! Can anyone help with the following question - if I am solving a symmetric SLAE (both structure and values), is it possible to store only the upper trinagular part of the matrix and apply all PETSC routines like KSPsolve etc.? I looked thorough the manual but it didn't find anything about it. The only thing I found was an option MAT_SYMMETRIC but as far as I understand it is just a flag for using certain types of methods. Therefore, it seems that I have to provide the full matrix to PETSC routines. Am I right? Would be very thankful for any comments! -- Best regards, Kirill Voronin From jed at jedbrown.org Tue Nov 11 04:01:19 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 11 Nov 2014 13:01:19 +0300 Subject: [petsc-users] PETSC for symmetric SLAE In-Reply-To: <63dd84852409b753b072170cbd892396.squirrel@mx.sscc.ru> References: <63dd84852409b753b072170cbd892396.squirrel@mx.sscc.ru> Message-ID: <8761emrr80.fsf@jedbrown.org> Kirill Voronin writes: > Hello everyone! > > Can anyone help with the following question - if I am solving a symmetric > SLAE (both structure and values), is it possible to store only the upper > trinagular part of the matrix and apply all PETSC routines like KSPsolve > etc.? Use the SBAIJ matrix format. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Tue Nov 11 04:03:39 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 11 Nov 2014 13:03:39 +0300 Subject: [petsc-users] Multiple solves with PCMG fail In-Reply-To: <2242449.GqaeaLRqZi@besikovitch-iii> References: <9621705.dtimTkejkQ@besikovitch-ii> <7794796.iyiicWNzpQ@besikovitch-iii> <2242449.GqaeaLRqZi@besikovitch-iii> Message-ID: <87389qrr44.fsf@jedbrown.org> Filippo Leonardi writes: > I'm stupid: somehow I was in master and didn't notice that. Sorry for the > confusion. The branch 'mark/ksp-zero-eig' has been merged to 'master' now. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From lawrence.mitchell at imperial.ac.uk Tue Nov 11 05:04:51 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Tue, 11 Nov 2014 11:04:51 +0000 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: <87ppcus6na.fsf@jedbrown.org> References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> <87ppcus6na.fsf@jedbrown.org> Message-ID: <5FDBB21C-A76B-48F2-B89C-02CD1DB2FAB2@imperial.ac.uk> On 11 Nov 2014, at 04:28, Jed Brown wrote: > Matthew Knepley writes: >>> at which point the nullspace is not attached, no? >>> >> >> Crap, that is right. I think we should propagate the nullspace from pmat[1]. > > Uh, there is no reason pmat[1] can be expected to have the same null > space. pmat[1] is the zero matrix in some important cases, and the null > space of the Schur complement (or its SIMPLE-type approximation) depends > on boundary conditions in a different block (thus problematic for > segregated assembly). I've been unsuccessful so far in my search for an > elegant solution, but passing the null space along in this way is > encouraging overt lying and can be expected to break other algorithms as > well as any consistency tests we might add in the future. OK, so it's sounds like this is probably a deliberate choice. I note that the code currently does this: fieldsplit.c:703 ierr = MatSetFromOptions(jac->schur);CHKERRQ(ierr); ierr = MatGetNullSpace(jac->pmat[1], &sp);CHKERRQ(ierr); if (sp) { ierr = MatSetNullSpace(jac->schur, sp);CHKERRQ(ierr); } i.e. takes the nullspace from pmat[1] and hangs it on the schur complement. I was under the impression that when I'm informing the field split setup of the nullspace, composing the nullspace with the IS that defines split 1 was informing the PC that the schur complement has this nullspace, but this discussion now has me a little confused. I guess my best option is to user a "user" PC for the schur complement, because then I can set everything up by hand. Cheers, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From knepley at gmail.com Tue Nov 11 05:45:12 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 11 Nov 2014 05:45:12 -0600 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: <87ppcus6na.fsf@jedbrown.org> References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> <87ppcus6na.fsf@jedbrown.org> Message-ID: On Mon, Nov 10, 2014 at 10:28 PM, Jed Brown wrote: > Matthew Knepley writes: > >> at which point the nullspace is not attached, no? > >> > > > > Crap, that is right. I think we should propagate the nullspace from > pmat[1]. > > Uh, there is no reason pmat[1] can be expected to have the same null > space. pmat[1] is the zero matrix in some important cases, and the null > space of the Schur complement (or its SIMPLE-type approximation) depends > on boundary conditions in a different block (thus problematic for > segregated assembly). I've been unsuccessful so far in my search for an > elegant solution, but passing the null space along in this way is > encouraging overt lying and can be expected to break other algorithms as > well as any consistency tests we might add in the future. > I know it can break, however here is my thinking. The user is completely in control here, and this is the only channel we have to pass the information. So I would make them responsible for breakage. We have excellent diagnostics here, so we can tell who has a null space right away. We have a lot of interface that allow a user to break PETSc in order to allow them to customize it sufficiently. Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Vincent.De-Groof at uibk.ac.at Tue Nov 11 06:27:24 2014 From: Vincent.De-Groof at uibk.ac.at (De Groof, Vincent Frans Maria) Date: Tue, 11 Nov 2014 12:27:24 +0000 Subject: [petsc-users] MatSolve in CG - PCPreSolve Message-ID: <17A78B9D13564547AC894B88C1596747203B9483@XMBX4.uibk.ac.at> Dear all, I think I further pinpointed the problem to the PCPreSolve. Both mine, as well as Petsc's PCG call PCPreSolve and enter the (pc->ops->presolve). But mine does not apply the presolve. Do I need to activate the PCPreSolve manually when I have written a KSP that I've registered using KSPRegister? thanks, Vincent ________________________________ Von: De Groof, Vincent Frans Maria Gesendet: Montag, 10. November 2014 17:57 An: petsc-users at mcs.anl.gov Betreff: MatSolve in CG Dear all, I am writing a few variations on the PCG algorithm. Starting with my own PCG implementation, I noticed that my routine is quite a bit slower (~20%). I think I tracked it down to the MatSolve calls in PCApply. I am using BDDC as the preconditioner. In the Petsc PCG function, each PCApply calls the MatSolve 3 times. It also calls MatSolve once somewhere between calling KSPSolve and KSPSolve_CG (KSP_SetUp has been called before). So MatSolve is called 1+3*(# iterations) per KSPSolve. In my own PCG variation, the initial MatSolve is not called. But MatSolve is called 4 times in each iteration, so in total 4*(# iterations). Where does this MatSolve in between KSPSolve and KSPSolve_CG come from? And how can I make my function use less MatSolve's? I hope that I explained the problem clearly, and maybe this is a very trivial question. kind regards, Vincent -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 11 07:54:18 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 11 Nov 2014 07:54:18 -0600 Subject: [petsc-users] MatSolve in CG - PCPreSolve In-Reply-To: <17A78B9D13564547AC894B88C1596747203B9483@XMBX4.uibk.ac.at> References: <17A78B9D13564547AC894B88C1596747203B9483@XMBX4.uibk.ac.at> Message-ID: On Tue, Nov 11, 2014 at 6:27 AM, De Groof, Vincent Frans Maria < Vincent.De-Groof at uibk.ac.at> wrote: > Dear all, > > > I think I further pinpointed the problem to the PCPreSolve. Both mine, > as well as Petsc's PCG call PCPreSolve and enter the (pc->ops->presolve). > But mine does not apply the presolve. > I do not understand what you mean here. It calls PCPreSolve from KSPSolve. Thanks, Matt > Do I need to activate the PCPreSolve manually when I have written a KSP > that I've registered using KSPRegister? > > > > thanks, > Vincent > > > ------------------------------ > *Von:* De Groof, Vincent Frans Maria > *Gesendet:* Montag, 10. November 2014 17:57 > *An:* petsc-users at mcs.anl.gov > *Betreff:* MatSolve in CG > > Dear all, > > > I am writing a few variations on the PCG algorithm. Starting with my own > PCG implementation, I noticed that my routine is quite a bit slower (~20%). > I think I tracked it down to the MatSolve calls in PCApply. I am using BDDC > as the preconditioner. > > In the Petsc PCG function, each PCApply calls the MatSolve 3 times. It > also calls MatSolve once somewhere between calling KSPSolve and KSPSolve_CG > (KSP_SetUp has been called before). So MatSolve is called 1+3*(# > iterations) per KSPSolve. In my own PCG variation, the initial MatSolve is > not called. But MatSolve is called 4 times in each iteration, so in total > 4*(# iterations). > > Where does this MatSolve in between KSPSolve and KSPSolve_CG come from? > And how can I make my function use less MatSolve's? > > > I hope that I explained the problem clearly, and maybe this is a very > trivial question. > > > kind regards, > Vincent > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.m.lacasse at gmail.com Tue Nov 11 08:51:01 2014 From: patrick.m.lacasse at gmail.com (Patrick Lacasse) Date: Tue, 11 Nov 2014 09:51:01 -0500 Subject: [petsc-users] schurcomplement of a rectangular system Message-ID: Hi, in my works, I'm computing a formula (involving an L2 projection) of the form: (D - M^{-1} N) n where D,M and B are matrices and n is a vector. Operator (D - M^{-1} N) appears to be the Schur complement of the system ( M N ) ( I D ) and it is convenient for me to use a Mat of type schurcomplement to compute it. However, my matrix D is not a square matrix (this system is not invertible, it is rectangular). Assertions in src/ksp/ksp/utils/schurm.c prohibit the use of schurcomplement for rectangular system, but I can't figure where it is needed by schurcomplement itself. Could it be possible to remove those two assertions (see my patch)? Patrick Lacasse -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Remove-an-assertion-in-MatSchurComplement.patch Type: text/x-patch Size: 1638 bytes Desc: not available URL: From lb2653 at columbia.edu Tue Nov 11 09:01:07 2014 From: lb2653 at columbia.edu (Luc Berger-Vergiat) Date: Tue, 11 Nov 2014 10:01:07 -0500 Subject: [petsc-users] Lookup error in PetscTableFind() In-Reply-To: References: <544E943F.5070007@purdue.edu> <544FB07E.1020502@columbi.edu> <68E7C94C-4D7C-43EC-BC90-CBC00747660D@mcs.anl.gov> <544FD2BB.6070500@columbi.edu> <4D0B23B5-69EB-4E98-993C-90BC15B5A0CE@mcs.anl.gov> <544FE57C.8020303@columbi.edu> <7955DA16-318F-4E9B-B563-B8FBAB623095@mcs.anl.gov> <54524973.8010103@columbi.edu> <5B51768E-D3B7-4FE9-A2D7-A540C9750DC7@mcs.anl.gov> <5453ECEB.2090207@columbi.edu> Message-ID: <546224B3.3000802@columbi.edu> Hi Barry, sorry for the late answer but I finally figured out what was causing all my issues. I have an integer declared as short somewhere in my code. That obviously is a poor decision that was made when the code was initially written. Given the size of my problem this lead to an integer overflow... I had never dreamed until now I would see this kind of problem outside of a text book but I guess there is always a first! Sorry to have used up all this time just for this, and thanks for your help! Best, Luc On 10/31/2014 05:48 PM, Barry Smith wrote: > I would first go back and build another PETSC_ARCH without optimization and run your job that previously crashed with that and see if it still crashes? If it runs then I would one at a time add back the various optimization flags -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction to see which triggers the problem. > > You could also overload your desktop with the much bigger problem to see if it finds anything else (sure it will run slowly but so what). > > Barry > > >> On Oct 31, 2014, at 3:11 PM, Luc Berger-Vergiat wrote: >> >> So when I run with -no-signal-handler I get an error from valgrind saying that it received a signal and was aborting... >> Is there a way to prune a little what PETSc will get from argc/argv in order to let it run correctly? >> >> I also ran an example on my local computer with the same parameters and with 4 mpi ranks (that's all my desktop can do) and got the following outputs: >> >> luc at euler:~/research/simulations/Petsc_FS/parallel_ISFS/10elem_test/4cores$ mpirun -n 4 /usr/bin/valgrind --leak-check=full --tool=memcheck --track-origins=yes -q /home/luc/research/feap_repo/ShearBands/parfeap/feap -ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps -ksp_diagonal_scale >> >> >> F I N I T E E L E M E N T A N A L Y S I S P R O G R A M >> >> FEAP (C) Regents of the University of California >> All Rights Reserved. >> VERSION: Release 8.3.19 >> DATE: 29 March 2011 >> >> Files are set as: Status Filename >> >> Input (read ) : Exists ILU_0001 >> Output (write) : Exists OLU_0001 >> Restart (read ) : New RLU_0001 >> Restart (write) : New RLU_0001 >> Plots (write) : New PLU_0001 >> >> Caution, existing write files will be overwritten. >> >> Are filenames correct? ( y or n; s = stop) :y >> >> R U N N I N G F E A P P R O B L E M N O W >> >> --> Please report errors by e-mail to: >> feap at ce.berkeley.edu >> >> Saving Parallel data to PLU_000000.pvtu >> ==7933== Syscall param writev(vector[...]) points to uninitialised byte(s) >> ==7933== at 0x6C2EF57: writev (writev.c:49) >> ==7933== by 0x7360F30: MPL_large_writev (mplsock.c:32) >> ==7933== by 0x65AB588: MPIDU_Sock_writev (sock_immed.i:610) >> ==7933== by 0x65940B3: MPIDI_CH3_iSendv (ch3_isendv.c:84) >> ==7933== by 0x65862AC: MPIDI_CH3_EagerContigIsend (ch3u_eager.c:550) >> ==7933== by 0x658AB5C: MPID_Isend (mpid_isend.c:131) >> ==7933== by 0x6625309: PMPI_Isend (isend.c:122) >> ==7933== by 0x656671D: PMPI_ISEND (isendf.c:267) >> ==7933== by 0x1AE7745: __dmumps_comm_buffer_MOD_dmumps_62 (dmumps_comm_buffer.F:567) >> ==7933== by 0x1B6D1FC: dmumps_242_ (dmumps_part2.F:739) >> ==7933== by 0x1AB264C: dmumps_249_ (dmumps_part8.F:6541) >> ==7933== by 0x1AAB0F6: dmumps_245_ (dmumps_part8.F:3885) >> ==7933== Address 0x8959098 is 8 bytes inside a block of size 336 alloc'd >> ==7933== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) >> ==7933== by 0x1AE9465: __dmumps_comm_buffer_MOD_dmumps_2 (dmumps_comm_buffer.F:175) >> ==7933== by 0x1AE9852: __dmumps_comm_buffer_MOD_dmumps_55 (dmumps_comm_buffer.F:123) >> ==7933== by 0x1AC5FD5: dmumps_301_ (dmumps_part8.F:989) >> ==7933== by 0x1B36855: dmumps_ (dmumps_part1.F:665) >> ==7933== by 0x1A170C7: dmumps_f77_ (dmumps_part3.F:6651) >> ==7933== by 0x19EDDCA: dmumps_c (mumps_c.c:422) >> ==7933== by 0x13AF869: MatSolve_MUMPS (mumps.c:606) >> ==7933== by 0xB58981: MatSolve (matrix.c:3122) >> ==7933== by 0x152174C: PCApply_LU (lu.c:198) >> ==7933== by 0x14BAB70: PCApply (precon.c:440) >> ==7933== by 0x164AB13: KSP_PCApply (kspimpl.h:230) >> ==7933== Uninitialised value was created by a stack allocation >> ==7933== at 0x1AAE33D: dmumps_249_ (dmumps_part8.F:5817) >> ==7933== >> ==7932== Syscall param writev(vector[...]) points to uninitialised byte(s) >> ==7932== at 0x6C2EF57: writev (writev.c:49) >> ==7932== by 0x7360F30: MPL_large_writev (mplsock.c:32) >> ==7932== by 0x65AB588: MPIDU_Sock_writev (sock_immed.i:610) >> ==7932== by 0x65940B3: MPIDI_CH3_iSendv (ch3_isendv.c:84) >> ==7932== by 0x65862AC: MPIDI_CH3_EagerContigIsend (ch3u_eager.c:550) >> ==7932== by 0x658AB5C: MPID_Isend (mpid_isend.c:131) >> ==7932== by 0x6625309: PMPI_Isend (isend.c:122) >> ==7932== by 0x656671D: PMPI_ISEND (isendf.c:267) >> ==7932== by 0x1AE7745: __dmumps_comm_buffer_MOD_dmumps_62 (dmumps_comm_buffer.F:567) >> ==7932== by 0x1B6D1FC: dmumps_242_ (dmumps_part2.F:739) >> ==7932== by 0x1AB264C: dmumps_249_ (dmumps_part8.F:6541) >> ==7932== by 0x1AAB0F6: dmumps_245_ (dmumps_part8.F:3885) >> ==7932== Address 0x86e4448 is 8 bytes inside a block of size 336 alloc'd >> ==7932== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) >> ==7932== by 0x1AE9465: __dmumps_comm_buffer_MOD_dmumps_2 (dmumps_comm_buffer.F:175) >> ==7932== by 0x1AE9852: __dmumps_comm_buffer_MOD_dmumps_55 (dmumps_comm_buffer.F:123) >> ==7932== by 0x1AC5FD5: dmumps_301_ (dmumps_part8.F:989) >> ==7932== by 0x1B36855: dmumps_ (dmumps_part1.F:665) >> ==7932== by 0x1A170C7: dmumps_f77_ (dmumps_part3.F:6651) >> ==7932== by 0x19EDDCA: dmumps_c (mumps_c.c:422) >> ==7932== by 0x13AF869: MatSolve_MUMPS (mumps.c:606) >> ==7932== by 0xB58981: MatSolve (matrix.c:3122) >> ==7932== by 0x152174C: PCApply_LU (lu.c:198) >> ==7932== by 0x14BAB70: PCApply (precon.c:440) >> ==7932== by 0x164AB13: KSP_PCApply (kspimpl.h:230) >> ==7932== Uninitialised value was created by a stack allocation >> ==7932== at 0x1AAE33D: dmumps_249_ (dmumps_part8.F:5817) >> ==7932== >> ==7934== Syscall param writev(vector[...]) points to uninitialised byte(s) >> ==7934== at 0x6C2EF57: writev (writev.c:49) >> ==7934== by 0x7360F30: MPL_large_writev (mplsock.c:32) >> ==7934== by 0x65AB588: MPIDU_Sock_writev (sock_immed.i:610) >> ==7934== by 0x65940B3: MPIDI_CH3_iSendv (ch3_isendv.c:84) >> ==7934== by 0x65862AC: MPIDI_CH3_EagerContigIsend (ch3u_eager.c:550) >> ==7934== by 0x658AB5C: MPID_Isend (mpid_isend.c:131) >> ==7934== by 0x6625309: PMPI_Isend (isend.c:122) >> ==7934== by 0x656671D: PMPI_ISEND (isendf.c:267) >> ==7934== by 0x1AE7745: __dmumps_comm_buffer_MOD_dmumps_62 (dmumps_comm_buffer.F:567) >> ==7934== by 0x1B6D1FC: dmumps_242_ (dmumps_part2.F:739) >> ==7934== by 0x1AB264C: dmumps_249_ (dmumps_part8.F:6541) >> ==7934== by 0x1AAB0F6: dmumps_245_ (dmumps_part8.F:3885) >> ==7934== Address 0x89ec648 is 8 bytes inside a block of size 336 alloc'd >> ==7934== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) >> ==7934== by 0x1AE9465: __dmumps_comm_buffer_MOD_dmumps_2 (dmumps_comm_buffer.F:175) >> ==7934== by 0x1AE9852: __dmumps_comm_buffer_MOD_dmumps_55 (dmumps_comm_buffer.F:123) >> ==7934== by 0x1AC5FD5: dmumps_301_ (dmumps_part8.F:989) >> ==7934== by 0x1B36855: dmumps_ (dmumps_part1.F:665) >> ==7934== by 0x1A170C7: dmumps_f77_ (dmumps_part3.F:6651) >> ==7934== by 0x19EDDCA: dmumps_c (mumps_c.c:422) >> ==7934== by 0x13AF869: MatSolve_MUMPS (mumps.c:606) >> ==7934== by 0xB58981: MatSolve (matrix.c:3122) >> ==7934== by 0x152174C: PCApply_LU (lu.c:198) >> ==7934== by 0x14BAB70: PCApply (precon.c:440) >> ==7934== by 0x164AB13: KSP_PCApply (kspimpl.h:230) >> ==7934== Uninitialised value was created by a stack allocation >> ==7934== at 0x1AAE33D: dmumps_249_ (dmumps_part8.F:5817) >> ==7934== >> ==7935== Syscall param writev(vector[...]) points to uninitialised byte(s) >> ==7935== at 0x6C2EF57: writev (writev.c:49) >> ==7935== by 0x7360F30: MPL_large_writev (mplsock.c:32) >> ==7935== by 0x65AB588: MPIDU_Sock_writev (sock_immed.i:610) >> ==7935== by 0x65940B3: MPIDI_CH3_iSendv (ch3_isendv.c:84) >> ==7935== by 0x65862AC: MPIDI_CH3_EagerContigIsend (ch3u_eager.c:550) >> ==7935== by 0x658AB5C: MPID_Isend (mpid_isend.c:131) >> ==7935== by 0x6625309: PMPI_Isend (isend.c:122) >> ==7935== by 0x656671D: PMPI_ISEND (isendf.c:267) >> ==7935== by 0x1AE7745: __dmumps_comm_buffer_MOD_dmumps_62 (dmumps_comm_buffer.F:567) >> ==7935== by 0x1B6D1FC: dmumps_242_ (dmumps_part2.F:739) >> ==7935== by 0x1AB264C: dmumps_249_ (dmumps_part8.F:6541) >> ==7935== by 0x1AAB0F6: dmumps_245_ (dmumps_part8.F:3885) >> ==7935== Address 0x85d8968 is 8 bytes inside a block of size 336 alloc'd >> ==7935== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) >> ==7935== by 0x1AE9465: __dmumps_comm_buffer_MOD_dmumps_2 (dmumps_comm_buffer.F:175) >> ==7935== by 0x1AE9852: __dmumps_comm_buffer_MOD_dmumps_55 (dmumps_comm_buffer.F:123) >> ==7935== by 0x1AC5FD5: dmumps_301_ (dmumps_part8.F:989) >> ==7935== by 0x1B36855: dmumps_ (dmumps_part1.F:665) >> ==7935== by 0x1A170C7: dmumps_f77_ (dmumps_part3.F:6651) >> ==7935== by 0x19EDDCA: dmumps_c (mumps_c.c:422) >> ==7935== by 0x13AF869: MatSolve_MUMPS (mumps.c:606) >> ==7935== by 0xB58981: MatSolve (matrix.c:3122) >> ==7935== by 0x152174C: PCApply_LU (lu.c:198) >> ==7935== by 0x14BAB70: PCApply (precon.c:440) >> ==7935== by 0x164AB13: KSP_PCApply (kspimpl.h:230) >> ==7935== Uninitialised value was created by a stack allocation >> ==7935== at 0x1AAE33D: dmumps_249_ (dmumps_part8.F:5817) >> ==7935== >> Saving Parallel data to PLU_000001.pvtu >> Saving Parallel data to PLU_000002.pvtu >> Saving Parallel data to PLU_000003.pvtu >> Saving Parallel data to PLU_000004.pvtu >> Saving Parallel data to PLU_000005.pvtu >> Saving Parallel data to PLU_000006.pvtu >> Saving Parallel data to PLU_000007.pvtu >> Saving Parallel data to PLU_000008.pvtu >> Saving Parallel data to PLU_000009.pvtu >> Saving Parallel data to PLU_000010.pvtu >> ==7932== 13 bytes in 13 blocks are definitely lost in loss record 7 of 51 >> ==7932== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) >> ==7932== by 0x1BBB3E1: mumps_397.3460 (mumps_static_mapping.F:3746) >> ==7932== by 0x1BB8283: __mumps_static_mapping_MOD_mumps_369 (mumps_static_mapping.F:302) >> ==7932== by 0x1A4BBF5: dmumps_537_ (dmumps_part5.F:1706) >> ==7932== by 0x1A550AC: dmumps_26_ (dmumps_part5.F:447) >> ==7932== by 0x1B346D6: dmumps_ (dmumps_part1.F:409) >> ==7932== by 0x1A170C7: dmumps_f77_ (dmumps_part3.F:6651) >> ==7932== by 0x19EDDCA: dmumps_c (mumps_c.c:422) >> ==7932== by 0x13B2F2B: MatLUFactorSymbolic_AIJMUMPS (mumps.c:972) >> ==7932== by 0xB54FB5: MatLUFactorSymbolic (matrix.c:2842) >> ==7932== by 0x15205A9: PCSetUp_LU (lu.c:127) >> ==7932== by 0x14C0685: PCSetUp (precon.c:902) >> ==7932== >> ==7932== 13 bytes in 13 blocks are definitely lost in loss record 8 of 51 >> ==7932== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) >> ==7932== by 0x1BBB5C7: mumps_397.3460 (mumps_static_mapping.F:3746) >> ==7932== by 0x1BB8283: __mumps_static_mapping_MOD_mumps_369 (mumps_static_mapping.F:302) >> ==7932== by 0x1A4BBF5: dmumps_537_ (dmumps_part5.F:1706) >> ==7932== by 0x1A550AC: dmumps_26_ (dmumps_part5.F:447) >> ==7932== by 0x1B346D6: dmumps_ (dmumps_part1.F:409) >> ==7932== by 0x1A170C7: dmumps_f77_ (dmumps_part3.F:6651) >> ==7932== by 0x19EDDCA: dmumps_c (mumps_c.c:422) >> ==7932== by 0x13B2F2B: MatLUFactorSymbolic_AIJMUMPS (mumps.c:972) >> ==7932== by 0xB54FB5: MatLUFactorSymbolic (matrix.c:2842) >> ==7932== by 0x15205A9: PCSetUp_LU (lu.c:127) >> ==7932== by 0x14C0685: PCSetUp (precon.c:902) >> ==7932== >> All these mainly come from mumps and I don't think that they would create the memory problem that I described earlier. >> It really seems that the issue comes with larger problems with more mpi ranks. >> Best, >> Luc >> >> On 10/30/2014 12:23 PM, Barry Smith wrote: >>> Run with the additional PETSc option -no_signal_handler then PETSc won?t mess with the signals and it may get you past this point. >>> >>> Barry >>> >>> >>> >>> >>>> On Oct 30, 2014, at 9:21 AM, Luc Berger-Vergiat >>>> wrote: >>>> >>>> Sorry for the late reply, it took longer than I thought. >>>> So a little update on my situation: >>>> I have to use a custom version of valgrind on CETUS which has to be linked to my code using -Wl,-e,_start_valgrind at compilation (I also add object file and libraries). >>>> After that I can run my code with the following arguments: >>>> feap --ignore-ranges=0x4000000000000-0x4063000000000,0x003fdc0000000-0x003fe00000000 --suppressions=/soft/perftools/valgrind/cnk-baseline.supp >>>> but I can't use the usual petsc argument (-ksp_type -pc_type ...) since valgrind does not recognize them. >>>> So I decided to use the PETSC_OPTIONS='-ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps -ksp_diagonal_scale', to pass my arguments to PETSc. >>>> However I am a little worried that Petsc still receive the command line arguments that are used by valgrind since I get the following stderr from valgrind: >>>> >>>> stderr[0]: ==1== by 0x38BEA3F: handle_SCSS_change (m_signals.c:963) >>>> stderr[0]: ==1== by 0x38C13A7: vgPlain_do_sys_sigaction (m_signals.c:1114) >>>> stderr[0]: ==1== by 0x3962FBF: vgSysWrap_linux_sys_rt_sigaction_before (syswrap-linux.c:3073) >>>> stderr[0]: ==1== by 0x3928BF7: vgPlain_client_syscall (syswrap-main.c:1464) >>>> stderr[0]: ==1== by 0x3925F4B: vgPlain_scheduler (scheduler.c:1061) >>>> stderr[0]: ==1== by 0x3965EB3: run_a_thread_NORETURN (syswrap-linux.c:103) >>>> stderr[0]: >>>> >>> >>> >>>> stderr[0]: sched status: >>>> stderr[0]: running_tid=1 >>>> stderr[0]: >>>> stderr[0]: Thread 1: status = VgTs_Runnable >>>> stderr[0]: ==1== at 0x34290E4: __libc_sigaction (sigaction.c:80) >>>> stderr[0]: ==1== by 0x3BFF3A7: signal (signal.c:49) >>>> stderr[0]: ==1== by 0x1E710DF: PetscPushSignalHandler (in /projects/shearbands/ShearBands/parfeap/feap) >>>> stderr[0]: ==1== by 0x18BEA87: PetscOptionsCheckInitial_Private (in /projects/shearbands/ShearBands/parfeap/feap) >>>> stderr[0]: ==1== by 0x18E132F: petscinitialize (in /projects/shearbands/ShearBands/parfeap/feap) >>>> stderr[0]: ==1== by 0x1027557: pstart (in /projects/shearbands/ShearBands/parfeap/feap) >>>> stderr[0]: ==1== by 0x1000B1F: MAIN__ (feap83.f:213) >>>> stderr[0]: ==1== by 0x342ABD7: main (fmain.c:21) >>>> stderr[0]: >>>> stderr[0]: >>>> stderr[0]: Note: see also the FAQ in the source distribution. >>>> stderr[0]: It contains workarounds to several common problems. >>>> stderr[0]: In particular, if Valgrind aborted or crashed after >>>> stderr[0]: identifying problems in your program, there's a good chance >>>> stderr[0]: that fixing those problems will prevent Valgrind aborting or >>>> stderr[0]: crashing, especially if it happened in m_mallocfree.c. >>>> stderr[0]: >>>> stderr[0]: If that doesn't help, please report this bug to: >>>> www.valgrind.org >>>> >>>> stderr[0]: >>>> stderr[0]: In the bug report, send all the above text, the valgrind >>>> stderr[0]: version, and what OS and version you are using. Thank >>>> stderr[0]: s. >>>> stderr[0]: >>>> >>>> I am only showing the output of rank[0] but it seems that all ranks have about the same error message. >>>> Since my problem happens in petscinitialize I have little possibilities to check what's wrong... >>>> Any ideas? >>>> Best, >>>> Luc >>>> >>>> On 10/28/2014 02:53 PM, Barry Smith wrote: >>>> >>>>> You don?t care about checking for leaks. I use >>>>> >>>>> -q --tool=memcheck --num-callers=20 --track-origins=yes >>>>> >>>>> >>>>> >>>>>> On Oct 28, 2014, at 1:50 PM, Luc Berger-Vergiat >>>>>> >>>>>> wrote: >>>>>> >>>>>> Yes, I am running with --leak-check=full >>>>>> Reconfiguring and recompiling the whole library and my code in debug mode does take quite some time on CETUS/MIRA... >>>>>> Hopefully the queue will go up fast and I can give you some details about the issue. >>>>>> >>>>>> Best, >>>>>> Luc >>>>>> >>>>>> On 10/28/2014 02:25 PM, Barry Smith wrote: >>>>>> >>>>>> >>>>>>> You need to pass some options to valgrind telling it to check for memory corruption issues >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Oct 28, 2014, at 12:30 PM, Luc Berger-Vergiat >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Ok, I'm recompiling PETSc in debug mode then. >>>>>>>> Do you know what the call sequence should be on CETUS to get valgrind attached to PETSc? >>>>>>>> Would this work for example: >>>>>>>> runjob --np 32 -p 8 --block $COBALT_PARTNAME --cwd /projects/shearbands/job1/200/4nodes_32cores/LU --verbose=INFO --envs FEAPHOME8_3=/projects/shearbands/ShearBands352 PETSC_DIR=/projects/shearbands/petsc-3.5.2 PETSC_ARCH=arch-linux2-c-opt : /usr/bin/valgrind --log-file=valgrind.log.%p /projects/shearbands/ShearBands352/parfeap/feap -ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps -ksp_diagonal_scale < /projects/shearbands/job1/yesfile >>>>>>>> >>>>>>>> >>>>>>>> Best, >>>>>>>> Luc >>>>>>>> >>>>>>>> On 10/28/2014 12:33 PM, Barry Smith wrote: >>>>>>>> >>>>>>>> >>>>>>>>> Hmm, this should never happen. In the code >>>>>>>>> >>>>>>>>> ierr = PetscTableCreate(aij->B->rmap->n,mat->cmap->N+1,&gid1_lid1);CHKERRQ(ierr); >>>>>>>>> for (i=0; iB->rmap->n; i++) { >>>>>>>>> for (j=0; jilen[i]; j++) { >>>>>>>>> PetscInt data,gid1 = aj[B->i[i] + j] + 1; >>>>>>>>> ierr = PetscTableFind(gid1_lid1,gid1,&data);CHKERRQ(ierr); >>>>>>>>> >>>>>>>>> Now mat->cmap->N+1 is the total number of columns in the matrix and gid1 are column entries which must always be smaller. Most likely there has been memory corruption somewhere before this point. Can you run with valgrind? >>>>>>>>> >>>>>>>>> >>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Oct 28, 2014, at 10:04 AM, Luc Berger-Vergiat >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> I am running a code on CETUS and I use PETSc for as a linear solver. >>>>>>>>>> Here is my submission command: >>>>>>>>>> qsub -A shearbands -t 60 -n 4 -O 4nodes_32cores_Mult --mode script 4nodes_32cores_LU >>>>>>>>>> >>>>>>>>>> Here is "4nodes_32cores_LU": >>>>>>>>>> #!/bin/sh >>>>>>>>>> >>>>>>>>>> LOCARGS="--block $COBALT_PARTNAME ${COBALT_CORNER:+--corner} $COBALT_CORNER ${COBALT_SHAPE:+--shape} $COBALT_SHAPE" >>>>>>>>>> echo "Cobalt location args: $LOCARGS" >&2 >>>>>>>>>> >>>>>>>>>> ################################ >>>>>>>>>> # 32 cores on 4 nodes jobs # >>>>>>>>>> ################################ >>>>>>>>>> runjob --np 32 -p 8 --block $COBALT_PARTNAME --cwd /projects/shearbands/job1/200/4nodes_32cores/LU --verbose=INFO --envs FEAPHOME8_3=/projects/shearbands/ShearBands352 PETSC_DIR=/projects/shearbands/petsc-3.5.2 PETSC_ARCH=arch-linux2-c-opt : /projects/shearbands/ShearBands352/parfeap/feap -ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps -ksp_diagonal_scale -malloc_log mlog -log_summary time.log < /projects/shearbands/job1/yesfile >>>>>>>>>> >>>>>>>>>> I get the following error message: >>>>>>>>>> >>>>>>>>>> [7]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>>> [7]PETSC ERROR: Argument out of range >>>>>>>>>> [7]PETSC ERROR: Petsc Release Version 3.5.2, unknown >>>>>>>>>> [7]PETSC ERROR: key 532150 is greater than largest key allowed 459888 >>>>>>>>>> [7]PETSC ERROR: Configure options --known-mpi-int64_t=1 --download-cmake=1 --download-hypre=1 --download-metis=1 --download-parmetis=1 --download-plapack=1 --download-superlu_dist=1 --download-mumps=1 --download-ml=1 --known-bits-per-byte=8 --known-level1-dcache-assoc=0 --known-level1-dcache-linesize=32 --known-level1-dcache-size=32768 --known-memcmp-ok=1 --known-mpi-c-double-complex=1 --known-mpi-long-double=1 --known-mpi-shared-libraries=0 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8 --known-sizeof-void-p=8 --with-batch=1 --with-blacs-include=/soft/libraries/alcf/current/gcc/SCALAPACK/ --with-blacs-lib=/soft/libraries/alcf/current/gcc/SCALAPACK/lib/libscalapack.a --with-blas-lapack-lib="-L/soft/libraries/alcf/current/gcc/LAPACK/lib -llapack -L/soft/libraries/alcf/current/gcc/BLAS/lib >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -lblas" - >>>>>>>>>> -with-cc=mpicc --with-cxx=mpicxx --with-debugging=0 --with-fc=mpif90 --with-fortran-kernels=1 --with-is-color-value-type=short --with-scalapack-include=/soft/libraries/alcf/current/gcc/SCALAPACK/ --with-scalapack-lib=/soft/libraries/alcf/current/gcc/SCALAPACK/lib/libscalapack.a --with-shared-libraries=0 --with-x=0 -COPTFLAGS=" -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction" -CXXOPTFLAGS=" -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction" -FOPTFLAGS=" -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction" >>>>>>>>>> >>>>>>>>>> [7]PETSC ERROR: #1 PetscTableFind() line 126 in /gpfs/mira-fs1/projects/shearbands/petsc-3.5.2/include/petscctable.h >>>>>>>>>> [7]PETSC ERROR: #2 MatSetUpMultiply_MPIAIJ() line 33 in /gpfs/mira-fs1/projects/shearbands/petsc-3.5.2/src/mat/impls/aij/mpi/mmaij.c >>>>>>>>>> [7]PETSC ERROR: #3 MatAssemblyEnd_MPIAIJ() line 702 in /gpfs/mira-fs1/projects/shearbands/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c >>>>>>>>>> [7]PETSC ERROR: #4 MatAssemblyEnd() line 4900 in /gpfs/mira-fs1/projects/shearbands/petsc-3.5.2/src/mat/interface/matrix.c >>>>>>>>>> >>>>>>>>>> Well at least that is what I think comes out after I read all the jammed up messages from my MPI processes... >>>>>>>>>> >>>>>>>>>> I would guess that I am trying to allocate more memory than I should which seems strange since the same problem runs fine on 2 nodes with 16 cores/node >>>>>>>>>> >>>>>>>>>> Thanks for the help >>>>>>>>>> Best, >>>>>>>>>> Luc >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> >>>>>> From jed at jedbrown.org Tue Nov 11 09:18:21 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 11 Nov 2014 18:18:21 +0300 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: <5FDBB21C-A76B-48F2-B89C-02CD1DB2FAB2@imperial.ac.uk> References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> <87ppcus6na.fsf@jedbrown.org> <5FDBB21C-A76B-48F2-B89C-02CD1DB2FAB2@imperial.ac.uk> Message-ID: <87zjbxrcjm.fsf@jedbrown.org> Lawrence Mitchell writes: > OK, so it's sounds like this is probably a deliberate choice. I note that the code currently does this: > > fieldsplit.c:703 > > ierr = MatSetFromOptions(jac->schur);CHKERRQ(ierr); > ierr = MatGetNullSpace(jac->pmat[1], &sp);CHKERRQ(ierr); > if (sp) { > ierr = MatSetNullSpace(jac->schur, sp);CHKERRQ(ierr); > } > > i.e. takes the nullspace from pmat[1] and hangs it on the schur > complement. I was under the impression that when I'm informing the > field split setup of the nullspace, composing the nullspace with the > IS that defines split 1 was informing the PC that the schur complement > has this nullspace, but this discussion now has me a little confused. Matt and I have a long-running disagreement about nullspace handling in PETSc. He added the code that hangs it on an IS despite it not logically belonging on IS, simply because the implementation in PETSc happens to pass the IS along to the place he needed a nullspace. He's also making a bunch of assumptions about nullspaces agreeing in cases where there is no such guarantee (demonstrably, and in problems that we care about, but not on the problem he was solving at that time). Matt would be happy if someone fixed the code to handle nullspaces in a logically consistent way, but we can't just tear out his inconsistent code without having a replacement. Sadly, all the replacements I can think of are onerous for the user and potentially hard to implement/maintain. So I mostly stay quiet and work on other things, but it pains me every time a user grows a dependency on the abuse of ISs and perverse assumptions about nullspaces agreeing. I'd love to hear suggestions about how to get nullspace information into inner solvers without violating the object model. > I guess my best option is to user a "user" PC for the schur > complement, because then I can set everything up by hand. That isn't a bad choice. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From lb2653 at columbia.edu Tue Nov 11 09:22:23 2014 From: lb2653 at columbia.edu (Luc Berger-Vergiat) Date: Tue, 11 Nov 2014 10:22:23 -0500 Subject: [petsc-users] passing solver options in fieldsplit Message-ID: <546229AF.1010808@columbi.edu> Hi, I am using Petsc to solver a multiphysics problem and I have the following issue. I partition my problem by declaring two fields: -ksp_type gmres -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type full -pc_fieldsplit_schur_precondition selfp -pc_fieldsplit_0_fields 2,3 -pc_fieldsplit_1_fields 0,1 I want to solve the matrix representing field 0 with mumps so I pass the following following arguments: -fieldsplit_0_ksp_type preonly -fieldsplit_0_pc_type lu -fieldsplit_0_pc_factor_mat_solver_package mumps When I do this I get an error from mumps: INFO(1)=-9, INFO(2)=12532. This means that mumps main internal real workarray is too small and 12532 are missing. To try to mitigate this I need to want to set mumps ICNTL(14)=30 (by default it is 20). Reading Petsc documentation I find that I have to pass the following argument to my program: -mat_mumps_icntl_14 30 which does work fine when I work without fieldsplit but not when I use field split. I also tried: -fieldsplit_0_mat_mumps_icntl_14 30 which does not work any better. Any idea how I should pass the icntl_14 information to mumps in this case? -- Best, Luc -------------- next part -------------- An HTML attachment was scrubbed... URL: From lb2653 at columbia.edu Tue Nov 11 09:33:28 2014 From: lb2653 at columbia.edu (Luc Berger-Vergiat) Date: Tue, 11 Nov 2014 10:33:28 -0500 Subject: [petsc-users] passing solver options in fieldsplit In-Reply-To: <546229AF.1010808@columbi.edu> References: <546229AF.1010808@columbi.edu> Message-ID: <54622C48.5000803@columbi.edu> Hi, I am using Petsc to solver a multiphysics problem and I have the following issue. I partition my problem by declaring two fields: -ksp_type gmres -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type full -pc_fieldsplit_schur_precondition selfp -pc_fieldsplit_0_fields 2,3 -pc_fieldsplit_1_fields 0,1 I want to solve the matrix representing field 0 with mumps so I pass the following following arguments: -fieldsplit_0_ksp_type preonly -fieldsplit_0_pc_type lu -fieldsplit_0_pc_factor_mat_solver_package mumps When I do this I get an error from mumps: INFO(1)=-9, INFO(2)=12532. This means that mumps main internal real workarray is too small and 12532 are missing. To try to mitigate this I need to want to set mumps ICNTL(14)=30 (by default it is 20). Reading Petsc documentation I find that I have to pass the following argument to my program: -mat_mumps_icntl_14 30 which does work fine when I work without fieldsplit but not when I use field split. I also tried: -fieldsplit_0_mat_mumps_icntl_14 30 which does not work any better. Any idea how I should pass the icntl_14 information to mumps in this case? -- Best, Luc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Nov 11 10:07:34 2014 From: hzhang at mcs.anl.gov (Hong) Date: Tue, 11 Nov 2014 10:07:34 -0600 Subject: [petsc-users] passing solver options in fieldsplit In-Reply-To: <546229AF.1010808@columbi.edu> References: <546229AF.1010808@columbi.edu> Message-ID: Luc: Run your code with option '-help |grep mumps', then you'll see what prefix should be used in your case with the mumps option '-mat_mumps_icntl_14 30'. You may try even larger icntl_14. Hong Hi, I am using Petsc to solver a multiphysics problem and I have the > following issue. > I partition my problem by declaring two fields: > > -ksp_type gmres -pc_type fieldsplit -pc_fieldsplit_type schur > -pc_fieldsplit_schur_factorization_type full > -pc_fieldsplit_schur_precondition selfp -pc_fieldsplit_0_fields 2,3 > -pc_fieldsplit_1_fields 0,1 > > I want to solve the matrix representing field 0 with mumps so I pass the > following following arguments: > > -fieldsplit_0_ksp_type preonly -fieldsplit_0_pc_type lu > -fieldsplit_0_pc_factor_mat_solver_package mumps > > When I do this I get an error from mumps: INFO(1)=-9, INFO(2)=12532. This > means that mumps main internal real workarray is too small and 12532 are > missing. To try to mitigate this I need to want to set mumps ICNTL(14)=30 > (by default it is 20). > Reading Petsc documentation I find that I have to pass the following > argument to my program: > > -mat_mumps_icntl_14 30 > > which does work fine when I work without fieldsplit but not when I use > field split. > I also tried: > > -fieldsplit_0_mat_mumps_icntl_14 30 > > which does not work any better. > Any idea how I should pass the icntl_14 information to mumps in this case? > > -- > Best, > Luc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Nov 11 18:00:29 2014 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 12 Nov 2014 01:00:29 +0100 Subject: [petsc-users] Origin of f2cblaslapack source code Message-ID: Hello, A colleague of mine asked me about quad precision support in blas/lapack libraries. Naturally I said just use petsc... however he couldn't be convinced. I thought that the f2cblaslapack source which petsc's configure downloaded and compiled would be identical to that found at Netlib. However, a quick inspection of both source codes reveals that this is not the case. I asked Matt and he told me that he thought Satish generated the particular source which petsc is fetching when --download-f2cblaslapack is invoked. I was wondering (i) how was the quad precision source actually generated and (ii) how extensively tested is that source code when used for quad precision arithmetic? Cheers, Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Nov 11 18:37:55 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 11 Nov 2014 18:37:55 -0600 Subject: [petsc-users] Origin of f2cblaslapack source code In-Reply-To: References: Message-ID: <20A7C2DC-62A6-4443-AB6A-30CC48388408@mcs.anl.gov> The tool is /bin/maint/toclapack.sh and is applied to the Fortran BLAS/LAPACK source. Credits are at the top of the file. We don't run the script each time but run the script once and generate a tarball which is downloaded on request. The testing of the source code for quad precision amounts to running the PETSc/TAO/SLEPc test suite, so it is not as extensive as it could be and only tests the functionality that we use. It would be great if someone figured out how to do a comprehensive test suite. I love quad precision but the general numerical computing community doesn't seem to care much about it hence such tools are really underdeveloped. I don't know of other commonly used numerical packages besides PETSc/TAO/SLEPc that that advantage of the __float128 support in GNU. Barry The Netlib effort to generate C code from Fortran BLAS/LAPACK source is really a piss-poor effort. > On Nov 11, 2014, at 6:00 PM, Dave May wrote: > > Hello, > > A colleague of mine asked me about quad precision support in blas/lapack libraries. Naturally I said just use petsc... however he couldn't be convinced. > > I thought that the f2cblaslapack source which petsc's configure downloaded and compiled would be identical to that found at Netlib. However, a quick inspection of both source codes reveals that this is not the case. > > I asked Matt and he told me that he thought Satish generated the particular source which petsc is fetching when --download-f2cblaslapack is invoked. > > I was wondering > (i) how was the quad precision source actually generated and > (ii) how extensively tested is that source code when used for quad precision arithmetic? > > > Cheers, > Dave From lawrence.mitchell at imperial.ac.uk Wed Nov 12 05:21:56 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Wed, 12 Nov 2014 11:21:56 +0000 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: <87zjbxrcjm.fsf@jedbrown.org> References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> <87ppcus6na.fsf@jedbrown.org> <5FDBB21C-A76B-48F2-B89C-02CD1DB2FAB2@imperial.ac.uk> <87zjbxrcjm.fsf@jedbrown.org> Message-ID: <546342D4.7080508@imperial.ac.uk> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/11/14 15:18, Jed Brown wrote: ... > Matt and I have a long-running disagreement about nullspace > handling in PETSc. He added the code that hangs it on an IS > despite it not logically belonging on IS, simply because the > implementation in PETSc happens to pass the IS along to the place > he needed a nullspace. He's also making a bunch of assumptions > about nullspaces agreeing in cases where there is no such guarantee > (demonstrably, and in problems that we care about, but not on the > problem he was solving at that time). > > Matt would be happy if someone fixed the code to handle nullspaces > in a logically consistent way, but we can't just tear out his > inconsistent code without having a replacement. Sadly, all the > replacements I can think of are onerous for the user and > potentially hard to implement/maintain. So I mostly stay quiet and > work on other things, but it pains me every time a user grows a > dependency on the abuse of ISs and perverse assumptions about > nullspaces agreeing. I'd love to hear suggestions about how to get > nullspace information into inner solvers without violating the > object model. So is the problem that the nullspace belongs to the appropriate operator, but at the point you tell the solver, you don't necessarily have all the operators to hand? Moreover, depending on the PC choice, the nullspaces of the operator blocks may or may not be relevant? >> I guess my best option is to user a "user" PC for the schur >> complement, because then I can set everything up by hand. > > That isn't a bad choice. I'll go with this, who knows, it may cause me to unearth some better ideas. Cheers, Lawrence -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQEcBAEBAgAGBQJUY0LUAAoJECOc1kQ8PEYvN/kH/2wtgzGEowR1m0hVnt4kPksw ZaRCgx4seU+yrXQceegIgGtWI8JKoZBljh7115Z2jxyeCbW1MNxSYKlPWfOY8DKE gph3QnnTEmlH/s4SMQ5utzhHFWSDAk2EOkVegoPkWAwbrgc6C9amsTm9Fut25CGI b15WzWwvlKu+5Y8jbHQpYXyHwyLCF3F9fG1ERdtjBuPZHZdxPwneufnJSwnLGX6+ SKrWVuyJMts5trOsRpxHLIAF8I/Fw36vyVAOx8lu4vSSHpTjKVhEtDesacjJo/i6 U4O8GjozoozdbSOtkoWK8uLO1kpFcn7p9Z554AketKTTXj5ef00hZytZOnFu3aI= =ZPaE -----END PGP SIGNATURE----- From christopher.thiele at itwm.fraunhofer.de Wed Nov 12 09:56:18 2014 From: christopher.thiele at itwm.fraunhofer.de (Christopher Thiele) Date: Wed, 12 Nov 2014 16:56:18 +0100 Subject: [petsc-users] Extract subsection of DMDA Message-ID: <54638322.5040407@itwm.fraunhofer.de> Hello, I have a 3d DMDA and a vector created by DMCreateGlobalVector. I want to visualize and therefore export a subset of the values, i.e. I want to "remove" some points at the boundary. For example, if the DMDA represents the domain [0,1]^3, I want to extract [0.25,0.75]^3. I already tried to create another (smaller) DMDA, use DMDAVecGetArray on both of them and then copy the values. The problem is that the vectors are distributed differently, so this approach leads to segfaults. Is there another way to do this? Regards, Christopher From knepley at gmail.com Wed Nov 12 10:36:04 2014 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Nov 2014 10:36:04 -0600 Subject: [petsc-users] Extract subsection of DMDA In-Reply-To: <54638322.5040407@itwm.fraunhofer.de> References: <54638322.5040407@itwm.fraunhofer.de> Message-ID: On Wed, Nov 12, 2014 at 9:56 AM, Christopher Thiele < christopher.thiele at itwm.fraunhofer.de> wrote: > Hello, > > I have a 3d DMDA and a vector created by DMCreateGlobalVector. I want to > visualize and therefore export a subset of the values, i.e. I want to > "remove" some points at the boundary. For example, if the DMDA > represents the domain [0,1]^3, I want to extract [0.25,0.75]^3. > I already tried to create another (smaller) DMDA, use DMDAVecGetArray on > both of them and then copy the values. The problem is that the vectors > are distributed differently, so this approach leads to segfaults. > Is there another way to do this? > You can force the same layout using lx, ly, lz in the constructor. However, this kind of clipping is usually much easier in a visualization program. Thanks, Matt > Regards, > Christopher > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From stm8086 at yahoo.com Wed Nov 12 13:32:10 2014 From: stm8086 at yahoo.com (Steena M) Date: Wed, 12 Nov 2014 11:32:10 -0800 Subject: [petsc-users] Setting up MPIBAIJ for MatMult Message-ID: <1415820730.7969.YahooMailBasic@web125402.mail.ne1.yahoo.com> I am trying to read in a sparse matrix file in binary format (.dat file) to set it up as MPIBAIJ for MPIMatMult. I don't think I'm doing the file reading or the matrix and vector setup correctly. Should the file loading, matrix setup, and loading be done only on rank 0 together with the vector setup? After looking through several examples, this is the structure of my code right now: Vec x,y; Mat A; PetscViewer fd; int rank, global_row_size, global_col_size,bs; PetscBool PetscPreLoad = PETSC_FALSE; char filein[PETSC_MAX_PATH_LEN] /*binary .dat matrix file */ PetscScalar one = 1.0; /* Reading relevant block size from ENV */ char* bs_env; bs_env = getenv ("BLOCK_SIZE_ENV"); bs = (PetscInt)atoi(bs_env); /*Reading in matrix row/cols size from ENV. Matrices are always square: global_col_size=global_row_size*/ char* m_env; m_env = getenv ("MATRIX_ROWS_ENV"); global_row_size = (PetscInt)atoi(m_env); global_col_size = global _row_size; PetscInitialize(&argc,&args,(char *)0,help); ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); /*Matrix reading, setup, assembly*/ if (flg) PetscPreLoad = PETSC_TRUE; PetscPreLoadBegin(PetscPreLoad,"Load"); ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,filein[PetscPreLoadIt],FILE_MODE_READ,&fd);CHKERRQ(ierr); ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,global_row_size,global_col_size);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); ierr = MatSetType(A, MATMPIBAIJ); ierr = MatMPIBAIJSetPreallocation(A,bs,5,PETSC_NULL,5,PETSC_NULL);CHKERRQ(ierr); ierr = MatSeqBAIJSetPreallocation(A,bs,5,PETSC_NULL);CHKERRQ(ierr); ierr = PetscLogStageRegister("Assembly", &stage);CHKERRQ(ierr); ierr = PetscLogStagePush(stage);CHKERRQ(ierr); if (rank==0) { ierr = MatLoad(A,fd);CHKERRQ(ierr); } ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = PetscLogStagePop();CHKERRQ(ierr); ierr = PetscViewerDestroy(&fd);CHKERRQ(ierr); /*Trying two types of vector assembly*/ /* Vector setup Attempt 1*/ ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); ierr = VecSetSizes(x,PETSC_DECIDE,global_row_size);CHKERRQ(ierr); ierr = VecSetFromOptions(x);CHKERRQ(ierr); ierr = VecDuplicate(x,&y);CHKERRQ(ierr); ierr = VecSetFromOptions(y);CHKERRQ(ierr); ierr = VecSet(x,one);CHKERRQ(ierr); ierr = VecSet(y,one); CHKERRQ(ierr); /*Vector setup Attempt 2*/ ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); ierr = VecGetSize(x,&global_row_size);CHKERRQ(ierr); if (rank==0) { for (i = 0; i References: <1415820730.7969.YahooMailBasic@web125402.mail.ne1.yahoo.com> Message-ID: <3469D6F4-7E56-41E6-8E86-237F7FFE2791@mcs.anl.gov> 1) remove all the "preload" stuff 2) MatLoad doesn't require or use the Mat preallocation stuff so remove all of that 3) MatLoad determines the global matrix size from the binary file so do not set that before calling MatLoad 4) All processes in the comm (in your case MPI_COMM_WORLD) need to call MatLoad 5) You can call MatSetBlockSize() to set the block size before MatLoad() 6) You do not need to call MatAssemblyBegin/End after a MatLoad() it would do nothing > On Nov 12, 2014, at 1:32 PM, Steena M wrote: > > I am trying to read in a sparse matrix file in binary format (.dat file) to set it up as MPIBAIJ for MPIMatMult. I don't think I'm doing the file reading or the matrix and vector setup correctly. Should the file loading, matrix setup, and loading be done only on rank 0 together with the vector setup? After looking through several examples, this is the structure of my code right now: > > Vec x,y; > Mat A; > PetscViewer fd; > int rank, global_row_size, global_col_size,bs; > PetscBool PetscPreLoad = PETSC_FALSE; > char filein[PETSC_MAX_PATH_LEN] /*binary .dat matrix file */ > PetscScalar one = 1.0; > > > /* Reading relevant block size from ENV */ > char* bs_env; > bs_env = getenv ("BLOCK_SIZE_ENV"); > bs = (PetscInt)atoi(bs_env); > > /*Reading in matrix row/cols size from ENV. Matrices are always square: global_col_size=global_row_size*/ > > char* m_env; > m_env = getenv ("MATRIX_ROWS_ENV"); > global_row_size = (PetscInt)atoi(m_env); > global_col_size = global _row_size; > > PetscInitialize(&argc,&args,(char *)0,help); > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); > > /*Matrix reading, setup, assembly*/ > > if (flg) PetscPreLoad = PETSC_TRUE; > PetscPreLoadBegin(PetscPreLoad,"Load"); > ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,filein[PetscPreLoadIt],FILE_MODE_READ,&fd);CHKERRQ(ierr); > > > ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); > ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,global_row_size,global_col_size);CHKERRQ(ierr); > ierr = MatSetFromOptions(A);CHKERRQ(ierr); > ierr = MatSetType(A, MATMPIBAIJ); > ierr = MatMPIBAIJSetPreallocation(A,bs,5,PETSC_NULL,5,PETSC_NULL);CHKERRQ(ierr); > ierr = MatSeqBAIJSetPreallocation(A,bs,5,PETSC_NULL);CHKERRQ(ierr); > > ierr = PetscLogStageRegister("Assembly", &stage);CHKERRQ(ierr); > ierr = PetscLogStagePush(stage);CHKERRQ(ierr); > > > if (rank==0) > { > ierr = MatLoad(A,fd);CHKERRQ(ierr); > > } > > ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = PetscLogStagePop();CHKERRQ(ierr); > ierr = PetscViewerDestroy(&fd);CHKERRQ(ierr); > > /*Trying two types of vector assembly*/ > > /* Vector setup Attempt 1*/ > > ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); > ierr = VecSetSizes(x,PETSC_DECIDE,global_row_size);CHKERRQ(ierr); > ierr = VecSetFromOptions(x);CHKERRQ(ierr); > ierr = VecDuplicate(x,&y);CHKERRQ(ierr); > ierr = VecSetFromOptions(y);CHKERRQ(ierr); > ierr = VecSet(x,one);CHKERRQ(ierr); > ierr = VecSet(y,one); CHKERRQ(ierr); > > /*Vector setup Attempt 2*/ > > ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); > ierr = VecGetSize(x,&global_row_size);CHKERRQ(ierr); > > if (rank==0) > { > > for (i = 0; i { > ierr = VecSetValues(x,1,&i,one,INSERT_VALUES); > CHKERRQ(ierr); > } > > } > ierr = VecDuplicate(x,&y);CHKERRQ(ierr); > ierr = VecSetFromOptions(y);CHKERRQ(ierr); > > ierr = VecAssemblyBegin(x); CHKERRQ(ierr); > ierr = VecAssemblyEnd(x); CHKERRQ(ierr); > > > > /* SpMV*/ > ierr = MatMult(A,x,y);CHKERRQ(ierr); > > ierr = VecDestroy(&x);CHKERRQ(ierr); > ierr = VecDestroy(&y);CHKERRQ(ierr); > ierr = MatDestroy(&A);CHKERRQ(ierr); > > > > From brianyang1106 at gmail.com Wed Nov 12 14:23:44 2014 From: brianyang1106 at gmail.com (Brian Yang) Date: Wed, 12 Nov 2014 14:23:44 -0600 Subject: [petsc-users] Nonconforming object sizes ERROR Message-ID: Hi, I am solving an over-determined linear system Ax=b and b=0. A size: 296856 x 14430 x (non-zero guess): 14430 b (zero): 296856 All of them are using seq mode to create, so one node will solve this. However I got this error: [0]PETSC ERROR: Nonconforming object sizes! [0]PETSC ERROR: Mat mat,Vec y: global dim 296856 14430! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: *MatMult() line 2171 in src/mat/interface/matrix.c* [0]PETSC ERROR: KSP_MatMult() line 204 in src/ksp/ksp/impls/cg//datadb/raafat/src/petsc-3.4.3/include/petsc-private/kspimpl.h [0]PETSC ERROR: KSPSolve_CG() line 132 in src/ksp/ksp/impls/cg/cg.c [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c Any restrictions here or I misunderstood something? Thanks! -- Brian Yang U of Houston -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Nov 12 14:29:04 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 Nov 2014 14:29:04 -0600 Subject: [petsc-users] Nonconforming object sizes ERROR In-Reply-To: References: Message-ID: The calling sequence for KSPSolve is KSPSolve(KSP ksp,Vec b,Vec x), perhaps you have the b and the x backwards in your call? Barry > On Nov 12, 2014, at 2:23 PM, Brian Yang wrote: > > Hi, > > I am solving an over-determined linear system Ax=b and b=0. > > A size: 296856 x 14430 > x (non-zero guess): 14430 > b (zero): 296856 > > All of them are using seq mode to create, so one node will solve this. However I got this error: > > [0]PETSC ERROR: Nonconforming object sizes! > [0]PETSC ERROR: Mat mat,Vec y: global dim 296856 14430! > > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2171 in src/mat/interface/matrix.c > [0]PETSC ERROR: KSP_MatMult() line 204 in src/ksp/ksp/impls/cg//datadb/raafat/src/petsc-3.4.3/include/petsc-private/kspimpl.h > [0]PETSC ERROR: KSPSolve_CG() line 132 in src/ksp/ksp/impls/cg/cg.c > [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c > > > Any restrictions here or I misunderstood something? Thanks! > > -- > Brian Yang > U of Houston > > > From brianyang1106 at gmail.com Wed Nov 12 14:46:55 2014 From: brianyang1106 at gmail.com (Brian Yang) Date: Wed, 12 Nov 2014 14:46:55 -0600 Subject: [petsc-users] Nonconforming object sizes ERROR In-Reply-To: References: Message-ID: Thank you for the prompt response. I checked and my usage of KSPSolve is correct. Besides, the assertion line is: if (*mat->rmap->N* != *y->map->N*) SETERRQ2(PETSC_COMM_SELF,PETSC_ERR_ARG_SIZ,"Mat mat,Vec y: global dim %D %D",mat->rmap->N,y->map->N); On Wed, Nov 12, 2014 at 2:29 PM, Barry Smith wrote: > > The calling sequence for KSPSolve is KSPSolve(KSP ksp,Vec b,Vec x), > perhaps you have the b and the x backwards in your call? > > Barry > > > On Nov 12, 2014, at 2:23 PM, Brian Yang wrote: > > > > Hi, > > > > I am solving an over-determined linear system Ax=b and b=0. > > > > A size: 296856 x 14430 > > x (non-zero guess): 14430 > > b (zero): 296856 > > > > All of them are using seq mode to create, so one node will solve this. > However I got this error: > > > > [0]PETSC ERROR: Nonconforming object sizes! > > [0]PETSC ERROR: Mat mat,Vec y: global dim 296856 14430! > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: MatMult() line 2171 in src/mat/interface/matrix.c > > [0]PETSC ERROR: KSP_MatMult() line 204 in > src/ksp/ksp/impls/cg//datadb/raafat/src/petsc-3.4.3/include/petsc-private/kspimpl.h > > [0]PETSC ERROR: KSPSolve_CG() line 132 in src/ksp/ksp/impls/cg/cg.c > > [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c > > > > > > Any restrictions here or I misunderstood something? Thanks! > > > > -- > > Brian Yang > > U of Houston > > > > > > > > -- Brian Yang U of Houston -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Nov 12 15:06:28 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 Nov 2014 15:06:28 -0600 Subject: [petsc-users] Nonconforming object sizes ERROR In-Reply-To: References: Message-ID: <0F1FA588-10D1-4AE9-809D-2B0D7D0B8E6D@mcs.anl.gov> Oh, right CG only works for square matrices. I think you should be using lsqr. See http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPLSQR.html Note if you want to use preconditioning you need to pass for the second matrix the product of the transpose of the matrix times the matrix. There is also KSPCGNE but the docs say, I don't know why, that one should use lsqr for least squares problems. Basically our current support for overdetermined systems is not great and we need help. Barry > On Nov 12, 2014, at 2:46 PM, Brian Yang wrote: > > Thank you for the prompt response. I checked and my usage of KSPSolve is correct. > > Besides, the assertion line is: > if (mat->rmap->N != y->map->N) SETERRQ2(PETSC_COMM_SELF,PETSC_ERR_ARG_SIZ,"Mat mat,Vec y: global dim %D %D",mat->rmap->N,y->map->N); > > On Wed, Nov 12, 2014 at 2:29 PM, Barry Smith wrote: > > The calling sequence for KSPSolve is KSPSolve(KSP ksp,Vec b,Vec x), perhaps you have the b and the x backwards in your call? > > Barry > > > On Nov 12, 2014, at 2:23 PM, Brian Yang wrote: > > > > Hi, > > > > I am solving an over-determined linear system Ax=b and b=0. > > > > A size: 296856 x 14430 > > x (non-zero guess): 14430 > > b (zero): 296856 > > > > All of them are using seq mode to create, so one node will solve this. However I got this error: > > > > [0]PETSC ERROR: Nonconforming object sizes! > > [0]PETSC ERROR: Mat mat,Vec y: global dim 296856 14430! > > > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: MatMult() line 2171 in src/mat/interface/matrix.c > > [0]PETSC ERROR: KSP_MatMult() line 204 in src/ksp/ksp/impls/cg//datadb/raafat/src/petsc-3.4.3/include/petsc-private/kspimpl.h > > [0]PETSC ERROR: KSPSolve_CG() line 132 in src/ksp/ksp/impls/cg/cg.c > > [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c > > > > > > Any restrictions here or I misunderstood something? Thanks! > > > > -- > > Brian Yang > > U of Houston > > > > > > > > > > > -- > Brian Yang > U of Houston > > > From brianyang1106 at gmail.com Wed Nov 12 15:10:39 2014 From: brianyang1106 at gmail.com (Brian Yang) Date: Wed, 12 Nov 2014 15:10:39 -0600 Subject: [petsc-users] Nonconforming object sizes ERROR In-Reply-To: <0F1FA588-10D1-4AE9-809D-2B0D7D0B8E6D@mcs.anl.gov> References: <0F1FA588-10D1-4AE9-809D-2B0D7D0B8E6D@mcs.anl.gov> Message-ID: Wow, it works... LSQR is fine with the problem. Thanks! On Wed, Nov 12, 2014 at 3:06 PM, Barry Smith wrote: > > Oh, right CG only works for square matrices. > > I think you should be using lsqr. See > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPLSQR.html > Note if you want to use preconditioning you need to pass for the second > matrix the product of the transpose of the matrix times the matrix. > > There is also KSPCGNE but the docs say, I don't know why, that one > should use lsqr for least squares problems. > > Basically our current support for overdetermined systems is not great > and we need help. > > Barry > > > On Nov 12, 2014, at 2:46 PM, Brian Yang wrote: > > > > Thank you for the prompt response. I checked and my usage of KSPSolve is > correct. > > > > Besides, the assertion line is: > > if (mat->rmap->N != y->map->N) > SETERRQ2(PETSC_COMM_SELF,PETSC_ERR_ARG_SIZ,"Mat mat,Vec y: global dim %D > %D",mat->rmap->N,y->map->N); > > > > On Wed, Nov 12, 2014 at 2:29 PM, Barry Smith wrote: > > > > The calling sequence for KSPSolve is KSPSolve(KSP ksp,Vec b,Vec x), > perhaps you have the b and the x backwards in your call? > > > > Barry > > > > > On Nov 12, 2014, at 2:23 PM, Brian Yang > wrote: > > > > > > Hi, > > > > > > I am solving an over-determined linear system Ax=b and b=0. > > > > > > A size: 296856 x 14430 > > > x (non-zero guess): 14430 > > > b (zero): 296856 > > > > > > All of them are using seq mode to create, so one node will solve this. > However I got this error: > > > > > > [0]PETSC ERROR: Nonconforming object sizes! > > > [0]PETSC ERROR: Mat mat,Vec y: global dim 296856 14430! > > > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: MatMult() line 2171 in src/mat/interface/matrix.c > > > [0]PETSC ERROR: KSP_MatMult() line 204 in > src/ksp/ksp/impls/cg//datadb/raafat/src/petsc-3.4.3/include/petsc-private/kspimpl.h > > > [0]PETSC ERROR: KSPSolve_CG() line 132 in src/ksp/ksp/impls/cg/cg.c > > > [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c > > > > > > > > > Any restrictions here or I misunderstood something? Thanks! > > > > > > -- > > > Brian Yang > > > U of Houston > > > > > > > > > > > > > > > > > > > -- > > Brian Yang > > U of Houston > > > > > > > > -- Brian Yang U of Houston -------------- next part -------------- An HTML attachment was scrubbed... URL: From thiele at itwm.fraunhofer.de Wed Nov 12 15:16:20 2014 From: thiele at itwm.fraunhofer.de (thiele at itwm.fraunhofer.de) Date: Wed, 12 Nov 2014 22:16:20 +0100 Subject: [petsc-users] Extract subsection of DMDA In-Reply-To: References: <54638322.5040407@itwm.fraunhofer.de> Message-ID: <131cb7d93e82d9450667b43a6b1dc676.squirrel@webmail.itwm.fhg.de> Thanks for your reply. I thought about changing the constructor parameters, too, but I found it rather complicated and it exceeds my current PETSc knowledge. So maybe you are right and I should use the visualization tool if there is no other way to do it with PETSc. Regards, Christopher > On Wed, Nov 12, 2014 at 9:56 AM, Christopher Thiele < > christopher.thiele at itwm.fraunhofer.de> wrote: > >> Hello, >> >> I have a 3d DMDA and a vector created by DMCreateGlobalVector. I want to >> visualize and therefore export a subset of the values, i.e. I want to >> "remove" some points at the boundary. For example, if the DMDA >> represents the domain [0,1]^3, I want to extract [0.25,0.75]^3. >> I already tried to create another (smaller) DMDA, use DMDAVecGetArray on >> both of them and then copy the values. The problem is that the vectors >> are distributed differently, so this approach leads to segfaults. >> Is there another way to do this? >> > > You can force the same layout using lx, ly, lz in the constructor. > However, > this kind of clipping > is usually much easier in a visualization program. > > Thanks, > > Matt > > >> Regards, >> Christopher >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > From sghosh2012 at gatech.edu Wed Nov 12 15:48:03 2014 From: sghosh2012 at gatech.edu (Ghosh, Swarnava) Date: Wed, 12 Nov 2014 16:48:03 -0500 (EST) Subject: [petsc-users] Inserting nonlocal values in DMDA vector In-Reply-To: <444432108.31198113.1415827346609.JavaMail.root@mail.gatech.edu> Message-ID: <697045755.31212106.1415828883221.JavaMail.root@mail.gatech.edu> Hello, I have a DMDA vector and I need to set values in locations which are in different processors. I am going over the mesh points in z,y and x directions so I know the k,j,i indices of every point. However the 3D array obtained using DMDAVecGetArray will not let me access the points which are in other processors. Is there some function like "VecSetValuesStencil" similar to MatSetValuesStencil which will insert values at locations which are in other processors using k,j,i index? OR Is there any function that maps the k,j,i index to a linear index so I can use VecSetValues? Regards, Swarnava -- Swarnava Ghosh PhD Candidate, Structural Engineering, Mechanics and Materials School of Civil and Environmental Engineering Georgia Institute of Technology Atlanta, GA 30332 From bsmith at mcs.anl.gov Wed Nov 12 16:05:42 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 Nov 2014 16:05:42 -0600 Subject: [petsc-users] Inserting nonlocal values in DMDA vector In-Reply-To: <697045755.31212106.1415828883221.JavaMail.root@mail.gatech.edu> References: <697045755.31212106.1415828883221.JavaMail.root@mail.gatech.edu> Message-ID: > On Nov 12, 2014, at 3:48 PM, Ghosh, Swarnava wrote: > > > Hello, > > I have a DMDA vector and I need to set values in locations which are in different processors. I am going over the mesh points in z,y and x directions so I know the k,j,i indices of every point. However the 3D array obtained using DMDAVecGetArray will not let me access the points which are in other processors. I think you really want each process to handle only its own values and not have some other processor stick in those values. > > Is there some function like "VecSetValuesStencil" similar to MatSetValuesStencil which will insert values at locations which are in other processors using k,j,i index? OR Is there any function that maps the k,j,i index to a linear index so I can use VecSetValues? Note that MatSetValuesStencil() only works for matrix entries that arise from the ghosted region of the processors domain, you cannot stick values into any old locations with MatSetValuesStencil() it will generate an error. You can use DMDAGetAO() then if id[] contains the numbering of your points in the global natural ordering use AOApplicationToPetsc() to map them to the PETSc ordering and then us VecSetValues(). Try it on a small problem first to make sure the correct mapping is being done. Barry > > Regards, > Swarnava > -- > Swarnava Ghosh > PhD Candidate, > Structural Engineering, Mechanics and Materials > School of Civil and Environmental Engineering > Georgia Institute of Technology > Atlanta, GA 30332 From lb2653 at columbia.edu Wed Nov 12 16:24:26 2014 From: lb2653 at columbia.edu (Luc Berger-Vergiat) Date: Wed, 12 Nov 2014 17:24:26 -0500 Subject: [petsc-users] passing solver options in fieldsplit In-Reply-To: References: <546229AF.1010808@columbi.edu> Message-ID: <5463DE1A.7000308@columbi.edu> Thanks, the option -help was quite useful and I got things to work fine with icntl_14 50. Best, Luc On 11/11/2014 11:07 AM, Hong wrote: > Luc: > Run your code with option '-help |grep mumps', then you'll see what > prefix should be used in your case with the mumps option > '-mat_mumps_icntl_14 30'. > You may try even larger icntl_14. > > Hong > > Hi, I am using Petsc to solver a multiphysics problem and I have > the following issue. > I partition my problem by declaring two fields: > > -ksp_type gmres -pc_type fieldsplit -pc_fieldsplit_type schur > -pc_fieldsplit_schur_factorization_type full > -pc_fieldsplit_schur_precondition selfp > -pc_fieldsplit_0_fields 2,3 -pc_fieldsplit_1_fields 0,1 > > I want to solve the matrix representing field 0 with mumps so I > pass the following following arguments: > > -fieldsplit_0_ksp_type preonly -fieldsplit_0_pc_type lu > -fieldsplit_0_pc_factor_mat_solver_package mumps > > When I do this I get an error from mumps: INFO(1)=-9, > INFO(2)=12532. This means that mumps main internal real workarray > is too small and 12532 are missing. To try to mitigate this I need > to want to set mumps ICNTL(14)=30 (by default it is 20). > Reading Petsc documentation I find that I have to pass the > following argument to my program: > > -mat_mumps_icntl_14 30 > > which does work fine when I work without fieldsplit but not when I > use field split. > I also tried: > > -fieldsplit_0_mat_mumps_icntl_14 30 > > which does not work any better. > Any idea how I should pass the icntl_14 information to mumps in > this case? > > -- > Best, > Luc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stm8086 at yahoo.com Wed Nov 12 18:48:36 2014 From: stm8086 at yahoo.com (Steena M) Date: Wed, 12 Nov 2014 16:48:36 -0800 Subject: [petsc-users] Setting up MPIBAIJ for MatMult In-Reply-To: <3469D6F4-7E56-41E6-8E86-237F7FFE2791@mcs.anl.gov> Message-ID: <1415839716.75219.YahooMailBasic@web125403.mail.ne1.yahoo.com> Thank you, Barry. That took care of most of the issues. Two more things: 1) The code runs (MatMult executes) for block size =1. For block sizes > 1. For instance, block size = 3, throws: PETSC ERROR: Arguments are incompatible! [0]PETSC ERROR: Cannot change block size 3 to 1! Matrix setup: ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,filein,FILE_MODE_READ,&fd); ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetType(A, MATMPIBAIJ); ierr = MatSetBlockSize(A, bs); CHKERRQ(ierr); ierr = MatLoad(A,fd);CHKERRQ(ierr); ierr = PetscViewerDestroy(&fd);CHKERRQ(ierr); 2) I'm not sure if my vector setup is ideal. Should I be using VecSetValues instead? ierr = MatGetSize(A,&m,&n); ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); ierr = VecSetType(x,VECMPI); ierr = VecSetSizes(x,PETSC_DECIDE,m);CHKERRQ(ierr); ierr = VecSetFromOptions(x);CHKERRQ(ierr); -------------------------------------------- On Wed, 11/12/14, Barry Smith wrote: Subject: Re: [petsc-users] Setting up MPIBAIJ for MatMult To: "Steena M" Cc: petsc-users at mcs.anl.gov Date: Wednesday, November 12, 2014, 11:44 AM ? 1) remove all the "preload" stuff ? 2) MatLoad doesn't require or use the Mat preallocation stuff so remove all of that ???3) MatLoad determines the global matrix size from the binary file so do not set that before calling MatLoad ???4) All processes in the comm (in your case MPI_COMM_WORLD) need to call MatLoad ???5) You can call MatSetBlockSize() to set the block size before MatLoad() ???6) You do not need to call MatAssemblyBegin/End after a MatLoad() it would do nothing > On Nov 12, 2014, at 1:32 PM, Steena M wrote: > > I am trying to read in a sparse matrix file in binary format (.dat file) to set it up as MPIBAIJ for MPIMatMult. I don't think I'm doing the file reading or the matrix and vector setup correctly. Should the file loading, matrix setup, and loading be done only on rank 0 together with the vector setup?? After looking through several examples, this is the structure of my code right now: > > Vec? ? ? ? ? ? x,y;? > Mat? ? ? ? ? ? A;? > PetscViewer? ???fd; > int rank, global_row_size, global_col_size,bs; > PetscBool? ? ???PetscPreLoad = PETSC_FALSE; > char? ? ? ? ???filein[PETSC_MAX_PATH_LEN]? /*binary .dat matrix file */ > PetscScalar one = 1.0; > > >? /* Reading relevant block size from ENV */ >? char* bs_env; >? bs_env = getenv ("BLOCK_SIZE_ENV"); >? bs = (PetscInt)atoi(bs_env); > > /*Reading in matrix row/cols size from ENV. Matrices are always square: global_col_size=global_row_size*/ > >? char* m_env; >? m_env = getenv ("MATRIX_ROWS_ENV"); >? global_row_size = (PetscInt)atoi(m_env); >? global_col_size = global _row_size; > >? PetscInitialize(&argc,&args,(char *)0,help); >? ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); > > /*Matrix reading, setup, assembly*/ > >? if (flg) PetscPreLoad = PETSC_TRUE; >? PetscPreLoadBegin(PetscPreLoad,"Load"); >? ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,filein[PetscPreLoadIt],FILE_MODE_READ,&fd);CHKERRQ(ierr); > > >? ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); >? ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,global_row_size,global_col_size);CHKERRQ(ierr); >? ierr = MatSetFromOptions(A);CHKERRQ(ierr); >? ierr = MatSetType(A, MATMPIBAIJ); >? ierr = MatMPIBAIJSetPreallocation(A,bs,5,PETSC_NULL,5,PETSC_NULL);CHKERRQ(ierr); >? ierr = MatSeqBAIJSetPreallocation(A,bs,5,PETSC_NULL);CHKERRQ(ierr); > >? ierr = PetscLogStageRegister("Assembly", &stage);CHKERRQ(ierr); >? ierr = PetscLogStagePush(stage);CHKERRQ(ierr); > > >? if (rank==0) >? { >? ierr = MatLoad(A,fd);CHKERRQ(ierr); > >? } > > ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = PetscLogStagePop();CHKERRQ(ierr); > ierr = PetscViewerDestroy(&fd);CHKERRQ(ierr); > > /*Trying two types of vector assembly*/ > > /* Vector setup Attempt 1*/ > > ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); >? ierr = VecSetSizes(x,PETSC_DECIDE,global_row_size);CHKERRQ(ierr); >? ierr = VecSetFromOptions(x);CHKERRQ(ierr); >? ierr = VecDuplicate(x,&y);CHKERRQ(ierr); >? ierr = VecSetFromOptions(y);CHKERRQ(ierr); >? ierr = VecSet(x,one);CHKERRQ(ierr); > ierr = VecSet(y,one); CHKERRQ(ierr); > > /*Vector setup Attempt 2*/ > > ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); > ierr = VecGetSize(x,&global_row_size);CHKERRQ(ierr); > > if (rank==0) >? { > >? for (i = 0;? i? { >? ierr = VecSetValues(x,1,&i,one,INSERT_VALUES); >? CHKERRQ(ierr); >? } > >? } >? ierr = VecDuplicate(x,&y);CHKERRQ(ierr); >? ierr = VecSetFromOptions(y);CHKERRQ(ierr); > >? ierr = VecAssemblyBegin(x); CHKERRQ(ierr); >? ierr = VecAssemblyEnd(x); CHKERRQ(ierr); > > > > /* SpMV*/ > ierr = MatMult(A,x,y);CHKERRQ(ierr); > >? ierr = VecDestroy(&x);CHKERRQ(ierr); >? ierr = VecDestroy(&y);CHKERRQ(ierr); >? ierr = MatDestroy(&A);CHKERRQ(ierr); > > > > From jed at jedbrown.org Wed Nov 12 19:12:13 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 13 Nov 2014 04:12:13 +0300 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: <546342D4.7080508@imperial.ac.uk> References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> <87ppcus6na.fsf@jedbrown.org> <5FDBB21C-A76B-48F2-B89C-02CD1DB2FAB2@imperial.ac.uk> <87zjbxrcjm.fsf@jedbrown.org> <546342D4.7080508@imperial.ac.uk> Message-ID: <87389ngaz6.fsf@jedbrown.org> Lawrence Mitchell writes: > So is the problem that the nullspace belongs to the appropriate > operator, but at the point you tell the solver, you don't necessarily > have all the operators to hand? Moreover, depending on the PC choice, > the nullspaces of the operator blocks may or may not be relevant? Given the system [A B; C D], the null space of D has nothing to do with the null space of S = D - C A^{-1} B. In some important cases, D is the zero matrix, for example. Using MatSetNullSpace to set a null space on D (even if you have it in hand) is lying and may cause the wrong thing to happen if you use a different solver configuration (e.g., a relaxation split). Attaching a null space to an IS is damn perverse because an index set is a bag of integers and you're labeling it with information about a Mat that is the result of some other algorithm using lots more information than just the IS (and done in a non-unique way). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Nov 12 19:35:30 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 Nov 2014 19:35:30 -0600 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: <87389ngaz6.fsf@jedbrown.org> References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> <87ppcus6na.fsf@jedbrown.org> <5FDBB21C-A76B-48F2-B89C-02CD1DB2FAB2@imperial.ac.uk> <87zjbxrcjm.fsf@jedbrown.org> <546342D4.7080508@imperial.ac.uk> <87389ngaz6.fsf@jedbrown.org> Message-ID: I agree with what Jed says. One could argue that if the matrix was represented as a MatNest, the null space of the Schur complement is a property of the MatNest object and hence could/should be something that is attached to the matrix. Now if the Mat happens not to be a MatNest, the null space of the Schur complement is still a property of the Mat (and the index sets that define the part you are Schur complementing on), so again the null space could be attached to matrix (plus the index sets of the parts you are Schur complementing on). Perhaps we need a generic way of attaching null spaces to parts of matrices and to parts of matrices in different basis (which is what a Schur complement is). So (and this is likely overly simplistic) one could supply a null space, an index set, and a flag indicating either "part of the matrix" or the "Schur complement of the matrix with respect to an index set". These provided null spaces could then be used by the preconditioners (like Fieldsplit) that manipulate parts of the matrix or Schur complement. Barry > On Nov 12, 2014, at 7:12 PM, Jed Brown wrote: > > Lawrence Mitchell writes: >> So is the problem that the nullspace belongs to the appropriate >> operator, but at the point you tell the solver, you don't necessarily >> have all the operators to hand? Moreover, depending on the PC choice, >> the nullspaces of the operator blocks may or may not be relevant? > > Given the system [A B; C D], the null space of D has nothing to do with > the null space of S = D - C A^{-1} B. In some important cases, D is the > zero matrix, for example. Using MatSetNullSpace to set a null space on > D (even if you have it in hand) is lying and may cause the wrong thing > to happen if you use a different solver configuration (e.g., a > relaxation split). > > Attaching a null space to an IS is damn perverse because an index set is > a bag of integers and you're labeling it with information about a Mat > that is the result of some other algorithm using lots more information > than just the IS (and done in a non-unique way). From jed at jedbrown.org Wed Nov 12 19:49:05 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 13 Nov 2014 04:49:05 +0300 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> <87ppcus6na.fsf@jedbrown.org> <5FDBB21C-A76B-48F2-B89C-02CD1DB2FAB2@imperial.ac.uk> <87zjbxrcjm.fsf@jedbrown.org> <546342D4.7080508@imperial.ac.uk> <87389ngaz6.fsf@jedbrown.org> Message-ID: <87zjbveupa.fsf@jedbrown.org> Barry Smith writes: > Perhaps we need a generic way of attaching null spaces to parts of > matrices and to parts of matrices in different basis (which is what > a Schur complement is). So (and this is likely overly simplistic) > one could supply a null space, an index set, and a flag indicating > either "part of the matrix" or the "Schur complement of the matrix > with respect to an index set". These provided null spaces could > then be used by the preconditioners (like Fieldsplit) that > manipulate parts of the matrix or Schur complement. Yes, this is conceptually right, but it looks to me like it gets hairy for recursive splitting. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Nov 12 19:55:50 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 Nov 2014 19:55:50 -0600 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: <87zjbveupa.fsf@jedbrown.org> References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> <87ppcus6na.fsf@jedbrown.org> <5FDBB21C-A76B-48F2-B89C-02CD1DB2FAB2@imperial.ac.uk> <87zjbxrcjm.fsf@jedbrown.org> <546342D4.7080508@imperial.ac.uk> <87389ngaz6.fsf@jedbrown.org> <87zjbveupa.fsf@jedbrown.org> Message-ID: <6957E146-D9A9-4968-B8C0-81B2CA084351@mcs.anl.gov> > On Nov 12, 2014, at 7:49 PM, Jed Brown wrote: > > Barry Smith writes: >> Perhaps we need a generic way of attaching null spaces to parts of >> matrices and to parts of matrices in different basis (which is what >> a Schur complement is). So (and this is likely overly simplistic) >> one could supply a null space, an index set, and a flag indicating >> either "part of the matrix" or the "Schur complement of the matrix >> with respect to an index set". These provided null spaces could >> then be used by the preconditioners (like Fieldsplit) that >> manipulate parts of the matrix or Schur complement. > > Yes, this is conceptually right, but it looks to me like it gets hairy > for recursive splitting. Indeed it could. Perhaps we could start by unifying the concept of a sub matrix and a Schur complement of the same part of the matrix? I'm not sure how to do that. Barry From jed at jedbrown.org Wed Nov 12 20:10:18 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 13 Nov 2014 05:10:18 +0300 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: <6957E146-D9A9-4968-B8C0-81B2CA084351@mcs.anl.gov> References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> <87ppcus6na.fsf@jedbrown.org> <5FDBB21C-A76B-48F2-B89C-02CD1DB2FAB2@imperial.ac.uk> <87zjbxrcjm.fsf@jedbrown.org> <546342D4.7080508@imperial.ac.uk> <87389ngaz6.fsf@jedbrown.org> <87zjbveupa.fsf@jedbrown.org> <6957E146-D9A9-4968-B8C0-81B2CA084351@mcs.anl.gov> Message-ID: <87vbmjetpx.fsf@jedbrown.org> Barry Smith writes: > Indeed it could. Perhaps we could start by unifying the concept of a > sub matrix and a Schur complement of the same part of the matrix? > I'm not sure how to do that. We can certainly define an interface that works this way, but I'm not sure it's practical. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From karpeev at mcs.anl.gov Wed Nov 12 20:15:19 2014 From: karpeev at mcs.anl.gov (Dmitry Karpeyev) Date: Thu, 13 Nov 2014 02:15:19 +0000 Subject: [petsc-users] Nullspaces for schur complement PCs References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> <87ppcus6na.fsf@jedbrown.org> <5FDBB21C-A76B-48F2-B89C-02CD1DB2FAB2@imperial.ac.uk> <87zjbxrcjm.fsf@jedbrown.org> <546342D4.7080508@imperial.ac.uk> <87389ngaz6.fsf@jedbrown.org> <87zjbveupa.fsf@jedbrown.org> Message-ID: On Wed Nov 12 2014 at 7:49:28 PM Jed Brown wrote: > Barry Smith writes: > > Perhaps we need a generic way of attaching null spaces to parts of > > matrices and to parts of matrices in different basis (which is what > > a Schur complement is). So (and this is likely overly simplistic) > > one could supply a null space, an index set, and a flag indicating > > either "part of the matrix" or the "Schur complement of the matrix > > with respect to an index set". These provided null spaces could > > then be used by the preconditioners (like Fieldsplit) that > > manipulate parts of the matrix or Schur complement. > > Yes, this is conceptually right, but it looks to me like it gets hairy > for recursive splitting. > I don't think we can anticipate every possible solver configuration and hang enough additional information on the objects we passed down a deeply-nested solver hierarchy. Even if we could, we could end up with very long lists of flags and objects, and complicated case statements to sift through the flags attached to the Mat. Also, what if the split in question is a couple of levels down the solver hierarchy (not at all out of the question for many multiphysics problems)? The user doesn't get to create that Mat or PC and, therefore, doesn't have an opportunity to pass down the necessary extra information. I think the problem-specific solver configuration can be carried out by an additional user-specified callback (attached, e.g., using PCSetUserSetUp()) called from PCSetUp(). The problem of propagating it down the hierarchy can be solved by using DMs (even DMShell would do, as it carries an ApplicationCtx) and a consistent use of DMFieldSplitRestrictHook (by analogy with other restriction hooks, like DMSubDomainRestrictHook). DMFieldSplitRestrict() would be called when creating each split and one of the restriction hooks would transfer the necessary contexts. This is a bit heavyweight, but flexible and logically consistent, it seems to me, and no worse that what has to be done to propagate things like DMSNES/DMKSP through the MG hierarchy. (N)ASM uses something similar, too. Dmitry. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Nov 12 20:24:18 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 Nov 2014 20:24:18 -0600 Subject: [petsc-users] Setting up MPIBAIJ for MatMult In-Reply-To: <1415839716.75219.YahooMailBasic@web125403.mail.ne1.yahoo.com> References: <1415839716.75219.YahooMailBasic@web125403.mail.ne1.yahoo.com> Message-ID: <99419E8B-D018-41BE-B3BD-618E0A745F92@mcs.anl.gov> Ok, for now run with the additional command line option -matload_block_size 3 and I'll fix the code to take the size in the matrix. Barry > On Nov 12, 2014, at 6:48 PM, Steena M wrote: > > Thank you, Barry. That took care of most of the issues. Two more things: > > 1) The code runs (MatMult executes) for block size =1. For block sizes > 1. For instance, block size = 3, throws: > PETSC ERROR: Arguments are incompatible! > [0]PETSC ERROR: Cannot change block size 3 to 1! > > Matrix setup: > > ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,filein,FILE_MODE_READ,&fd); > ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); > ierr = MatSetType(A, MATMPIBAIJ); > ierr = MatSetBlockSize(A, bs); CHKERRQ(ierr); > ierr = MatLoad(A,fd);CHKERRQ(ierr); > ierr = PetscViewerDestroy(&fd);CHKERRQ(ierr); > > > 2) I'm not sure if my vector setup is ideal. Should I be using VecSetValues instead? > > ierr = MatGetSize(A,&m,&n); > > ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); > ierr = VecSetType(x,VECMPI); > ierr = VecSetSizes(x,PETSC_DECIDE,m);CHKERRQ(ierr); > ierr = VecSetFromOptions(x);CHKERRQ(ierr); > > -------------------------------------------- > On Wed, 11/12/14, Barry Smith wrote: > > Subject: Re: [petsc-users] Setting up MPIBAIJ for MatMult > To: "Steena M" > Cc: petsc-users at mcs.anl.gov > Date: Wednesday, November 12, 2014, 11:44 AM > > > 1) > remove all the "preload" stuff > > 2) MatLoad doesn't require or use the > Mat preallocation stuff so remove all of that > > 3) MatLoad > determines the global matrix size from the binary file so do > not set that before calling MatLoad > > 4) All processes in the comm (in > your case MPI_COMM_WORLD) need to call MatLoad > > 5) You can call > MatSetBlockSize() to set the block size before MatLoad() > > 6) You do not > need to call MatAssemblyBegin/End after a MatLoad() it would > do nothing > >> On Nov 12, 2014, at > 1:32 PM, Steena M > wrote: >> >> I am > trying to read in a sparse matrix file in binary format > (.dat file) to set it up as MPIBAIJ for MPIMatMult. I > don't think I'm doing the file reading or the matrix > and vector setup correctly. Should the file loading, matrix > setup, and loading be done only on rank 0 together with the > vector setup? After looking through several examples, this > is the structure of my code right now: >> > >> Vec x,y; >> Mat A; >> PetscViewer fd; >> int rank, global_row_size, > global_col_size,bs; >> PetscBool > PetscPreLoad = PETSC_FALSE; >> char > filein[PETSC_MAX_PATH_LEN] /*binary .dat > matrix file */ >> PetscScalar one = > 1.0; >> >> >> /* Reading relevant block size from ENV > */ >> char* bs_env; >> bs_env = getenv > ("BLOCK_SIZE_ENV"); >> bs = > (PetscInt)atoi(bs_env); >> >> /*Reading in matrix row/cols size from > ENV. Matrices are always square: > global_col_size=global_row_size*/ >> >> char* m_env; >> > m_env = getenv ("MATRIX_ROWS_ENV"); >> global_row_size = > (PetscInt)atoi(m_env); >> > global_col_size = global _row_size; >> >> > PetscInitialize(&argc,&args,(char *)0,help); >> ierr = > MPI_Comm_rank(PETSC_COMM_WORLD,&rank); >> >> /*Matrix reading, > setup, assembly*/ >> >> if (flg) PetscPreLoad = PETSC_TRUE; >> > PetscPreLoadBegin(PetscPreLoad,"Load"); >> ierr = > PetscViewerBinaryOpen(PETSC_COMM_WORLD,filein[PetscPreLoadIt],FILE_MODE_READ,&fd);CHKERRQ(ierr); >> >> >> ierr = > MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); >> ierr = > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,global_row_size,global_col_size);CHKERRQ(ierr); >> ierr = > MatSetFromOptions(A);CHKERRQ(ierr); >> > ierr = MatSetType(A, MATMPIBAIJ); >> > ierr = > MatMPIBAIJSetPreallocation(A,bs,5,PETSC_NULL,5,PETSC_NULL);CHKERRQ(ierr); >> ierr = > MatSeqBAIJSetPreallocation(A,bs,5,PETSC_NULL);CHKERRQ(ierr); >> >> ierr = > PetscLogStageRegister("Assembly", > &stage);CHKERRQ(ierr); >> ierr = > PetscLogStagePush(stage);CHKERRQ(ierr); >> > >> >> if > (rank==0) >> { >> > ierr = MatLoad(A,fd);CHKERRQ(ierr); >> > >> } >> >> ierr = > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = > PetscLogStagePop();CHKERRQ(ierr); >> ierr > = PetscViewerDestroy(&fd);CHKERRQ(ierr); >> >> /*Trying two types > of vector assembly*/ >> >> /* Vector setup Attempt 1*/ >> >> ierr = > VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); >> ierr = > VecSetSizes(x,PETSC_DECIDE,global_row_size);CHKERRQ(ierr); >> ierr = > VecSetFromOptions(x);CHKERRQ(ierr); >> > ierr = VecDuplicate(x,&y);CHKERRQ(ierr); >> ierr = > VecSetFromOptions(y);CHKERRQ(ierr); >> > ierr = VecSet(x,one);CHKERRQ(ierr); >> > ierr = VecSet(y,one); CHKERRQ(ierr); >> > >> /*Vector setup Attempt 2*/ >> >> ierr = > VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); >> ierr = > VecGetSize(x,&global_row_size);CHKERRQ(ierr); >> >> if (rank==0) >> { >> >> for (i = 0; i i++) >> { >> ierr = > VecSetValues(x,1,&i,one,INSERT_VALUES); >> CHKERRQ(ierr); >> > } >> >> } >> ierr = > VecDuplicate(x,&y);CHKERRQ(ierr); >> > ierr = VecSetFromOptions(y);CHKERRQ(ierr); >> >> ierr = > VecAssemblyBegin(x); CHKERRQ(ierr); >> > ierr = VecAssemblyEnd(x); CHKERRQ(ierr); >> >> >> >> /* SpMV*/ >> ierr = MatMult(A,x,y);CHKERRQ(ierr); >> >> ierr = > VecDestroy(&x);CHKERRQ(ierr); >> > ierr = VecDestroy(&y);CHKERRQ(ierr); >> ierr = > MatDestroy(&A);CHKERRQ(ierr); >> >> >> >> > From jed at jedbrown.org Wed Nov 12 20:39:13 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 13 Nov 2014 05:39:13 +0300 Subject: [petsc-users] Nullspaces for schur complement PCs In-Reply-To: References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> <87ppcus6na.fsf@jedbrown.org> <5FDBB21C-A76B-48F2-B89C-02CD1DB2FAB2@imperial.ac.uk> <87zjbxrcjm.fsf@jedbrown.org> <546342D4.7080508@imperial.ac.uk> <87389ngaz6.fsf@jedbrown.org> <87zjbveupa.fsf@jedbrown.org> Message-ID: <87ppcresdq.fsf@jedbrown.org> Dmitry Karpeyev writes: > I think the problem-specific solver configuration can be carried out by an > additional user-specified callback (attached, e.g., using PCSetUserSetUp()) > called from PCSetUp(). > The problem of propagating it down the hierarchy can be solved by using DMs > (even DMShell would do, as it carries an ApplicationCtx) and a consistent > use of DMFieldSplitRestrictHook (by analogy with other restriction hooks, > like DMSubDomainRestrictHook). DMFieldSplitRestrict() would be called when > creating each split and one of the restriction hooks would transfer the > necessary contexts. > > This is a bit heavyweight, but flexible and logically consistent, it seems > to me, and no worse that what has to be done to propagate things like > DMSNES/DMKSP through the MG hierarchy. (N)ASM uses something similar, too. Previously discussed. http://lists.mcs.anl.gov/pipermail/petsc-dev/2012-August/009291.html It makes things more dependent on DM, but that may be less bad than the alternatives. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From karpeev at mcs.anl.gov Wed Nov 12 23:07:20 2014 From: karpeev at mcs.anl.gov (Dmitry Karpeyev) Date: Thu, 13 Nov 2014 05:07:20 +0000 Subject: [petsc-users] Nullspaces for schur complement PCs References: <21DB12C2-95FC-41A2-979C-E777B4AEFFCE@imperial.ac.uk> <87ppcus6na.fsf@jedbrown.org> <5FDBB21C-A76B-48F2-B89C-02CD1DB2FAB2@imperial.ac.uk> <87zjbxrcjm.fsf@jedbrown.org> <546342D4.7080508@imperial.ac.uk> <87389ngaz6.fsf@jedbrown.org> <87zjbveupa.fsf@jedbrown.org> <87ppcresdq.fsf@jedbrown.org> Message-ID: I'm actually not proposing putting extra intelligence into the DM, but into the ApplicationCtx and the associated callbacks that the user can write. The DM is used to propagate this information to the deeper levels of a nested solver hierarchy by "restricting" the relevant ctx/callbacks to the split's subdm. Restriction is a standard PETSc way of propagating information through the solver hierarchy (e.g., MG, (N)ASM), but it hasn't been used in the FieldSplit context. Currently even a user-defined preconditioner for a Schur complement that sits at a deeper level in the hierarchy cannot be set very easily, especially when the solvers are configured dynamically. Yes, this does place a greater emphasis on DM, but not overly so, in my view. On Wed Nov 12 2014 at 8:39:23 PM Jed Brown wrote: > Dmitry Karpeyev writes: > > > I think the problem-specific solver configuration can be carried out by > an > > additional user-specified callback (attached, e.g., using > PCSetUserSetUp()) > > called from PCSetUp(). > > The problem of propagating it down the hierarchy can be solved by using > DMs > > (even DMShell would do, as it carries an ApplicationCtx) and a consistent > > use of DMFieldSplitRestrictHook (by analogy with other restriction hooks, > > like DMSubDomainRestrictHook). DMFieldSplitRestrict() would be called > when > > creating each split and one of the restriction hooks would transfer the > > necessary contexts. > > > > This is a bit heavyweight, but flexible and logically consistent, it > seems > > to me, and no worse that what has to be done to propagate things like > > DMSNES/DMKSP through the MG hierarchy. (N)ASM uses something similar, > too. > > Previously discussed. > > http://lists.mcs.anl.gov/pipermail/petsc-dev/2012-August/009291.html > > It makes things more dependent on DM, but that may be less bad than the > alternatives. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leoni.massimiliano1 at gmail.com Thu Nov 13 05:39:21 2014 From: leoni.massimiliano1 at gmail.com (Massimiliano Leoni) Date: Thu, 13 Nov 2014 12:39:21 +0100 Subject: [petsc-users] Roles of ksp_type and pc_type Message-ID: <2594845.ya19CE4v59@debianxps> Hi petsc-users, I am relatively new to PETSc and so a few things that you'd call obvious still puzzle me. I hope someone can help me clarify this. Consider the system Ax = b and KSPSetOperators(ksp,A,A,...); now, please correct me wherever I'm wrong: [1] if I set -ksp_type preonly, -pc_type lu it solves Ax = b with direct LU [2] if I set -ksp_type preonly, -pc_type hypre it solves Px = b where P is the AMG preconditioner of A build by HYPRE ? Question: how does PETSc solve this? [3] if I set -ksp_type gmres, -pc_type hypre it solves Ax = b as P^{-1}Ax = P^{-1}b with gmres, where P is again the AMG prec build by HYPRE. [4] suppose I have the classic Stokes matrix A = [[C,B^T],[B,O]] and want to use the block preconditioner P = [[C,O],[O,Mp/nu]]. What I'm doing now is KSPSetOperators(ksp,A,P,...); set ksp_type gmres, pc_type fieldsplit, inner ksps gmres and inner pcs hypre, so I expect PETSc to solve Pz = r by blocks, C z_u = r_u first, as in [3] Mp/nu z_p = r_p later, again as in [3] Is this what is actually happening? [5] what happens instead if I set, in the same situation, ksp_type preonly, pc_type fieldsplit, inner ksps gmres and inner pcs ? I expect PETSc to solve Px = b with a block method as before. [This doesn't actually solve Stokes]. I'm sorry for the many questions, I am trying to go through the User Manual and will be glad if anyone points me to further resources. Thanks in advance for any help, Massimiliano From knepley at gmail.com Thu Nov 13 05:59:17 2014 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 13 Nov 2014 05:59:17 -0600 Subject: [petsc-users] Roles of ksp_type and pc_type In-Reply-To: <2594845.ya19CE4v59@debianxps> References: <2594845.ya19CE4v59@debianxps> Message-ID: On Thu, Nov 13, 2014 at 5:39 AM, Massimiliano Leoni < leoni.massimiliano1 at gmail.com> wrote: > Hi petsc-users, > I am relatively new to PETSc and so a few things that you'd call obvious > still puzzle me. I hope someone can help me clarify this. > > Consider the system Ax = b and > KSPSetOperators(ksp,A,A,...); > > now, please correct me wherever I'm wrong: > [1] if I set -ksp_type preonly, -pc_type lu it solves Ax = b with direct > LU > > [2] if I set -ksp_type preonly, -pc_type hypre it solves Px = b where P > is > the AMG preconditioner of A build by HYPRE > ? Question: how does PETSc solve this? > We call Hypre directly to apply P^{-1}. > > [3] if I set -ksp_type gmres, -pc_type hypre it solves Ax = b as > P^{-1}Ax = P^{-1}b with gmres, where P is again the AMG prec build > by > HYPRE. > > [4] suppose I have the classic Stokes matrix A = [[C,B^T],[B,O]] and want > to > use the block preconditioner P = [[C,O],[O,Mp/nu]]. What I'm doing > now is > KSPSetOperators(ksp,A,P,...); > set ksp_type gmres, pc_type fieldsplit, inner ksps gmres and inner > pcs > hypre, so I expect PETSc to solve Pz = r by blocks, > C z_u = r_u first, as in [3] > Mp/nu z_p = r_p later, again as in [3] > Is this what is actually happening? > It should be. Send -ksp_view and we can verify. You should use -pc_fieldsplit_schur_factorization_type diag > [5] what happens instead if I set, in the same situation, > ksp_type preonly, pc_type fieldsplit, inner ksps gmres and inner > pcs ? > I expect PETSc to solve Px = b with a block method as before. [This > doesn't actually solve Stokes]. > Yes. Thanks, Matt > I'm sorry for the many questions, I am trying to go through the User Manual > and will be glad if anyone points me to further resources. > > Thanks in advance for any help, > Massimiliano > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From leoni.massimiliano1 at gmail.com Thu Nov 13 06:24:30 2014 From: leoni.massimiliano1 at gmail.com (Massimiliano Leoni) Date: Thu, 13 Nov 2014 13:24:30 +0100 Subject: [petsc-users] Roles of ksp_type and pc_type In-Reply-To: References: <2594845.ya19CE4v59@debianxps> Message-ID: <2663092.Eyt2Jgar1o@debianxps> In data gioved? 13 novembre 2014 05:59:17, hai scritto: > On Thu, Nov 13, 2014 at 5:39 AM, Massimiliano Leoni < > We call Hypre directly to apply P^{-1}. > > It should be. Send -ksp_view and we can verify. It looks like that's what's happening! :D Here it is http://pastebin.com/6J3Rj4x1 > You should use > -pc_fieldsplit_schur_factorization_type diag Ok, This is not clear to me. According to what I studied, Schur decomposition is when I formally eliminate a variable [for Stokes, velocity u] and get a system in the others [sole pressure], which I then solve by some smart method and substitute back to find u. This is done on the full Stokes matrix A, not on the preconditioner... It should be not what I want to do, as I'm facing the whole system at once. What does PETSc mean by "Schur" here? > Thanks, > > Matt Thank you very much! Massimiliano From knepley at gmail.com Thu Nov 13 06:39:38 2014 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 13 Nov 2014 06:39:38 -0600 Subject: [petsc-users] Roles of ksp_type and pc_type In-Reply-To: <2663092.Eyt2Jgar1o@debianxps> References: <2594845.ya19CE4v59@debianxps> <2663092.Eyt2Jgar1o@debianxps> Message-ID: On Thu, Nov 13, 2014 at 6:24 AM, Massimiliano Leoni < leoni.massimiliano1 at gmail.com> wrote: > In data gioved? 13 novembre 2014 05:59:17, hai scritto: > > On Thu, Nov 13, 2014 at 5:39 AM, Massimiliano Leoni < > > We call Hypre directly to apply P^{-1}. > > > > It should be. Send -ksp_view and we can verify. > > It looks like that's what's happening! :D > Here it is > http://pastebin.com/6J3Rj4x1 This is not exactly right because here you are just using additive fieldsplit. The mass matrix is a good preconditioner for the Schur complement, not the zero matrix. Take a look at these slides: http://www.mcs.anl.gov/petsc/documentation/tutorials/ParisTutorial.pdf at slide 115, or my talk from SIAM CS&E http://people.cs.uchicago.edu/~knepley/presentations/SIAMCSE13.pdf Thanks, Matt > > > You should use > > -pc_fieldsplit_schur_factorization_type diag > Ok, This is not clear to me. > According to what I studied, Schur decomposition is when I formally > eliminate > a variable [for Stokes, velocity u] and get a system in the others [sole > pressure], which I then solve by some smart method and substitute back to > find > u. This is done on the full Stokes matrix A, not on the preconditioner... > It should be not what I want to do, as I'm facing the whole system at once. > > What does PETSc mean by "Schur" here? > > > Thanks, > > > > Matt > > Thank you very much! > Massimiliano > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From leoni.massimiliano1 at gmail.com Thu Nov 13 08:07:40 2014 From: leoni.massimiliano1 at gmail.com (Massimiliano Leoni) Date: Thu, 13 Nov 2014 15:07:40 +0100 Subject: [petsc-users] Roles of ksp_type and pc_type In-Reply-To: References: <2594845.ya19CE4v59@debianxps> <2663092.Eyt2Jgar1o@debianxps> Message-ID: <1928038.ts4lkFBGPS@debianxps> In data gioved? 13 novembre 2014 06:39:38, Matthew Knepley ha scritto: > On Thu, Nov 13, 2014 at 6:24 AM, Massimiliano Leoni < > > This is not exactly right because here you are just using additive > fieldsplit. AFAI understood from the user manual, this should be right because my preconditioner is block diagonal, so I want to use block Jacobi. If I used the other one, P = [[C,O],[B,Mp/nu]] then I would need a multiplicative fieldsplit because this would be lower triangular. Is this correct? > The mass matrix > is a good preconditioner for the Schur complement, not the zero matrix. > Take a look at these slides: What do you mean? I know from theory that the preconditioner I am using is optimal [iteration number independent of grid size]. I read the slides and the manual and the Schur complement is what I was talking about earlier, so it's not what I wanna do now -- even though knowing that it's so easy to implement my be very useful. Anyway, thanks for the slides, they have been enlightening on the power of command line options! > Thanks, > > Matt > Thanks again, Massimiliano From bsmith at mcs.anl.gov Thu Nov 13 13:19:34 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 13 Nov 2014 13:19:34 -0600 Subject: [petsc-users] Setting up MPIBAIJ for MatMult In-Reply-To: <1415839716.75219.YahooMailBasic@web125403.mail.ne1.yahoo.com> References: <1415839716.75219.YahooMailBasic@web125403.mail.ne1.yahoo.com> Message-ID: I have put support in for allowing setting the block size before the MatLoad() in the branches barry/fix-matsetsize-matload and next. It will be in the next patch release. Thanks Barry > On Nov 12, 2014, at 6:48 PM, Steena M wrote: > > Thank you, Barry. That took care of most of the issues. Two more things: > > 1) The code runs (MatMult executes) for block size =1. For block sizes > 1. For instance, block size = 3, throws: > PETSC ERROR: Arguments are incompatible! > [0]PETSC ERROR: Cannot change block size 3 to 1! > > Matrix setup: > > ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,filein,FILE_MODE_READ,&fd); > ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); > ierr = MatSetType(A, MATMPIBAIJ); > ierr = MatSetBlockSize(A, bs); CHKERRQ(ierr); > ierr = MatLoad(A,fd);CHKERRQ(ierr); > ierr = PetscViewerDestroy(&fd);CHKERRQ(ierr); > > > 2) I'm not sure if my vector setup is ideal. Should I be using VecSetValues instead? > > ierr = MatGetSize(A,&m,&n); > > ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); > ierr = VecSetType(x,VECMPI); > ierr = VecSetSizes(x,PETSC_DECIDE,m);CHKERRQ(ierr); > ierr = VecSetFromOptions(x);CHKERRQ(ierr); > > -------------------------------------------- > On Wed, 11/12/14, Barry Smith wrote: > > Subject: Re: [petsc-users] Setting up MPIBAIJ for MatMult > To: "Steena M" > Cc: petsc-users at mcs.anl.gov > Date: Wednesday, November 12, 2014, 11:44 AM > > > 1) > remove all the "preload" stuff > > 2) MatLoad doesn't require or use the > Mat preallocation stuff so remove all of that > > 3) MatLoad > determines the global matrix size from the binary file so do > not set that before calling MatLoad > > 4) All processes in the comm (in > your case MPI_COMM_WORLD) need to call MatLoad > > 5) You can call > MatSetBlockSize() to set the block size before MatLoad() > > 6) You do not > need to call MatAssemblyBegin/End after a MatLoad() it would > do nothing > >> On Nov 12, 2014, at > 1:32 PM, Steena M > wrote: >> >> I am > trying to read in a sparse matrix file in binary format > (.dat file) to set it up as MPIBAIJ for MPIMatMult. I > don't think I'm doing the file reading or the matrix > and vector setup correctly. Should the file loading, matrix > setup, and loading be done only on rank 0 together with the > vector setup? After looking through several examples, this > is the structure of my code right now: >> > >> Vec x,y; >> Mat A; >> PetscViewer fd; >> int rank, global_row_size, > global_col_size,bs; >> PetscBool > PetscPreLoad = PETSC_FALSE; >> char > filein[PETSC_MAX_PATH_LEN] /*binary .dat > matrix file */ >> PetscScalar one = > 1.0; >> >> >> /* Reading relevant block size from ENV > */ >> char* bs_env; >> bs_env = getenv > ("BLOCK_SIZE_ENV"); >> bs = > (PetscInt)atoi(bs_env); >> >> /*Reading in matrix row/cols size from > ENV. Matrices are always square: > global_col_size=global_row_size*/ >> >> char* m_env; >> > m_env = getenv ("MATRIX_ROWS_ENV"); >> global_row_size = > (PetscInt)atoi(m_env); >> > global_col_size = global _row_size; >> >> > PetscInitialize(&argc,&args,(char *)0,help); >> ierr = > MPI_Comm_rank(PETSC_COMM_WORLD,&rank); >> >> /*Matrix reading, > setup, assembly*/ >> >> if (flg) PetscPreLoad = PETSC_TRUE; >> > PetscPreLoadBegin(PetscPreLoad,"Load"); >> ierr = > PetscViewerBinaryOpen(PETSC_COMM_WORLD,filein[PetscPreLoadIt],FILE_MODE_READ,&fd);CHKERRQ(ierr); >> >> >> ierr = > MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); >> ierr = > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,global_row_size,global_col_size);CHKERRQ(ierr); >> ierr = > MatSetFromOptions(A);CHKERRQ(ierr); >> > ierr = MatSetType(A, MATMPIBAIJ); >> > ierr = > MatMPIBAIJSetPreallocation(A,bs,5,PETSC_NULL,5,PETSC_NULL);CHKERRQ(ierr); >> ierr = > MatSeqBAIJSetPreallocation(A,bs,5,PETSC_NULL);CHKERRQ(ierr); >> >> ierr = > PetscLogStageRegister("Assembly", > &stage);CHKERRQ(ierr); >> ierr = > PetscLogStagePush(stage);CHKERRQ(ierr); >> > >> >> if > (rank==0) >> { >> > ierr = MatLoad(A,fd);CHKERRQ(ierr); >> > >> } >> >> ierr = > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = > PetscLogStagePop();CHKERRQ(ierr); >> ierr > = PetscViewerDestroy(&fd);CHKERRQ(ierr); >> >> /*Trying two types > of vector assembly*/ >> >> /* Vector setup Attempt 1*/ >> >> ierr = > VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); >> ierr = > VecSetSizes(x,PETSC_DECIDE,global_row_size);CHKERRQ(ierr); >> ierr = > VecSetFromOptions(x);CHKERRQ(ierr); >> > ierr = VecDuplicate(x,&y);CHKERRQ(ierr); >> ierr = > VecSetFromOptions(y);CHKERRQ(ierr); >> > ierr = VecSet(x,one);CHKERRQ(ierr); >> > ierr = VecSet(y,one); CHKERRQ(ierr); >> > >> /*Vector setup Attempt 2*/ >> >> ierr = > VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); >> ierr = > VecGetSize(x,&global_row_size);CHKERRQ(ierr); >> >> if (rank==0) >> { >> >> for (i = 0; i i++) >> { >> ierr = > VecSetValues(x,1,&i,one,INSERT_VALUES); >> CHKERRQ(ierr); >> > } >> >> } >> ierr = > VecDuplicate(x,&y);CHKERRQ(ierr); >> > ierr = VecSetFromOptions(y);CHKERRQ(ierr); >> >> ierr = > VecAssemblyBegin(x); CHKERRQ(ierr); >> > ierr = VecAssemblyEnd(x); CHKERRQ(ierr); >> >> >> >> /* SpMV*/ >> ierr = MatMult(A,x,y);CHKERRQ(ierr); >> >> ierr = > VecDestroy(&x);CHKERRQ(ierr); >> > ierr = VecDestroy(&y);CHKERRQ(ierr); >> ierr = > MatDestroy(&A);CHKERRQ(ierr); >> >> >> >> > From bsmith at mcs.anl.gov Thu Nov 13 22:19:14 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 13 Nov 2014 22:19:14 -0600 Subject: [petsc-users] PETSc VecLoad not obeying HDF5 group? In-Reply-To: <545B3045.5060001@hakostra.net> References: <545B3045.5060001@hakostra.net> Message-ID: <1915FD1C-8B67-47C9-A7AB-F90B9C1B71BB@mcs.anl.gov> Sorry no one answered this before, I had assumed someone who knew something about HDF5 would answer but instead looks like I have to. Could you please send a simple example code (and data file) that reproduces the problem. I looked at the code and (in theory) it should handle the group ok. Again sorry for the long delay in anyone answering. Barry > On Nov 6, 2014, at 2:24 AM, H?kon Strandenes wrote: > > Hi, > > I am trying to load a dataset from a HDF5 file and into a vector in PETSc. For that purpose I use VecView etc in the following way: > > PetscViewerHDF5Open(PETSC_COMM_WORLD, "input.h5", FILE_MODE_READ, &H5viewer); > > PetscViewerHDF5PushGroup(H5viewer, "/FIELDS"); > > VecLoad(gSol, H5viewer); > > PetscViewerDestroy(&H5viewer); > > And yes, I also check the return variables for errors, I have just omitted them from this example to make it easier to read. > > My problem is: > If I place a dataset, with the correct name and dimensions, in the group "/FIELDS" in my input.h5, PETSc can't find it and won't load it. If I place the same dataset in the root of my HDF5 file, the dataset is loaded and everything works, even with PetscViewerHDF5PushGroup set to "/FIELDS"... > > Does actually VecLoad obey the group set by PetscViewerHDF5PushGroup? > > Best regards, > H?kon Strandenes From haakon at hakostra.net Fri Nov 14 02:09:05 2014 From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=) Date: Fri, 14 Nov 2014 09:09:05 +0100 Subject: [petsc-users] PETSc VecLoad not obeying HDF5 group? In-Reply-To: <1915FD1C-8B67-47C9-A7AB-F90B9C1B71BB@mcs.anl.gov> References: <545B3045.5060001@hakostra.net> <1915FD1C-8B67-47C9-A7AB-F90B9C1B71BB@mcs.anl.gov> Message-ID: <5465B8A1.1000302@hakostra.net> See attached file. The test is really simple. The HDF5 file contains two (equal) datasets/vectors: - One dataset u in the root of the file "/" - One dataset v in the group "/FIELDS" The example program tries to load two vectors, u and v, from the file. The PetscViewerHDF5PushGroup is set to "/FIELDS" for both. It will first try to load u, then v. If everything works, it SHOULD FAIL on loading u, since this vector is not stored in the correct location in the HDF5 file. PETSc should search in "/FIELDS" for this vector, not "/". It should however SUCCEED in loading "v", since this is located correctly. In my latest version of PETSc from Git, this is inverted. The program SUCCEED in loading u, and FAIL in loading v. Something is wrong... BTW: Is there an error in DM ex4? Is the vector u destroyed properly? H?kon On 14. nov. 2014 05:19, Barry Smith wrote: > > Sorry no one answered this before, I had assumed someone who knew something about HDF5 would answer but instead looks like I have to. > > Could you please send a simple example code (and data file) that reproduces the problem. I looked at the code and (in theory) it should handle the group ok. > > Again sorry for the long delay in anyone answering. > > Barry > > >> On Nov 6, 2014, at 2:24 AM, H?kon Strandenes wrote: >> >> Hi, >> >> I am trying to load a dataset from a HDF5 file and into a vector in PETSc. For that purpose I use VecView etc in the following way: >> >> PetscViewerHDF5Open(PETSC_COMM_WORLD, "input.h5", FILE_MODE_READ, &H5viewer); >> >> PetscViewerHDF5PushGroup(H5viewer, "/FIELDS"); >> >> VecLoad(gSol, H5viewer); >> >> PetscViewerDestroy(&H5viewer); >> >> And yes, I also check the return variables for errors, I have just omitted them from this example to make it easier to read. >> >> My problem is: >> If I place a dataset, with the correct name and dimensions, in the group "/FIELDS" in my input.h5, PETSc can't find it and won't load it. If I place the same dataset in the root of my HDF5 file, the dataset is loaded and everything works, even with PetscViewerHDF5PushGroup set to "/FIELDS"... >> >> Does actually VecLoad obey the group set by PetscViewerHDF5PushGroup? >> >> Best regards, >> H?kon Strandenes > > -------------- next part -------------- A non-text attachment was scrubbed... Name: VecLoadTest.tar.gz Type: application/gzip Size: 2749 bytes Desc: not available URL: From popov at uni-mainz.de Fri Nov 14 08:18:15 2014 From: popov at uni-mainz.de (anton) Date: Fri, 14 Nov 2014 15:18:15 +0100 Subject: [petsc-users] DMLocalToLocal Message-ID: <54660F27.5040606@uni-mainz.de> Hi, can I call DMLocalToLocalBegin\End on the same vector (g and l) in the INSERT_VALUES mode? The motivation is not to create the global buffer vector to be used with DMGlobalToLocalBegin\End. I just need to scatter ghost values to the neighbors. Thanks, Anton From knepley at gmail.com Fri Nov 14 08:37:42 2014 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 14 Nov 2014 08:37:42 -0600 Subject: [petsc-users] DMLocalToLocal In-Reply-To: <54660F27.5040606@uni-mainz.de> References: <54660F27.5040606@uni-mainz.de> Message-ID: On Fri, Nov 14, 2014 at 8:18 AM, anton wrote: > Hi, > > can I call DMLocalToLocalBegin\End on the same vector (g and l) in the > INSERT_VALUES mode? > The motivation is not to create the global buffer vector to be used with > DMGlobalToLocalBegin\End. > I just need to scatter ghost values to the neighbors. > Yes, that should work. Matt > Thanks, > Anton > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Fri Nov 14 08:39:27 2014 From: popov at uni-mainz.de (anton) Date: Fri, 14 Nov 2014 15:39:27 +0100 Subject: [petsc-users] DMLocalToLocal In-Reply-To: References: <54660F27.5040606@uni-mainz.de> Message-ID: <5466141F.9010006@uni-mainz.de> On 11/14/2014 03:37 PM, Matthew Knepley wrote: > On Fri, Nov 14, 2014 at 8:18 AM, anton > wrote: > > Hi, > > can I call DMLocalToLocalBegin\End on the same vector (g and l) in > the INSERT_VALUES mode? > The motivation is not to create the global buffer vector to be > used with DMGlobalToLocalBegin\End. > I just need to scatter ghost values to the neighbors. > > > Yes, that should work. > That's just great! Thanks Matt. > Matt > > Thanks, > Anton > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Nov 14 16:41:25 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 14 Nov 2014 16:41:25 -0600 Subject: [petsc-users] PETSc VecLoad not obeying HDF5 group? In-Reply-To: <5465B8A1.1000302@hakostra.net> References: <545B3045.5060001@hakostra.net> <1915FD1C-8B67-47C9-A7AB-F90B9C1B71BB@mcs.anl.gov> <5465B8A1.1000302@hakostra.net> Message-ID: <1469376F-F1A6-4541-8EB4-35294D79651E@mcs.anl.gov> Thanks. I found the bug. The routine VecLoad_HDF5_DA() did not use the group properly (though the regular VecLoad_HDF5() did). Please find attached a patch to apply to the petsc root directory to fix the problem. The fix is also in the branch barry/fix-vecload-group-handling This will also be fixed in our next patch release. Thanks for reporting the problem. Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-vecload-group-handling.patch Type: application/octet-stream Size: 1793 bytes Desc: not available URL: -------------- next part -------------- > On Nov 14, 2014, at 2:09 AM, H?kon Strandenes wrote: > > See attached file. > > The test is really simple. The HDF5 file contains two (equal) datasets/vectors: > > - One dataset u in the root of the file "/" > - One dataset v in the group "/FIELDS" > > The example program tries to load two vectors, u and v, from the file. The PetscViewerHDF5PushGroup is set to "/FIELDS" for both. It will first try to load u, then v. > > If everything works, it SHOULD FAIL on loading u, since this vector is not stored in the correct location in the HDF5 file. PETSc should search in "/FIELDS" for this vector, not "/". It should however SUCCEED in loading "v", since this is located correctly. > > In my latest version of PETSc from Git, this is inverted. The program SUCCEED in loading u, and FAIL in loading v. Something is wrong... > > BTW: Is there an error in DM ex4? Is the vector u destroyed properly? > > H?kon > > > On 14. nov. 2014 05:19, Barry Smith wrote: >> >> Sorry no one answered this before, I had assumed someone who knew something about HDF5 would answer but instead looks like I have to. >> >> Could you please send a simple example code (and data file) that reproduces the problem. I looked at the code and (in theory) it should handle the group ok. >> >> Again sorry for the long delay in anyone answering. >> >> Barry >> >> >>> On Nov 6, 2014, at 2:24 AM, H?kon Strandenes wrote: >>> >>> Hi, >>> >>> I am trying to load a dataset from a HDF5 file and into a vector in PETSc. For that purpose I use VecView etc in the following way: >>> >>> PetscViewerHDF5Open(PETSC_COMM_WORLD, "input.h5", FILE_MODE_READ, &H5viewer); >>> >>> PetscViewerHDF5PushGroup(H5viewer, "/FIELDS"); >>> >>> VecLoad(gSol, H5viewer); >>> >>> PetscViewerDestroy(&H5viewer); >>> >>> And yes, I also check the return variables for errors, I have just omitted them from this example to make it easier to read. >>> >>> My problem is: >>> If I place a dataset, with the correct name and dimensions, in the group "/FIELDS" in my input.h5, PETSc can't find it and won't load it. If I place the same dataset in the root of my HDF5 file, the dataset is loaded and everything works, even with PetscViewerHDF5PushGroup set to "/FIELDS"... >>> >>> Does actually VecLoad obey the group set by PetscViewerHDF5PushGroup? >>> >>> Best regards, >>> H?kon Strandenes >> >> > From bsmith at mcs.anl.gov Fri Nov 14 17:45:54 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 14 Nov 2014 17:45:54 -0600 Subject: [petsc-users] schurcomplement of a rectangular system In-Reply-To: References: Message-ID: I have put this into the branches barry/allow-nonsquare-schurcomplement and next Barry > On Nov 11, 2014, at 8:51 AM, Patrick Lacasse wrote: > > Hi, > > in my works, I'm computing a formula (involving an L2 projection) of the form: > > (D - M^{-1} N) n > > where D,M and B are matrices and n is a vector. > > Operator (D - M^{-1} N) appears to be the Schur complement of the system > ( M N ) > ( I D ) > and it is convenient for me to use a Mat of type schurcomplement to compute it. > However, my matrix D is not a square matrix (this system is not invertible, it is rectangular). > > Assertions in src/ksp/ksp/utils/schurm.c prohibit the use of schurcomplement for rectangular system, but I can't figure where it is needed by schurcomplement itself. > Could it be possible to remove those two assertions (see my patch)? > > > Patrick Lacasse > <0001-Remove-an-assertion-in-MatSchurComplement.patch> From thronesf at gmail.com Sat Nov 15 17:08:40 2014 From: thronesf at gmail.com (Sharp Stone) Date: Sat, 15 Nov 2014 18:08:40 -0500 Subject: [petsc-users] Mat does not support LU ? Message-ID: Hi all, I got an error saying "Matrix format mpiaij does not have a built-in PETSc LU" when I try to set pc type. Any suggestions are highly appreciated. -- Best regards, Feng -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Nov 15 17:34:50 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 15 Nov 2014 17:34:50 -0600 Subject: [petsc-users] Mat does not support LU ? In-Reply-To: References: Message-ID: On Sat, Nov 15, 2014 at 5:08 PM, Sharp Stone wrote: > Hi all, > > I got an error saying "Matrix format mpiaij does not have a built-in PETSc > LU" when I try to set pc type. Any suggestions are highly appreciated. > --download-superlu_dist http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html Matt > -- > Best regards, > > Feng > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sat Nov 15 15:30:24 2014 From: jed at jedbrown.org (Jed Brown) Date: Sat, 15 Nov 2014 14:30:24 -0700 Subject: [petsc-users] MatSolve in CG In-Reply-To: <17A78B9D13564547AC894B88C1596747203B9455@XMBX4.uibk.ac.at> References: <17A78B9D13564547AC894B88C1596747203B9455@XMBX4.uibk.ac.at> Message-ID: <87y4rcjgnj.fsf@jedbrown.org> "De Groof, Vincent Frans Maria" writes: > Dear all, > > > I am writing a few variations on the PCG algorithm. Starting with my own PCG implementation, I noticed that my routine is quite a bit slower (~20%). I think I tracked it down to the MatSolve calls in PCApply. I am using BDDC as the preconditioner. > > In the Petsc PCG function, each PCApply calls the MatSolve 3 times. It also calls MatSolve once somewhere between calling KSPSolve and KSPSolve_CG (KSP_SetUp has been called before). So MatSolve is called 1+3*(# iterations) per KSPSolve. In my own PCG variation, the initial MatSolve is not called. But MatSolve is called 4 times in each iteration, so in total 4*(# iterations). > > Where does this MatSolve in between KSPSolve and KSPSolve_CG come from? And how can I make my function use less MatSolve's? I recommend running in a debugger to get the stack trace. I don't know what is different between your Krylov implementation and CG, but it should be obvious when you compare. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From evanum at gmail.com Sat Nov 15 21:24:31 2014 From: evanum at gmail.com (Evan Um) Date: Sat, 15 Nov 2014 19:24:31 -0800 Subject: [petsc-users] KSPSolve() get slower when preconditioner or Cholesky factor is re-used with many multiple RHS. Message-ID: Dear PETSC users, I would like to show you a performance issue when Cholesky factor is re-used as a direct solver or pre-conditioner many times with many right-hand side vectors. Does anyone suggest a solution about this issue? In advance, thanks for your help. Regards, Evan Example 1: I used MUMPS as a direct solver and measured backward/forward substitution solution time for selected RHS vectors. A simplified test code is shown in code 1 and time measurements in result 1. As shown below, backward and forward solution time is nearly constant (e.g. about 2.2 seconds) in early time, but later the solution time overall increases. In contrast, in late time, it takes about 12.8 seconds. It is hard to understand why the backward/forward solution time gets longer although the same numerical operation is carried out with each RHS vector (for i). For many RHS vectors, is there any parameter that I need to reset to stably take care of lots of RHS vectors? Example 2: In this case, I use the Cholesky factor as a preconditioner for CG solver. One iteration is performed. Its sample code and time measurments are shown in Code 2 and Result 2. Again, KSPSolve() gets slower as the preconditioner is re-used with many RHS vectors. For example, In early stage, it took about 4.6 seconds. Later, it took 16 seconds. Does anyone observe such a performance issue? Do you know any solution for this problem? In the two experiments, I expected Example 2 would show shorter solution time with each RHS vector than Example 1 because Example 2 uses scalable matrix-vector multiplication. Instead, when MUMPS is used in Example 1, MUMPS carries out back/forward substitution that are inherently un-scalable. However, my experiments consistently showed that the back/forward substitution of MUMPS is faster than a single PCG iteration with Cholesky preconditioner. Can anyone explain this? FYI, in both example 1 and 2, I used a very large test problem (about sparse SPD 4,000,000-by-4,000,000 matrix). Code 1: KSPCreate(PETSC_COMM_WORLD, &ksp_fetd_dt); KSPSetOperators(ksp_fetd_dt, A_dt, A_dt); KSPSetType (ksp_fetd_dt, KSPPREONLY); KSPGetPC(ksp_fetd_dt, &pc_fetd_dt); MatSetOption(A_dt, MAT_SPD, PETSC_TRUE); PCSetType(pc_fetd_dt, PCCHOLESKY); PCFactorSetMatSolverPackage(pc_fetd_dt, MATSOLVERMUMPS); PCFactorSetUpMatSolverPackage(pc_fetd_dt); PCFactorGetMatrix(pc_fetd_dt, &F_dt); for (int i=0; i<1000; i++) { // Create a new RHS vector B_dt time1=my_timer_function(); KSPSolve(ksp_fetd_dt,B_dt,solution); time2=my_timer_function () // Output solution time=time2-time1; } Result1: nTS:12 Time(s):2.400e-04 Sol. time(s):2.246e+00 PCG niter:1 nTS:13 Time(s):2.600e-04 Sol. time(s):2.329e+00 PCG niter:1 nTS:14 Time(s):2.800e-04 Sol. time(s):2.289e+00 PCG niter:1 nTS:15 Time(s):3.000e-04 Sol. time(s):2.239e+00 PCG niter:1 . . nTS:267 Time(s):5.340e-03 Sol. time(s):2.152e+00 PCG niter:1 nTS:280 Time(s):5.600e-03 Sol. time(s):2.255e+00 PCG niter:1 nTS:293 Time(s):5.860e-03 Sol. time(s):1.087e+01 PCG niter:1 nTS:307 Time(s):6.140e-03 Sol. time(s):1.225e+01 PCG niter:1 nTS:321 Time(s):6.420e-03 Sol. time(s):1.280e+01 PCG niter:1 nTS:337 Time(s):6.740e-03 Sol. time(s):1.243e+01 PCG niter:1 nTS:353 Time(s):7.060e-03 Sol. time(s):6.953e+00 PCG niter:1 nTS:369 Time(s):7.380e-03 Sol. time(s):8.712e+00 PCG niter:1 nTS:387 Time(s):7.740e-03 Sol. time(s):9.048e+00 PCG niter:1 nTS:405 Time(s):8.100e-03 Sol. time(s):1.099e+01 PCG niter:1 nTS:424 Time(s):8.480e-03 Sol. time(s):1.171e+01 PCG niter:1 nTS:445 Time(s):8.900e-03 Sol. time(s):1.294e+01 PCG niter:1 nTS:466 Time(s):9.320e-03 Sol. time(s):1.227e+01 PCG niter:1 nTS:488 Time(s):9.760e-03 Sol. time(s):8.581e+00 PCG niter:1 nTS:511 Time(s):1.022e-02 Sol. time(s):1.059e+01 PCG niter:1 nTS:535 Time(s):1.070e-02 Sol. time(s):8.452e+00 PCG niter:1 Code 2: KSPCreate(PETSC_COMM_WORLD, &ksp_fetd_dt); KSPSetOperators(ksp_fetd_dt, A_dt, A_dt); KSPSetType (ksp_fetd_dt, KSPPREONLY); KSPGetPC(ksp_fetd_dt, &pc_fetd_dt); MatSetOption(A_dt, MAT_SPD, PETSC_TRUE); PCSetType(pc_fetd_dt, PCCHOLESKY); PCFactorSetMatSolverPackage(pc_fetd_dt, MATSOLVERMUMPS); PCFactorSetUpMatSolverPackage(pc_fetd_dt); PCFactorGetMatrix(pc_fetd_dt, &F_dt); KSPSetType(ksp_fetd_dt, KSPCG); KSPSetTolerances(ksp_fetd_dt, 1e-9, 1.0e-50, 1.0e10, ksp_iter); for (int i=0; i<1000; i++) { // Create a new RHS vector B_dt time1=my_timer_function(); KSPSolve(ksp_fetd_dt,B_dt,solution); time2=my_timer_function () // Output solution time=time2-time1; } Result 2: nTS:10 Time(s):2.000e-04 Sol. time(s):4.644e+00 PCG niter:1 Norm:2.937e-10 nTS:11 Time(s):2.200e-04 Sol. time(s):4.585e+00 PCG niter:1 Norm:3.151e-10 nTS:12 Time(s):2.400e-04 Sol. time(s):4.737e+00 PCG niter:1 Norm:2.719e-10 nTS:13 Time(s):2.600e-04 Sol. time(s):4.537e+00 PCG niter:1 Norm:3.982e-10 nTS:14 Time(s):2.800e-04 Sol. time(s):4.578e+00 PCG niter:1 Norm:3.221e-10 nTS:15 Time(s):3.000e-04 Sol. time(s):4.678e+00 PCG niter:1 Norm:3.008e-10 nTS:16 Time(s):3.200e-04 Sol. time(s):4.619e+00 PCG niter:1 Norm:3.005e-10 nTS:46 Time(s):9.200e-04 Sol. time(s):6.862e+00 PCG niter:1 Norm:2.759e-10 nTS:48 Time(s):9.600e-04 Sol. time(s):4.475e+00 PCG niter:1 Norm:3.071e-10 nTS:50 Time(s):1.000e-03 Sol. time(s):6.068e+00 PCG niter:1 Norm:3.848e-10 nTS:52 Time(s):1.040e-03 Sol. time(s):5.024e+00 PCG niter:1 Norm:3.639e-10 nTS:55 Time(s):1.100e-03 Sol. time(s):1.333e+01 PCG niter:1 Norm:3.049e-10 nTS:58 Time(s):1.160e-03 Sol. time(s):1.440e+01 PCG niter:1 Norm:3.467e-10 nTS:60 Time(s):1.200e-03 Sol. time(s):1.475e+01 PCG niter:1 Norm:2.951e-10 nTS:63 Time(s):1.260e-03 Sol. time(s):9.899e+00 PCG niter:1 Norm:3.018e-10 nTS:66 Time(s):1.320e-03 Sol. time(s):1.097e+01 PCG niter:1 Norm:3.320e-10 nTS:69 Time(s):1.380e-03 Sol. time(s):1.485e+01 PCG niter:1 Norm:2.811e-10 nTS:73 Time(s):1.460e-03 Sol. time(s):1.199e+01 PCG niter:1 Norm:2.999e-10 nTS:76 Time(s):1.520e-03 Sol. time(s):1.109e+01 PCG niter:1 Norm:3.359e-10 nTS:80 Time(s):1.600e-03 Sol. time(s):1.473e+01 PCG niter:1 Norm:3.336e-10 nTS:84 Time(s):1.680e-03 Sol. time(s):1.444e+01 PCG niter:1 Norm:3.181e-10 nTS:88 Time(s):1.760e-03 Sol. time(s):1.639e+01 PCG niter:1 Norm:2.956e-10 nTS:92 Time(s):1.840e-03 Sol. time(s):1.713e+01 PCG niter:1 Norm:3.552e-10 nTS:96 Time(s):1.920e-03 Sol. time(s):1.562e+01 PCG niter:1 Norm:2.843e-10 nTS:101 Time(s):2.020e-03 Sol. time(s):1.631e+01 PCG niter:1 Norm:3.687e-10 nTS:105 Time(s):2.100e-03 Sol. time(s):1.496e+01 PCG niter:1 Norm:3.567e-10 nTS:111 Time(s):2.220e-03 Sol. time(s):1.607e+01 PCG niter:1 Norm:3.392e-10 -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at kaust.edu.sa Sun Nov 16 00:45:39 2014 From: stefano.zampini at kaust.edu.sa (Stefano Zampini) Date: Sun, 16 Nov 2014 06:45:39 +0000 Subject: [petsc-users] MatSolve in CG In-Reply-To: <87y4rcjgnj.fsf@jedbrown.org> References: <17A78B9D13564547AC894B88C1596747203B9455@XMBX4.uibk.ac.at>, <87y4rcjgnj.fsf@jedbrown.org> Message-ID: Vincent, PCBDDC has three nested solvers in it (Dirichlet, Neumann and Coarse solver), each of them calling MatSolve if you are using direct solvers for them (which is the default). When used in conjunction with conjugate gradients, BDDC can take advantage of that and just use one dirichlet solve per iteration (this is a well known trick for hybrid Schwarz algorithms); if the KSP is not of the CG type, PCBDDC will instead need 2 Dirichlet solvers per iteration Thus the difference in the number of MatSolve per iteration: in your PCG implementation, your CG kernel is not aware of this and does not propagate the information to PCBDDC.. Regarding the additional solve you report somewhere-in-between KSPSolve and KSPSolve_CG: the above trick for hybrid Shwarz algorithms come at a cost, and this cost is an additional Dirichlet solve inside PCPresolve_BDDC. Stefano ________________________________________ Da: Jed Brown [jed at jedbrown.org] Inviato: domenica 16 novembre 2014 0.30 A: De Groof, Vincent Frans Maria; petsc-users at mcs.anl.gov Cc: Stefano Zampini Oggetto: Re: [petsc-users] MatSolve in CG "De Groof, Vincent Frans Maria" writes: > Dear all, > > > I am writing a few variations on the PCG algorithm. Starting with my own PCG implementation, I noticed that my routine is quite a bit slower (~20%). I think I tracked it down to the MatSolve calls in PCApply. I am using BDDC as the preconditioner. > > In the Petsc PCG function, each PCApply calls the MatSolve 3 times. It also calls MatSolve once somewhere between calling KSPSolve and KSPSolve_CG (KSP_SetUp has been called before). So MatSolve is called 1+3*(# iterations) per KSPSolve. In my own PCG variation, the initial MatSolve is not called. But MatSolve is called 4 times in each iteration, so in total 4*(# iterations). > > Where does this MatSolve in between KSPSolve and KSPSolve_CG come from? And how can I make my function use less MatSolve's? I recommend running in a debugger to get the stack trace. I don't know what is different between your Krylov implementation and CG, but it should be obvious when you compare. ________________________________ This message and its contents including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. From knepley at gmail.com Sun Nov 16 05:57:53 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 16 Nov 2014 05:57:53 -0600 Subject: [petsc-users] KSPSolve() get slower when preconditioner or Cholesky factor is re-used with many multiple RHS. In-Reply-To: References: Message-ID: On Sat, Nov 15, 2014 at 9:24 PM, Evan Um wrote: > Dear PETSC users, > > I would like to show you a performance issue when Cholesky factor is > re-used as a direct solver or pre-conditioner many times with many > right-hand side vectors. Does anyone suggest a solution about this issue? > In advance, thanks for your help. > > Regards, > Evan > > Example 1: I used MUMPS as a direct solver and measured backward/forward > substitution solution time for selected RHS vectors. A simplified test code > is shown in code 1 and time measurements in result 1. As shown below, > backward and forward solution time is nearly constant (e.g. about 2.2 > seconds) in early time, but later the solution time overall increases. In > contrast, in late time, it takes about 12.8 seconds. It is hard to > understand why the backward/forward solution time gets longer although the > same numerical operation is carried out with each RHS vector (for i). For > many RHS vectors, is there any parameter that I need to reset to stably > take care of lots of RHS vectors? > > Example 2: In this case, I use the Cholesky factor as a preconditioner > for CG solver. One iteration is performed. Its sample code and time > measurments are shown in Code 2 and Result 2. Again, KSPSolve() gets slower > as the preconditioner is re-used with many RHS vectors. For example, In > early stage, it took about 4.6 seconds. Later, it took 16 seconds. Does > anyone observe such a performance issue? Do you know any solution for this > problem? > In the two experiments, I expected Example 2 would show shorter solution > time with each RHS vector than Example 1 because Example 2 uses scalable > matrix-vector multiplication. Instead, when MUMPS is used in Example 1, > MUMPS carries out back/forward substitution that are inherently > un-scalable. However, my experiments consistently showed that the > back/forward substitution of MUMPS is faster than a single PCG iteration > with Cholesky preconditioner. Can anyone explain this? FYI, in both example > 1 and 2, I used a very large test problem (about sparse SPD > 4,000,000-by-4,000,000 matrix). > > Code 1: > KSPCreate(PETSC_COMM_WORLD, &ksp_fetd_dt); > KSPSetOperators(ksp_fetd_dt, A_dt, A_dt); > KSPSetType (ksp_fetd_dt, KSPPREONLY); > KSPGetPC(ksp_fetd_dt, &pc_fetd_dt); > MatSetOption(A_dt, MAT_SPD, PETSC_TRUE); > PCSetType(pc_fetd_dt, PCCHOLESKY); > PCFactorSetMatSolverPackage(pc_fetd_dt, MATSOLVERMUMPS); > PCFactorSetUpMatSolverPackage(pc_fetd_dt); > PCFactorGetMatrix(pc_fetd_dt, &F_dt); > for (int i=0; i<1000; i++) { > // Create a new RHS vector B_dt > Are you calling VecCreate() each time here? If so, you could be filling up the memory of your computer as you run, causing it to slow down due to swapping. For any performance question, ALWAYS send the output of -log_summary. For your question, also divide this loop into log stages for each iteration, so we can see what specific operations slow down. Matt > time1=my_timer_function(); > KSPSolve(ksp_fetd_dt,B_dt,solution); > time2=my_timer_function () > // Output solution time=time2-time1; > } > > Result1: > nTS:12 Time(s):2.400e-04 Sol. time(s):2.246e+00 PCG niter:1 > nTS:13 Time(s):2.600e-04 Sol. time(s):2.329e+00 PCG niter:1 > nTS:14 Time(s):2.800e-04 Sol. time(s):2.289e+00 PCG niter:1 > nTS:15 Time(s):3.000e-04 Sol. time(s):2.239e+00 PCG niter:1 > . > . > nTS:267 Time(s):5.340e-03 Sol. time(s):2.152e+00 PCG niter:1 > nTS:280 Time(s):5.600e-03 Sol. time(s):2.255e+00 PCG niter:1 > nTS:293 Time(s):5.860e-03 Sol. time(s):1.087e+01 PCG niter:1 > nTS:307 Time(s):6.140e-03 Sol. time(s):1.225e+01 PCG niter:1 > nTS:321 Time(s):6.420e-03 Sol. time(s):1.280e+01 PCG niter:1 > nTS:337 Time(s):6.740e-03 Sol. time(s):1.243e+01 PCG niter:1 > nTS:353 Time(s):7.060e-03 Sol. time(s):6.953e+00 PCG niter:1 > nTS:369 Time(s):7.380e-03 Sol. time(s):8.712e+00 PCG niter:1 > nTS:387 Time(s):7.740e-03 Sol. time(s):9.048e+00 PCG niter:1 > nTS:405 Time(s):8.100e-03 Sol. time(s):1.099e+01 PCG niter:1 > nTS:424 Time(s):8.480e-03 Sol. time(s):1.171e+01 PCG niter:1 > nTS:445 Time(s):8.900e-03 Sol. time(s):1.294e+01 PCG niter:1 > nTS:466 Time(s):9.320e-03 Sol. time(s):1.227e+01 PCG niter:1 > nTS:488 Time(s):9.760e-03 Sol. time(s):8.581e+00 PCG niter:1 > nTS:511 Time(s):1.022e-02 Sol. time(s):1.059e+01 PCG niter:1 > nTS:535 Time(s):1.070e-02 Sol. time(s):8.452e+00 PCG niter:1 > > Code 2: > KSPCreate(PETSC_COMM_WORLD, &ksp_fetd_dt); > KSPSetOperators(ksp_fetd_dt, A_dt, A_dt); > KSPSetType (ksp_fetd_dt, KSPPREONLY); > KSPGetPC(ksp_fetd_dt, &pc_fetd_dt); > MatSetOption(A_dt, MAT_SPD, PETSC_TRUE); > PCSetType(pc_fetd_dt, PCCHOLESKY); > PCFactorSetMatSolverPackage(pc_fetd_dt, MATSOLVERMUMPS); > PCFactorSetUpMatSolverPackage(pc_fetd_dt); > PCFactorGetMatrix(pc_fetd_dt, &F_dt); > KSPSetType(ksp_fetd_dt, KSPCG); > KSPSetTolerances(ksp_fetd_dt, 1e-9, 1.0e-50, 1.0e10, ksp_iter); > for (int i=0; i<1000; i++) { > // Create a new RHS vector B_dt > time1=my_timer_function(); > KSPSolve(ksp_fetd_dt,B_dt,solution); > time2=my_timer_function () > // Output solution time=time2-time1; > } > > Result 2: > nTS:10 Time(s):2.000e-04 Sol. time(s):4.644e+00 PCG niter:1 Norm:2.937e-10 > nTS:11 Time(s):2.200e-04 Sol. time(s):4.585e+00 PCG niter:1 Norm:3.151e-10 > nTS:12 Time(s):2.400e-04 Sol. time(s):4.737e+00 PCG niter:1 Norm:2.719e-10 > nTS:13 Time(s):2.600e-04 Sol. time(s):4.537e+00 PCG niter:1 Norm:3.982e-10 > nTS:14 Time(s):2.800e-04 Sol. time(s):4.578e+00 PCG niter:1 Norm:3.221e-10 > nTS:15 Time(s):3.000e-04 Sol. time(s):4.678e+00 PCG niter:1 Norm:3.008e-10 > nTS:16 Time(s):3.200e-04 Sol. time(s):4.619e+00 PCG niter:1 Norm:3.005e-10 > nTS:46 Time(s):9.200e-04 Sol. time(s):6.862e+00 PCG niter:1 Norm:2.759e-10 > nTS:48 Time(s):9.600e-04 Sol. time(s):4.475e+00 PCG niter:1 Norm:3.071e-10 > nTS:50 Time(s):1.000e-03 Sol. time(s):6.068e+00 PCG niter:1 Norm:3.848e-10 > nTS:52 Time(s):1.040e-03 Sol. time(s):5.024e+00 PCG niter:1 Norm:3.639e-10 > nTS:55 Time(s):1.100e-03 Sol. time(s):1.333e+01 PCG niter:1 Norm:3.049e-10 > nTS:58 Time(s):1.160e-03 Sol. time(s):1.440e+01 PCG niter:1 Norm:3.467e-10 > nTS:60 Time(s):1.200e-03 Sol. time(s):1.475e+01 PCG niter:1 Norm:2.951e-10 > nTS:63 Time(s):1.260e-03 Sol. time(s):9.899e+00 PCG niter:1 Norm:3.018e-10 > nTS:66 Time(s):1.320e-03 Sol. time(s):1.097e+01 PCG niter:1 Norm:3.320e-10 > nTS:69 Time(s):1.380e-03 Sol. time(s):1.485e+01 PCG niter:1 Norm:2.811e-10 > nTS:73 Time(s):1.460e-03 Sol. time(s):1.199e+01 PCG niter:1 Norm:2.999e-10 > nTS:76 Time(s):1.520e-03 Sol. time(s):1.109e+01 PCG niter:1 Norm:3.359e-10 > nTS:80 Time(s):1.600e-03 Sol. time(s):1.473e+01 PCG niter:1 Norm:3.336e-10 > nTS:84 Time(s):1.680e-03 Sol. time(s):1.444e+01 PCG niter:1 Norm:3.181e-10 > nTS:88 Time(s):1.760e-03 Sol. time(s):1.639e+01 PCG niter:1 Norm:2.956e-10 > nTS:92 Time(s):1.840e-03 Sol. time(s):1.713e+01 PCG niter:1 Norm:3.552e-10 > nTS:96 Time(s):1.920e-03 Sol. time(s):1.562e+01 PCG niter:1 Norm:2.843e-10 > nTS:101 Time(s):2.020e-03 Sol. time(s):1.631e+01 PCG niter:1 > Norm:3.687e-10 > nTS:105 Time(s):2.100e-03 Sol. time(s):1.496e+01 PCG niter:1 > Norm:3.567e-10 > nTS:111 Time(s):2.220e-03 Sol. time(s):1.607e+01 PCG niter:1 > Norm:3.392e-10 > > > > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From evanum at gmail.com Sun Nov 16 10:11:42 2014 From: evanum at gmail.com (Evan Um) Date: Sun, 16 Nov 2014 08:11:42 -0800 Subject: [petsc-users] KSPSolve() get slower when preconditioner or Cholesky factor is re-used with many multiple RHS. In-Reply-To: References: Message-ID: Dear Matthew, Thanks for your comments. I will prepare log summary. To activate an separate log stage for each iteration (i.e. each RHS vector), I tried the code below, but got an error. Could you give me a comment? Does PetscLogView() allow users to use different output file names such that each iteration generates its own log file? Evan /* work */ PetscViewer viewer; for (int i=0; i<1000; i++) { PetscLogBegin(); /* work to do*/ PetscLogView(viewer); } /* work */ Errors: Number of PCG iterations:1 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Corrupt argument: http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: Invalid type of object: Parameter # 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 [0]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 by esum Sun Nov 16 08:08:53 2014 [0]PETSC ERROR: Configure options --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 --with-ptscotch=1 --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ --download-scalapack --download-metis=1 --download-superlu=1 --download-superlu_dist=1 --download-hypre=1 --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ [0]PETSC ERROR: #1 PetscObjectTypeCompare() line 145 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/sys/objects/destroy.c [0]PETSC ERROR: #2 PetscLogView() line 1807 in /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/sys/logging/plog.c On Sun, Nov 16, 2014 at 3:57 AM, Matthew Knepley wrote: > On Sat, Nov 15, 2014 at 9:24 PM, Evan Um wrote: > >> Dear PETSC users, >> >> I would like to show you a performance issue when Cholesky factor is >> re-used as a direct solver or pre-conditioner many times with many >> right-hand side vectors. Does anyone suggest a solution about this issue? >> In advance, thanks for your help. >> >> Regards, >> Evan >> >> Example 1: I used MUMPS as a direct solver and measured backward/forward >> substitution solution time for selected RHS vectors. A simplified test code >> is shown in code 1 and time measurements in result 1. As shown below, >> backward and forward solution time is nearly constant (e.g. about 2.2 >> seconds) in early time, but later the solution time overall increases. In >> contrast, in late time, it takes about 12.8 seconds. It is hard to >> understand why the backward/forward solution time gets longer although the >> same numerical operation is carried out with each RHS vector (for i). For >> many RHS vectors, is there any parameter that I need to reset to stably >> take care of lots of RHS vectors? >> >> Example 2: In this case, I use the Cholesky factor as a preconditioner >> for CG solver. One iteration is performed. Its sample code and time >> measurments are shown in Code 2 and Result 2. Again, KSPSolve() gets slower >> as the preconditioner is re-used with many RHS vectors. For example, In >> early stage, it took about 4.6 seconds. Later, it took 16 seconds. Does >> anyone observe such a performance issue? Do you know any solution for this >> problem? >> In the two experiments, I expected Example 2 would show shorter solution >> time with each RHS vector than Example 1 because Example 2 uses scalable >> matrix-vector multiplication. Instead, when MUMPS is used in Example 1, >> MUMPS carries out back/forward substitution that are inherently >> un-scalable. However, my experiments consistently showed that the >> back/forward substitution of MUMPS is faster than a single PCG iteration >> with Cholesky preconditioner. Can anyone explain this? FYI, in both example >> 1 and 2, I used a very large test problem (about sparse SPD >> 4,000,000-by-4,000,000 matrix). >> >> Code 1: >> KSPCreate(PETSC_COMM_WORLD, &ksp_fetd_dt); >> KSPSetOperators(ksp_fetd_dt, A_dt, A_dt); >> KSPSetType (ksp_fetd_dt, KSPPREONLY); >> KSPGetPC(ksp_fetd_dt, &pc_fetd_dt); >> MatSetOption(A_dt, MAT_SPD, PETSC_TRUE); >> PCSetType(pc_fetd_dt, PCCHOLESKY); >> PCFactorSetMatSolverPackage(pc_fetd_dt, MATSOLVERMUMPS); >> PCFactorSetUpMatSolverPackage(pc_fetd_dt); >> PCFactorGetMatrix(pc_fetd_dt, &F_dt); >> for (int i=0; i<1000; i++) { >> // Create a new RHS vector B_dt >> > > Are you calling VecCreate() each time here? If so, you could be filling up > the memory > of your computer as you run, causing it to slow down due to swapping. > > For any performance question, ALWAYS send the output of -log_summary. For > your > question, also divide this loop into log stages for each iteration, so we > can see what > specific operations slow down. > > Matt > > >> time1=my_timer_function(); >> KSPSolve(ksp_fetd_dt,B_dt,solution); >> time2=my_timer_function () >> // Output solution time=time2-time1; >> } >> >> Result1: >> nTS:12 Time(s):2.400e-04 Sol. time(s):2.246e+00 PCG niter:1 >> nTS:13 Time(s):2.600e-04 Sol. time(s):2.329e+00 PCG niter:1 >> nTS:14 Time(s):2.800e-04 Sol. time(s):2.289e+00 PCG niter:1 >> nTS:15 Time(s):3.000e-04 Sol. time(s):2.239e+00 PCG niter:1 >> . >> . >> nTS:267 Time(s):5.340e-03 Sol. time(s):2.152e+00 PCG niter:1 >> nTS:280 Time(s):5.600e-03 Sol. time(s):2.255e+00 PCG niter:1 >> nTS:293 Time(s):5.860e-03 Sol. time(s):1.087e+01 PCG niter:1 >> nTS:307 Time(s):6.140e-03 Sol. time(s):1.225e+01 PCG niter:1 >> nTS:321 Time(s):6.420e-03 Sol. time(s):1.280e+01 PCG niter:1 >> nTS:337 Time(s):6.740e-03 Sol. time(s):1.243e+01 PCG niter:1 >> nTS:353 Time(s):7.060e-03 Sol. time(s):6.953e+00 PCG niter:1 >> nTS:369 Time(s):7.380e-03 Sol. time(s):8.712e+00 PCG niter:1 >> nTS:387 Time(s):7.740e-03 Sol. time(s):9.048e+00 PCG niter:1 >> nTS:405 Time(s):8.100e-03 Sol. time(s):1.099e+01 PCG niter:1 >> nTS:424 Time(s):8.480e-03 Sol. time(s):1.171e+01 PCG niter:1 >> nTS:445 Time(s):8.900e-03 Sol. time(s):1.294e+01 PCG niter:1 >> nTS:466 Time(s):9.320e-03 Sol. time(s):1.227e+01 PCG niter:1 >> nTS:488 Time(s):9.760e-03 Sol. time(s):8.581e+00 PCG niter:1 >> nTS:511 Time(s):1.022e-02 Sol. time(s):1.059e+01 PCG niter:1 >> nTS:535 Time(s):1.070e-02 Sol. time(s):8.452e+00 PCG niter:1 >> >> Code 2: >> KSPCreate(PETSC_COMM_WORLD, &ksp_fetd_dt); >> KSPSetOperators(ksp_fetd_dt, A_dt, A_dt); >> KSPSetType (ksp_fetd_dt, KSPPREONLY); >> KSPGetPC(ksp_fetd_dt, &pc_fetd_dt); >> MatSetOption(A_dt, MAT_SPD, PETSC_TRUE); >> PCSetType(pc_fetd_dt, PCCHOLESKY); >> PCFactorSetMatSolverPackage(pc_fetd_dt, MATSOLVERMUMPS); >> PCFactorSetUpMatSolverPackage(pc_fetd_dt); >> PCFactorGetMatrix(pc_fetd_dt, &F_dt); >> KSPSetType(ksp_fetd_dt, KSPCG); >> KSPSetTolerances(ksp_fetd_dt, 1e-9, 1.0e-50, 1.0e10, ksp_iter); >> for (int i=0; i<1000; i++) { >> // Create a new RHS vector B_dt >> time1=my_timer_function(); >> KSPSolve(ksp_fetd_dt,B_dt,solution); >> time2=my_timer_function () >> // Output solution time=time2-time1; >> } >> >> Result 2: >> nTS:10 Time(s):2.000e-04 Sol. time(s):4.644e+00 PCG niter:1 >> Norm:2.937e-10 >> nTS:11 Time(s):2.200e-04 Sol. time(s):4.585e+00 PCG niter:1 >> Norm:3.151e-10 >> nTS:12 Time(s):2.400e-04 Sol. time(s):4.737e+00 PCG niter:1 >> Norm:2.719e-10 >> nTS:13 Time(s):2.600e-04 Sol. time(s):4.537e+00 PCG niter:1 >> Norm:3.982e-10 >> nTS:14 Time(s):2.800e-04 Sol. time(s):4.578e+00 PCG niter:1 >> Norm:3.221e-10 >> nTS:15 Time(s):3.000e-04 Sol. time(s):4.678e+00 PCG niter:1 >> Norm:3.008e-10 >> nTS:16 Time(s):3.200e-04 Sol. time(s):4.619e+00 PCG niter:1 >> Norm:3.005e-10 >> nTS:46 Time(s):9.200e-04 Sol. time(s):6.862e+00 PCG niter:1 >> Norm:2.759e-10 >> nTS:48 Time(s):9.600e-04 Sol. time(s):4.475e+00 PCG niter:1 >> Norm:3.071e-10 >> nTS:50 Time(s):1.000e-03 Sol. time(s):6.068e+00 PCG niter:1 >> Norm:3.848e-10 >> nTS:52 Time(s):1.040e-03 Sol. time(s):5.024e+00 PCG niter:1 >> Norm:3.639e-10 >> nTS:55 Time(s):1.100e-03 Sol. time(s):1.333e+01 PCG niter:1 >> Norm:3.049e-10 >> nTS:58 Time(s):1.160e-03 Sol. time(s):1.440e+01 PCG niter:1 >> Norm:3.467e-10 >> nTS:60 Time(s):1.200e-03 Sol. time(s):1.475e+01 PCG niter:1 >> Norm:2.951e-10 >> nTS:63 Time(s):1.260e-03 Sol. time(s):9.899e+00 PCG niter:1 >> Norm:3.018e-10 >> nTS:66 Time(s):1.320e-03 Sol. time(s):1.097e+01 PCG niter:1 >> Norm:3.320e-10 >> nTS:69 Time(s):1.380e-03 Sol. time(s):1.485e+01 PCG niter:1 >> Norm:2.811e-10 >> nTS:73 Time(s):1.460e-03 Sol. time(s):1.199e+01 PCG niter:1 >> Norm:2.999e-10 >> nTS:76 Time(s):1.520e-03 Sol. time(s):1.109e+01 PCG niter:1 >> Norm:3.359e-10 >> nTS:80 Time(s):1.600e-03 Sol. time(s):1.473e+01 PCG niter:1 >> Norm:3.336e-10 >> nTS:84 Time(s):1.680e-03 Sol. time(s):1.444e+01 PCG niter:1 >> Norm:3.181e-10 >> nTS:88 Time(s):1.760e-03 Sol. time(s):1.639e+01 PCG niter:1 >> Norm:2.956e-10 >> nTS:92 Time(s):1.840e-03 Sol. time(s):1.713e+01 PCG niter:1 >> Norm:3.552e-10 >> nTS:96 Time(s):1.920e-03 Sol. time(s):1.562e+01 PCG niter:1 >> Norm:2.843e-10 >> nTS:101 Time(s):2.020e-03 Sol. time(s):1.631e+01 PCG niter:1 >> Norm:3.687e-10 >> nTS:105 Time(s):2.100e-03 Sol. time(s):1.496e+01 PCG niter:1 >> Norm:3.567e-10 >> nTS:111 Time(s):2.220e-03 Sol. time(s):1.607e+01 PCG niter:1 >> Norm:3.392e-10 >> >> >> >> >> >> >> >> >> >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Nov 16 10:44:04 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 16 Nov 2014 10:44:04 -0600 Subject: [petsc-users] KSPSolve() get slower when preconditioner or Cholesky factor is re-used with many multiple RHS. In-Reply-To: References: Message-ID: On Sun, Nov 16, 2014 at 10:11 AM, Evan Um wrote: > Dear Matthew, > > Thanks for your comments. I will prepare log summary. To activate an > separate log stage for each iteration (i.e. each RHS vector), I tried the > code below, but got an error. Could you give me a comment? Does > PetscLogView() allow users to use different output file names such that > each iteration generates its own log file? > You use http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Profiling/PetscLogStageRegister.html for each stage, and then http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Profiling/PetscLogStagePush.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Profiling/PetscLogStagePop.html#PetscLogStagePop Matt > Evan > > /* work */ > PetscViewer viewer; > for (int i=0; i<1000; i++) { > PetscLogBegin(); > /* work to do*/ > PetscLogView(viewer); > } > /* work */ > > Errors: > Number of PCG iterations:1 > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Corrupt argument: > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: Invalid type of object: Parameter # 1 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.0, Jun, 30, 2014 > [0]PETSC ERROR: fetdem3dp on a arch-linux2-c-debug named n0024.voltaire0 > by esum Sun Nov 16 08:08:53 2014 > [0]PETSC ERROR: Configure options > --prefix=/clusterfs/voltaire/home/software/modules/petsc/3.5.0 > --download-fblaslapack=1 --download-mumps=1 --download-parmetis=1 > --with-ptscotch=1 > --with-ptscotch-dir=/clusterfs/voltaire/home/software/modules/scotch/5.1.12-gcc/ > --download-scalapack --download-metis=1 --download-superlu=1 > --download-superlu_dist=1 --download-hypre=1 > --with-mpi-dir=/global/software/sl-6.x86_64/modules/gcc/4.4.7/openmpi/1.6.5-gcc/ > [0]PETSC ERROR: #1 PetscObjectTypeCompare() line 145 in > /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/sys/objects/destroy.c > [0]PETSC ERROR: #2 PetscLogView() line 1807 in > /clusterfs/voltaire/home/software/source/petsc-3.5.0/src/sys/logging/plog.c > > > On Sun, Nov 16, 2014 at 3:57 AM, Matthew Knepley > wrote: > >> On Sat, Nov 15, 2014 at 9:24 PM, Evan Um wrote: >> >>> Dear PETSC users, >>> >>> I would like to show you a performance issue when Cholesky factor is >>> re-used as a direct solver or pre-conditioner many times with many >>> right-hand side vectors. Does anyone suggest a solution about this issue? >>> In advance, thanks for your help. >>> >>> Regards, >>> Evan >>> >>> Example 1: I used MUMPS as a direct solver and measured backward/forward >>> substitution solution time for selected RHS vectors. A simplified test code >>> is shown in code 1 and time measurements in result 1. As shown below, >>> backward and forward solution time is nearly constant (e.g. about 2.2 >>> seconds) in early time, but later the solution time overall increases. In >>> contrast, in late time, it takes about 12.8 seconds. It is hard to >>> understand why the backward/forward solution time gets longer although the >>> same numerical operation is carried out with each RHS vector (for i). For >>> many RHS vectors, is there any parameter that I need to reset to stably >>> take care of lots of RHS vectors? >>> >>> Example 2: In this case, I use the Cholesky factor as a preconditioner >>> for CG solver. One iteration is performed. Its sample code and time >>> measurments are shown in Code 2 and Result 2. Again, KSPSolve() gets slower >>> as the preconditioner is re-used with many RHS vectors. For example, In >>> early stage, it took about 4.6 seconds. Later, it took 16 seconds. Does >>> anyone observe such a performance issue? Do you know any solution for this >>> problem? >>> In the two experiments, I expected Example 2 would show shorter solution >>> time with each RHS vector than Example 1 because Example 2 uses scalable >>> matrix-vector multiplication. Instead, when MUMPS is used in Example 1, >>> MUMPS carries out back/forward substitution that are inherently >>> un-scalable. However, my experiments consistently showed that the >>> back/forward substitution of MUMPS is faster than a single PCG iteration >>> with Cholesky preconditioner. Can anyone explain this? FYI, in both example >>> 1 and 2, I used a very large test problem (about sparse SPD >>> 4,000,000-by-4,000,000 matrix). >>> >>> Code 1: >>> KSPCreate(PETSC_COMM_WORLD, &ksp_fetd_dt); >>> KSPSetOperators(ksp_fetd_dt, A_dt, A_dt); >>> KSPSetType (ksp_fetd_dt, KSPPREONLY); >>> KSPGetPC(ksp_fetd_dt, &pc_fetd_dt); >>> MatSetOption(A_dt, MAT_SPD, PETSC_TRUE); >>> PCSetType(pc_fetd_dt, PCCHOLESKY); >>> PCFactorSetMatSolverPackage(pc_fetd_dt, MATSOLVERMUMPS); >>> PCFactorSetUpMatSolverPackage(pc_fetd_dt); >>> PCFactorGetMatrix(pc_fetd_dt, &F_dt); >>> for (int i=0; i<1000; i++) { >>> // Create a new RHS vector B_dt >>> >> >> Are you calling VecCreate() each time here? If so, you could be filling >> up the memory >> of your computer as you run, causing it to slow down due to swapping. >> >> For any performance question, ALWAYS send the output of -log_summary. For >> your >> question, also divide this loop into log stages for each iteration, so we >> can see what >> specific operations slow down. >> >> Matt >> >> >>> time1=my_timer_function(); >>> KSPSolve(ksp_fetd_dt,B_dt,solution); >>> time2=my_timer_function () >>> // Output solution time=time2-time1; >>> } >>> >>> Result1: >>> nTS:12 Time(s):2.400e-04 Sol. time(s):2.246e+00 PCG niter:1 >>> nTS:13 Time(s):2.600e-04 Sol. time(s):2.329e+00 PCG niter:1 >>> nTS:14 Time(s):2.800e-04 Sol. time(s):2.289e+00 PCG niter:1 >>> nTS:15 Time(s):3.000e-04 Sol. time(s):2.239e+00 PCG niter:1 >>> . >>> . >>> nTS:267 Time(s):5.340e-03 Sol. time(s):2.152e+00 PCG niter:1 >>> nTS:280 Time(s):5.600e-03 Sol. time(s):2.255e+00 PCG niter:1 >>> nTS:293 Time(s):5.860e-03 Sol. time(s):1.087e+01 PCG niter:1 >>> nTS:307 Time(s):6.140e-03 Sol. time(s):1.225e+01 PCG niter:1 >>> nTS:321 Time(s):6.420e-03 Sol. time(s):1.280e+01 PCG niter:1 >>> nTS:337 Time(s):6.740e-03 Sol. time(s):1.243e+01 PCG niter:1 >>> nTS:353 Time(s):7.060e-03 Sol. time(s):6.953e+00 PCG niter:1 >>> nTS:369 Time(s):7.380e-03 Sol. time(s):8.712e+00 PCG niter:1 >>> nTS:387 Time(s):7.740e-03 Sol. time(s):9.048e+00 PCG niter:1 >>> nTS:405 Time(s):8.100e-03 Sol. time(s):1.099e+01 PCG niter:1 >>> nTS:424 Time(s):8.480e-03 Sol. time(s):1.171e+01 PCG niter:1 >>> nTS:445 Time(s):8.900e-03 Sol. time(s):1.294e+01 PCG niter:1 >>> nTS:466 Time(s):9.320e-03 Sol. time(s):1.227e+01 PCG niter:1 >>> nTS:488 Time(s):9.760e-03 Sol. time(s):8.581e+00 PCG niter:1 >>> nTS:511 Time(s):1.022e-02 Sol. time(s):1.059e+01 PCG niter:1 >>> nTS:535 Time(s):1.070e-02 Sol. time(s):8.452e+00 PCG niter:1 >>> >>> Code 2: >>> KSPCreate(PETSC_COMM_WORLD, &ksp_fetd_dt); >>> KSPSetOperators(ksp_fetd_dt, A_dt, A_dt); >>> KSPSetType (ksp_fetd_dt, KSPPREONLY); >>> KSPGetPC(ksp_fetd_dt, &pc_fetd_dt); >>> MatSetOption(A_dt, MAT_SPD, PETSC_TRUE); >>> PCSetType(pc_fetd_dt, PCCHOLESKY); >>> PCFactorSetMatSolverPackage(pc_fetd_dt, MATSOLVERMUMPS); >>> PCFactorSetUpMatSolverPackage(pc_fetd_dt); >>> PCFactorGetMatrix(pc_fetd_dt, &F_dt); >>> KSPSetType(ksp_fetd_dt, KSPCG); >>> KSPSetTolerances(ksp_fetd_dt, 1e-9, 1.0e-50, 1.0e10, ksp_iter); >>> for (int i=0; i<1000; i++) { >>> // Create a new RHS vector B_dt >>> time1=my_timer_function(); >>> KSPSolve(ksp_fetd_dt,B_dt,solution); >>> time2=my_timer_function () >>> // Output solution time=time2-time1; >>> } >>> >>> Result 2: >>> nTS:10 Time(s):2.000e-04 Sol. time(s):4.644e+00 PCG niter:1 >>> Norm:2.937e-10 >>> nTS:11 Time(s):2.200e-04 Sol. time(s):4.585e+00 PCG niter:1 >>> Norm:3.151e-10 >>> nTS:12 Time(s):2.400e-04 Sol. time(s):4.737e+00 PCG niter:1 >>> Norm:2.719e-10 >>> nTS:13 Time(s):2.600e-04 Sol. time(s):4.537e+00 PCG niter:1 >>> Norm:3.982e-10 >>> nTS:14 Time(s):2.800e-04 Sol. time(s):4.578e+00 PCG niter:1 >>> Norm:3.221e-10 >>> nTS:15 Time(s):3.000e-04 Sol. time(s):4.678e+00 PCG niter:1 >>> Norm:3.008e-10 >>> nTS:16 Time(s):3.200e-04 Sol. time(s):4.619e+00 PCG niter:1 >>> Norm:3.005e-10 >>> nTS:46 Time(s):9.200e-04 Sol. time(s):6.862e+00 PCG niter:1 >>> Norm:2.759e-10 >>> nTS:48 Time(s):9.600e-04 Sol. time(s):4.475e+00 PCG niter:1 >>> Norm:3.071e-10 >>> nTS:50 Time(s):1.000e-03 Sol. time(s):6.068e+00 PCG niter:1 >>> Norm:3.848e-10 >>> nTS:52 Time(s):1.040e-03 Sol. time(s):5.024e+00 PCG niter:1 >>> Norm:3.639e-10 >>> nTS:55 Time(s):1.100e-03 Sol. time(s):1.333e+01 PCG niter:1 >>> Norm:3.049e-10 >>> nTS:58 Time(s):1.160e-03 Sol. time(s):1.440e+01 PCG niter:1 >>> Norm:3.467e-10 >>> nTS:60 Time(s):1.200e-03 Sol. time(s):1.475e+01 PCG niter:1 >>> Norm:2.951e-10 >>> nTS:63 Time(s):1.260e-03 Sol. time(s):9.899e+00 PCG niter:1 >>> Norm:3.018e-10 >>> nTS:66 Time(s):1.320e-03 Sol. time(s):1.097e+01 PCG niter:1 >>> Norm:3.320e-10 >>> nTS:69 Time(s):1.380e-03 Sol. time(s):1.485e+01 PCG niter:1 >>> Norm:2.811e-10 >>> nTS:73 Time(s):1.460e-03 Sol. time(s):1.199e+01 PCG niter:1 >>> Norm:2.999e-10 >>> nTS:76 Time(s):1.520e-03 Sol. time(s):1.109e+01 PCG niter:1 >>> Norm:3.359e-10 >>> nTS:80 Time(s):1.600e-03 Sol. time(s):1.473e+01 PCG niter:1 >>> Norm:3.336e-10 >>> nTS:84 Time(s):1.680e-03 Sol. time(s):1.444e+01 PCG niter:1 >>> Norm:3.181e-10 >>> nTS:88 Time(s):1.760e-03 Sol. time(s):1.639e+01 PCG niter:1 >>> Norm:2.956e-10 >>> nTS:92 Time(s):1.840e-03 Sol. time(s):1.713e+01 PCG niter:1 >>> Norm:3.552e-10 >>> nTS:96 Time(s):1.920e-03 Sol. time(s):1.562e+01 PCG niter:1 >>> Norm:2.843e-10 >>> nTS:101 Time(s):2.020e-03 Sol. time(s):1.631e+01 PCG niter:1 >>> Norm:3.687e-10 >>> nTS:105 Time(s):2.100e-03 Sol. time(s):1.496e+01 PCG niter:1 >>> Norm:3.567e-10 >>> nTS:111 Time(s):2.220e-03 Sol. time(s):1.607e+01 PCG niter:1 >>> Norm:3.392e-10 >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fd.kong at siat.ac.cn Mon Nov 17 00:18:56 2014 From: fd.kong at siat.ac.cn (Fande Kong) Date: Sun, 16 Nov 2014 23:18:56 -0700 Subject: [petsc-users] matrix assembly Message-ID: Hi all, If I do a matrix assembly with the flag MAT_FLUSH_ASSEMBLY, will it involve any communications across processors? Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Nov 17 00:24:34 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 17 Nov 2014 00:24:34 -0600 Subject: [petsc-users] matrix assembly In-Reply-To: References: Message-ID: Yes, flush assembly moves all the values you have accumulated so far to the correct process so if you set any off process values there will be communication. But this also means that the final assembly will have a corresponding less amount of communication. Barry > On Nov 17, 2014, at 12:18 AM, Fande Kong wrote: > > Hi all, > > If I do a matrix assembly with the flag MAT_FLUSH_ASSEMBLY, will it involve any communications across processors? > > Fande, From jed at jedbrown.org Mon Nov 17 00:25:01 2014 From: jed at jedbrown.org (Jed Brown) Date: Sun, 16 Nov 2014 23:25:01 -0700 Subject: [petsc-users] matrix assembly In-Reply-To: References: Message-ID: <87ppcmibsy.fsf@jedbrown.org> Fande Kong writes: > Hi all, > > If I do a matrix assembly with the flag MAT_FLUSH_ASSEMBLY, will it involve > any communications across processors? Yes, that's the point of this assembly mode. You call it to flush the stash of values that need to be sent to other processes, to prevent it From using too much memory. (Or, when you need to switch between ADD_VALUES and INSERT_VALUES.) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From mfadams at lbl.gov Mon Nov 17 09:12:02 2014 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 17 Nov 2014 10:12:02 -0500 Subject: [petsc-users] Verifying ParMetis used Message-ID: I have a code that repartitions the grid using PETSc. I would like to know the simplest instructions that I can give my users (Pat Worley in this case) to verify that ParMetis is actually called. I had difficulty with this my self in that -info did not seem to say ParMetis was being used. I did verify by printing vizing the partitions that partitioning was taking place. I am thinking I can say use -log_summary and search for download-parmetis -- is there a better way? Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Nov 17 09:46:28 2014 From: hzhang at mcs.anl.gov (Hong) Date: Mon, 17 Nov 2014 09:46:28 -0600 Subject: [petsc-users] Verifying ParMetis used In-Reply-To: References: Message-ID: Mark, I use ParMetis with mumps direct solver. I can test their installation with petsc/src/ksp/ksp/examples/tutorials mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 -ksp_view ... ICNTL(28) (use parallel or sequential ordering): 2 ICNTL(29) (parallel ordering): 2 To know what these parameters are, do mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps -help |grep mumps ... -mat_mumps_icntl_28 <1>: ICNTL(28): use 1 for sequential analysis and ictnl(7) ordering, or 2 for parallel analysis and ictnl(29) ordering (None) -mat_mumps_icntl_29 <0>: ICNTL(29): parallel ordering 1 = ptscotch 2 = parmetis (None) Hong On Mon, Nov 17, 2014 at 9:12 AM, Mark Adams wrote: > I have a code that repartitions the grid using PETSc. I would like to > know the simplest instructions that I can give my users (Pat Worley in this > case) to verify that ParMetis is actually called. I had difficulty with > this my self in that -info did not seem to say ParMetis was being used. I > did verify by printing vizing the partitions that partitioning was taking > place. I am thinking I can say use -log_summary and search for > download-parmetis -- is there a better way? > > Mark > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Nov 17 10:04:01 2014 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 17 Nov 2014 11:04:01 -0500 Subject: [petsc-users] Verifying ParMetis used In-Reply-To: References: Message-ID: Thanks Hong, I call ParMetis from the app code and not from within a solver. Perhaps -info should say "called ParMetis, edgecuts = %d\n", or something like that. Mark On Mon, Nov 17, 2014 at 10:46 AM, Hong wrote: > Mark, > I use ParMetis with mumps direct solver. I can test their installation > with > petsc/src/ksp/ksp/examples/tutorials > mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps > -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 -ksp_view > ... > ICNTL(28) (use parallel or sequential ordering): 2 > ICNTL(29) (parallel ordering): 2 > > To know what these parameters are, do > mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps -help > |grep mumps > ... > -mat_mumps_icntl_28 <1>: ICNTL(28): use 1 for sequential analysis and > ictnl(7) ordering, or 2 for parallel analysis and ictnl(29) ordering (None) > -mat_mumps_icntl_29 <0>: ICNTL(29): parallel ordering 1 = ptscotch 2 = > parmetis (None) > > Hong > > On Mon, Nov 17, 2014 at 9:12 AM, Mark Adams wrote: > >> I have a code that repartitions the grid using PETSc. I would like to >> know the simplest instructions that I can give my users (Pat Worley in this >> case) to verify that ParMetis is actually called. I had difficulty with >> this my self in that -info did not seem to say ParMetis was being used. I >> did verify by printing vizing the partitions that partitioning was taking >> place. I am thinking I can say use -log_summary and search for >> download-parmetis -- is there a better way? >> >> Mark >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Nov 17 10:22:13 2014 From: hzhang at mcs.anl.gov (Hong) Date: Mon, 17 Nov 2014 10:22:13 -0600 Subject: [petsc-users] Verifying ParMetis used In-Reply-To: References: Message-ID: Mark : > > I call ParMetis from the app code and not from within a solver. Perhaps > -info should say "called ParMetis, edgecuts = %d\n", or something like that. > Info like this needs to be put in petsc/ParMetis interface. We do not have it, do we? I use mumps/parmetis, thus have to get parmetis info via mumps. Hong > Mark > > On Mon, Nov 17, 2014 at 10:46 AM, Hong wrote: > >> Mark, >> I use ParMetis with mumps direct solver. I can test their installation >> with >> petsc/src/ksp/ksp/examples/tutorials >> mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps >> -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 -ksp_view >> ... >> ICNTL(28) (use parallel or sequential ordering): 2 >> ICNTL(29) (parallel ordering): 2 >> >> To know what these parameters are, do >> mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps -help >> |grep mumps >> ... >> -mat_mumps_icntl_28 <1>: ICNTL(28): use 1 for sequential analysis and >> ictnl(7) ordering, or 2 for parallel analysis and ictnl(29) ordering (None) >> -mat_mumps_icntl_29 <0>: ICNTL(29): parallel ordering 1 = ptscotch 2 = >> parmetis (None) >> >> Hong >> >> On Mon, Nov 17, 2014 at 9:12 AM, Mark Adams wrote: >> >>> I have a code that repartitions the grid using PETSc. I would like to >>> know the simplest instructions that I can give my users (Pat Worley in this >>> case) to verify that ParMetis is actually called. I had difficulty with >>> this my self in that -info did not seem to say ParMetis was being used. I >>> did verify by printing vizing the partitions that partitioning was taking >>> place. I am thinking I can say use -log_summary and search for >>> download-parmetis -- is there a better way? >>> >>> Mark >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Nov 17 10:28:28 2014 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 17 Nov 2014 17:28:28 +0100 Subject: [petsc-users] Verifying ParMetis used In-Reply-To: References: Message-ID: <12EABACA-68BC-4ED4-8532-39024D366218@dsic.upv.es> Are you using it via MatPartitioning? Then use -mat_partitioning_view Jose El 17/11/2014, a las 17:22, Hong escribi?: > Mark : > I call ParMetis from the app code and not from within a solver. Perhaps -info should say "called ParMetis, edgecuts = %d\n", or something like that. > > Info like this needs to be put in petsc/ParMetis interface. > We do not have it, do we? > I use mumps/parmetis, thus have to get parmetis info via mumps. > > Hong > > Mark > > On Mon, Nov 17, 2014 at 10:46 AM, Hong wrote: > Mark, > I use ParMetis with mumps direct solver. I can test their installation with > petsc/src/ksp/ksp/examples/tutorials > mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 -ksp_view > ... > ICNTL(28) (use parallel or sequential ordering): 2 > ICNTL(29) (parallel ordering): 2 > > To know what these parameters are, do > mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps -help |grep mumps > ... > -mat_mumps_icntl_28 <1>: ICNTL(28): use 1 for sequential analysis and ictnl(7) ordering, or 2 for parallel analysis and ictnl(29) ordering (None) > -mat_mumps_icntl_29 <0>: ICNTL(29): parallel ordering 1 = ptscotch 2 = parmetis (None) > > Hong > > On Mon, Nov 17, 2014 at 9:12 AM, Mark Adams wrote: > I have a code that repartitions the grid using PETSc. I would like to know the simplest instructions that I can give my users (Pat Worley in this case) to verify that ParMetis is actually called. I had difficulty with this my self in that -info did not seem to say ParMetis was being used. I did verify by printing vizing the partitions that partitioning was taking place. I am thinking I can say use -log_summary and search for download-parmetis -- is there a better way? > > Mark > > > From bsmith at mcs.anl.gov Mon Nov 17 10:42:46 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 17 Nov 2014 10:42:46 -0600 Subject: [petsc-users] Verifying ParMetis used In-Reply-To: References: Message-ID: <0ECA11BC-0792-4C14-B765-9F6CA8078733@mcs.anl.gov> MatPartitioningView() ? > On Nov 17, 2014, at 9:12 AM, Mark Adams wrote: > > I have a code that repartitions the grid using PETSc. I would like to know the simplest instructions that I can give my users (Pat Worley in this case) to verify that ParMetis is actually called. I had difficulty with this my self in that -info did not seem to say ParMetis was being used. I did verify by printing vizing the partitions that partitioning was taking place. I am thinking I can say use -log_summary and search for download-parmetis -- is there a better way? > > Mark From leoni.massimiliano1 at gmail.com Mon Nov 17 16:27:57 2014 From: leoni.massimiliano1 at gmail.com (Massimiliano Leoni) Date: Mon, 17 Nov 2014 23:27:57 +0100 Subject: [petsc-users] Roles of ksp_type and pc_type In-Reply-To: <1928038.ts4lkFBGPS@debianxps> References: <2594845.ya19CE4v59@debianxps> <1928038.ts4lkFBGPS@debianxps> Message-ID: <4302011.cIgiPZOLR6@debianxps> Hi again, I've been experimenting further with fieldsplit, following your slides, but for some reason -pc_fieldsplit_type multiplicative and schur segfault at KSPSolve. Additive seems to work fine. I could not reproduce ex62 as in your slides because I don't have the header file and the script PetscGenerateFEMQuadrature.py, which I downloaded from the github repo, complains that PETSC.FEM module is missing, so I used my code, which is supopsd to assemble the same matrix, then I call KSPSetOperators with Stokes matrix as both the matrix and the preconditioner. I use the same options as in slide 115, the "block jacobi, inexact" version works, while the "block Gauss-seidl, inexact" doesn't and segfaults. I attach backtrace and log files, according to what I was told last time. Thanks for any help, Massimiliano In data gioved? 13 novembre 2014 15:07:40, Massimiliano Leoni ha scritto: > In data gioved? 13 novembre 2014 06:39:38, Matthew Knepley ha scritto: > > On Thu, Nov 13, 2014 at 6:24 AM, Massimiliano Leoni < > > > > This is not exactly right because here you are just using additive > > fieldsplit. > > AFAI understood from the user manual, this should be right because my > preconditioner is block diagonal, so I want to use block Jacobi. > > If I used the other one, P = [[C,O],[B,Mp/nu]] then I would need a > multiplicative fieldsplit because this would be lower triangular. > > Is this correct? > > > The mass matrix > > is a good preconditioner for the Schur complement, not the zero matrix. > > > Take a look at these slides: > What do you mean? > I know from theory that the preconditioner I am using is optimal [iteration > number independent of grid size]. > > I read the slides and the manual and the Schur complement is what I was > talking about earlier, so it's not what I wanna do now -- even though > knowing that it's so easy to implement my be very useful. > > Anyway, thanks for the slides, they have been enlightening on the power of > command line options! > > > Thanks, > > > > Matt > > Thanks again, > Massimiliano -------------- next part -------------- (gdb) bt full #0 0x00007ffff3c1d8d8 in PCApply_FieldSplit(_p_PC*, _p_Vec*, _p_Vec*) () from /usr/lib/petscdir/3.4.2/linux-gnu-c-opt/lib/libpetsc.so.3.4.2 No symbol table info available. #1 0x00007ffff3ec407e in PCApply () from /usr/lib/petscdir/3.4.2/linux-gnu-c-opt/lib/libpetsc.so.3.4.2 No symbol table info available. #2 0x00007ffff3c17a25 in KSPFGMRESCycle(int*, _p_KSP*) () from /usr/lib/petscdir/3.4.2/linux-gnu-c-opt/lib/libpetsc.so.3.4.2 No symbol table info available. #3 0x00007ffff3c18478 in KSPSolve_FGMRES(_p_KSP*) () from /usr/lib/petscdir/3.4.2/linux-gnu-c-opt/lib/libpetsc.so.3.4.2 No symbol table info available. #4 0x00007ffff3ccab51 in KSPSolve () from /usr/lib/petscdir/3.4.2/linux-gnu-c-opt/lib/libpetsc.so.3.4.2 No symbol table info available. #5 0x00007ffff7bc52e9 in SSolver::solve (this=this at entry=0x7fffffffd9e0) at .../src/SSolver.cpp:57 solVec = 0xf12400 bVec = 0x12efb40 #6 0x0000000000402a60 in main (argc=, argv=) at .../apps/Stokes/cavity.cpp:111 resolution = 50 ufile = {_mpi_comm = 0x100000000, file = std::unique_ptr containing 0x100002b9d} u1 = mesh = zero_vector = noslip_domain = { = {}, } f = p1 = cavity = u_top = { = {}, } top_domain = { = {}, } p = bcu = std::vector of length 2, capacity 2 = {{first = 0x7fffffffd0c0, second = 0x7fffffffce80}, {first = 0x7fffffffcf50, second = 0x7fffffffce60}} nu = solver = {_vptr.SSolver = 0x4037d0 , mesh = , W = 0x6ece60, V = 0xb35980, bcu = std::vector of length 2, capacity 2 = {0xf13520, 0xf13b00}, a = 0xf11850, F = 0xf119e0, forcing = 0xf0b020, viscosity = 0xf11260, A = std::shared_ptr (count 2, weak 0) 0xf11bd0, b = 0xf11d40, P = std::shared_ptr (count 2, weak 0) 0xf11bd0, solution = 0xf12030, Prec = 0x0, genericSolver = 0xee2550} pfile = {_mpi_comm = 0x100000000, file = std::unique_ptr containing 0x0} -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: text/x-log Size: 52173 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log.zip Type: application/zip Size: 211738 bytes Desc: not available URL: From salazardetroya at gmail.com Tue Nov 18 11:19:41 2014 From: salazardetroya at gmail.com (Miguel Angel Salazar de Troya) Date: Tue, 18 Nov 2014 11:19:41 -0600 Subject: [petsc-users] PetscFunctionBegin, -malloc_dump and C++ classes with PETSc objects Message-ID: Hi I'm implementing a problem using the TS. Most of my functions are methods inside of a class, except for the callbacks (to form the RHS and the TS monitor), which are outside of the class, although in the same .C file where the class methods are implemented. For these callbacks I followed the network example: https://bitbucket.org/petsc/petsc/src/a614f7369d93d476173b8fc6bf2463276dcbdb3a/src/snes/examples/tutorials/network/pflow/pf.c?at=master Therefore, the callbacks have the PetscFunctionBegin at the beginning and PetscFunctionReturn(0) at the end. My problems come when I run the program with -malloc_dump and I get a lot of unfreed memory. Inspecting the output I see that the line of my code where the memory is allocated corresponds with the line when PetscFunctionBegin is called. Later in the file, I see that the function DMGetLocalVector() is called within a petsc internal routine (at the file dmget.c). I also call this routine in my callback methods few lines after PetscFunctionBegin. The procedure that I follow to use the local vectors is as the one in the network example. For vectors that I want to modify this is: ierr = DMGetLocalVector(networkdm,&localX);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); ierr = VecGetArray(localX,&xarr);CHKERRQ(ierr); Modify values in xarr ierr = VecRestoreArray(localX,&xarr);CHKERRQ(ierr); ierr = DMLocalToGlobalBegin(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); ierr = DMLocalToGlobalEnd(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); ierr = DMRestoreLocalVector(networkdm,&localX);CHKERRQ(ierr); One last thing that I think it might be a issue here is how I destroy the petsc objects. I create the petsc objects within a class. For instance, the class has a petsc vector that later passes to the TS object to get the solution. To destroy the petsc objects, I use the class destructor, where at the end I call PetscFinalize() Inside the class I pass the callbacks to the TS routines that need them (e.g. TSSetRHSFunction() ) I can compile the code and run it, but many memory allocations are not freed. What can be the issue here? Do you know of an example using C++ classes to implement PETSc methods? Thanks in advance. Miguel -- *Miguel Angel Salazar de Troya* Graduate Research Assistant Department of Mechanical Science and Engineering University of Illinois at Urbana-Champaign (217) 550-2360 salaza11 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Nov 18 11:32:34 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 18 Nov 2014 11:32:34 -0600 Subject: [petsc-users] PetscFunctionBegin, -malloc_dump and C++ classes with PETSc objects In-Reply-To: References: Message-ID: <6256BDCD-19E6-48F5-9872-E5DB97C981EA@mcs.anl.gov> > On Nov 18, 2014, at 11:19 AM, Miguel Angel Salazar de Troya wrote: > > Hi > > I'm implementing a problem using the TS. Most of my functions are methods inside of a class, except for the callbacks (to form the RHS and the TS monitor), which are outside of the class, although in the same .C file where the class methods are implemented. For these callbacks I followed the network example: > > https://bitbucket.org/petsc/petsc/src/a614f7369d93d476173b8fc6bf2463276dcbdb3a/src/snes/examples/tutorials/network/pflow/pf.c?at=master > > Therefore, the callbacks have the PetscFunctionBegin at the beginning and PetscFunctionReturn(0) at the end. My problems come when I run the program with -malloc_dump and I get a lot of unfreed memory. Inspecting the output I see that the line of my code where the memory is allocated corresponds with the line when PetscFunctionBegin is called. This is normal. We cannot register the exact line the memory allocated, only the location of the PETScFunctionBegin; > Later in the file, I see that the function DMGetLocalVector() is called within a petsc internal routine (at the file dmget.c). I also call this routine in my callback methods few lines after PetscFunctionBegin. The procedure that I follow to use the local vectors is as the one in the network example. For vectors that I want to modify this is: > > ierr = DMGetLocalVector(networkdm,&localX);CHKERRQ(ierr); > ierr = DMGlobalToLocalBegin(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); > ierr = DMGlobalToLocalEnd(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); > ierr = VecGetArray(localX,&xarr);CHKERRQ(ierr); > > Modify values in xarr > > ierr = VecRestoreArray(localX,&xarr);CHKERRQ(ierr); > ierr = DMLocalToGlobalBegin(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); > ierr = DMLocalToGlobalEnd(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); > ierr = DMRestoreLocalVector(networkdm,&localX);CHKERRQ(ierr); > > One last thing that I think it might be a issue here is how I destroy the petsc objects. I create the petsc objects within a class. For instance, the class has a petsc vector that later passes to the TS object to get the solution. To destroy the petsc objects, I use the class destructor, where at the end I call PetscFinalize() Inside the class I pass the callbacks to the TS routines that need them (e.g. TSSetRHSFunction() ) I can compile the code and run it, but many memory allocations are not freed. What can be the issue here? Do you know of an example using C++ classes to implement PETSc methods? Thanks in advance. Do you call the class destructor yourself explicitly before the PetscFinalize()? You need to, otherwise the class may not be destroyed until after PetscFinalize() and hence the PETSc objects won't be freed when you call PetscFinalize(). You also need to make sure that you destroy ALL PETSc objects, if you miss even one PETSc object, since the objects have references to each other it may be that many PETSc objects do not get freed and hence -malloc_dump shows many objects still alive. Barry > > Miguel > > -- > Miguel Angel Salazar de Troya > Graduate Research Assistant > Department of Mechanical Science and Engineering > University of Illinois at Urbana-Champaign > (217) 550-2360 > salaza11 at illinois.edu > From salazardetroya at gmail.com Tue Nov 18 11:54:42 2014 From: salazardetroya at gmail.com (Miguel Angel Salazar de Troya) Date: Tue, 18 Nov 2014 11:54:42 -0600 Subject: [petsc-users] PetscFunctionBegin, -malloc_dump and C++ classes with PETSc objects In-Reply-To: <6256BDCD-19E6-48F5-9872-E5DB97C981EA@mcs.anl.gov> References: <6256BDCD-19E6-48F5-9872-E5DB97C981EA@mcs.anl.gov> Message-ID: PetscFinalize() is inside of the class destructor (the last line of the destructor), so when the object goes out of scope, the class destructor is called and PetscFinalize() as well. Is it better to have PetscFinalize() outside of the destructor and call the destructor explicitly before? Miguel On Tue, Nov 18, 2014 at 11:32 AM, Barry Smith wrote: > > > On Nov 18, 2014, at 11:19 AM, Miguel Angel Salazar de Troya < > salazardetroya at gmail.com> wrote: > > > > Hi > > > > I'm implementing a problem using the TS. Most of my functions are > methods inside of a class, except for the callbacks (to form the RHS and > the TS monitor), which are outside of the class, although in the same .C > file where the class methods are implemented. For these callbacks I > followed the network example: > > > > > https://bitbucket.org/petsc/petsc/src/a614f7369d93d476173b8fc6bf2463276dcbdb3a/src/snes/examples/tutorials/network/pflow/pf.c?at=master > > > > Therefore, the callbacks have the PetscFunctionBegin at the beginning > and PetscFunctionReturn(0) at the end. My problems come when I run the > program with -malloc_dump and I get a lot of unfreed memory. Inspecting the > output I see that the line of my code where the memory is allocated > corresponds with the line when PetscFunctionBegin is called. > > This is normal. We cannot register the exact line the memory allocated, > only the location of the PETScFunctionBegin; > > > > Later in the file, I see that the function DMGetLocalVector() is called > within a petsc internal routine (at the file dmget.c). I also call this > routine in my callback methods few lines after PetscFunctionBegin. The > procedure that I follow to use the local vectors is as the one in the > network example. For vectors that I want to modify this is: > > > > ierr = DMGetLocalVector(networkdm,&localX);CHKERRQ(ierr); > > ierr = > DMGlobalToLocalBegin(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); > > ierr = > DMGlobalToLocalEnd(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); > > ierr = VecGetArray(localX,&xarr);CHKERRQ(ierr); > > > > Modify values in xarr > > > > ierr = VecRestoreArray(localX,&xarr);CHKERRQ(ierr); > > ierr = > DMLocalToGlobalBegin(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); > > ierr = > DMLocalToGlobalEnd(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); > > ierr = DMRestoreLocalVector(networkdm,&localX);CHKERRQ(ierr); > > > > One last thing that I think it might be a issue here is how I destroy > the petsc objects. I create the petsc objects within a class. For instance, > the class has a petsc vector that later passes to the TS object to get the > solution. To destroy the petsc objects, I use the class destructor, where > at the end I call PetscFinalize() Inside the class I pass the callbacks to > the TS routines that need them (e.g. TSSetRHSFunction() ) I can compile the > code and run it, but many memory allocations are not freed. What can be the > issue here? Do you know of an example using C++ classes to implement PETSc > methods? Thanks in advance. > > Do you call the class destructor yourself explicitly before the > PetscFinalize()? You need to, otherwise the class may not be destroyed > until after PetscFinalize() and hence the PETSc objects won't be freed when > you call PetscFinalize(). > > You also need to make sure that you destroy ALL PETSc objects, if you > miss even one PETSc object, since the objects have references to each other > it may be that many PETSc objects do not get freed and hence -malloc_dump > shows many objects still alive. > > Barry > > > > > Miguel > > > > -- > > Miguel Angel Salazar de Troya > > Graduate Research Assistant > > Department of Mechanical Science and Engineering > > University of Illinois at Urbana-Champaign > > (217) 550-2360 > > salaza11 at illinois.edu > > > > -- *Miguel Angel Salazar de Troya* Graduate Research Assistant Department of Mechanical Science and Engineering University of Illinois at Urbana-Champaign (217) 550-2360 salaza11 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Nov 18 12:54:30 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 18 Nov 2014 12:54:30 -0600 Subject: [petsc-users] PetscFunctionBegin, -malloc_dump and C++ classes with PETSc objects In-Reply-To: References: <6256BDCD-19E6-48F5-9872-E5DB97C981EA@mcs.anl.gov> Message-ID: <382BFA94-5183-4D5E-ADFE-306FCD43113A@mcs.anl.gov> > On Nov 18, 2014, at 11:54 AM, Miguel Angel Salazar de Troya wrote: > > PetscFinalize() is inside of the class destructor (the last line of the destructor), so when the object goes out of scope, the class destructor is called and PetscFinalize() as well. Is it better to have PetscFinalize() outside of the destructor and call the destructor explicitly before? It shouldn't really matter. My guess is you must be missing calling a destroy or restore on one of the PETSc objects. Barry > > Miguel > > On Tue, Nov 18, 2014 at 11:32 AM, Barry Smith wrote: > > > On Nov 18, 2014, at 11:19 AM, Miguel Angel Salazar de Troya wrote: > > > > Hi > > > > I'm implementing a problem using the TS. Most of my functions are methods inside of a class, except for the callbacks (to form the RHS and the TS monitor), which are outside of the class, although in the same .C file where the class methods are implemented. For these callbacks I followed the network example: > > > > https://bitbucket.org/petsc/petsc/src/a614f7369d93d476173b8fc6bf2463276dcbdb3a/src/snes/examples/tutorials/network/pflow/pf.c?at=master > > > > Therefore, the callbacks have the PetscFunctionBegin at the beginning and PetscFunctionReturn(0) at the end. My problems come when I run the program with -malloc_dump and I get a lot of unfreed memory. Inspecting the output I see that the line of my code where the memory is allocated corresponds with the line when PetscFunctionBegin is called. > > This is normal. We cannot register the exact line the memory allocated, only the location of the PETScFunctionBegin; > > > > Later in the file, I see that the function DMGetLocalVector() is called within a petsc internal routine (at the file dmget.c). I also call this routine in my callback methods few lines after PetscFunctionBegin. The procedure that I follow to use the local vectors is as the one in the network example. For vectors that I want to modify this is: > > > > ierr = DMGetLocalVector(networkdm,&localX);CHKERRQ(ierr); > > ierr = DMGlobalToLocalBegin(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); > > ierr = DMGlobalToLocalEnd(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); > > ierr = VecGetArray(localX,&xarr);CHKERRQ(ierr); > > > > Modify values in xarr > > > > ierr = VecRestoreArray(localX,&xarr);CHKERRQ(ierr); > > ierr = DMLocalToGlobalBegin(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); > > ierr = DMLocalToGlobalEnd(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); > > ierr = DMRestoreLocalVector(networkdm,&localX);CHKERRQ(ierr); > > > > One last thing that I think it might be a issue here is how I destroy the petsc objects. I create the petsc objects within a class. For instance, the class has a petsc vector that later passes to the TS object to get the solution. To destroy the petsc objects, I use the class destructor, where at the end I call PetscFinalize() Inside the class I pass the callbacks to the TS routines that need them (e.g. TSSetRHSFunction() ) I can compile the code and run it, but many memory allocations are not freed. What can be the issue here? Do you know of an example using C++ classes to implement PETSc methods? Thanks in advance. > > Do you call the class destructor yourself explicitly before the PetscFinalize()? You need to, otherwise the class may not be destroyed until after PetscFinalize() and hence the PETSc objects won't be freed when you call PetscFinalize(). > > You also need to make sure that you destroy ALL PETSc objects, if you miss even one PETSc object, since the objects have references to each other it may be that many PETSc objects do not get freed and hence -malloc_dump shows many objects still alive. > > Barry > > > > > Miguel > > > > -- > > Miguel Angel Salazar de Troya > > Graduate Research Assistant > > Department of Mechanical Science and Engineering > > University of Illinois at Urbana-Champaign > > (217) 550-2360 > > salaza11 at illinois.edu > > > > > > > -- > Miguel Angel Salazar de Troya > Graduate Research Assistant > Department of Mechanical Science and Engineering > University of Illinois at Urbana-Champaign > (217) 550-2360 > salaza11 at illinois.edu > From dave.mayhem23 at gmail.com Tue Nov 18 13:49:25 2014 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 18 Nov 2014 20:49:25 +0100 Subject: [petsc-users] PetscQuadratureCreate and PETSC_OBJECT_CLASSID Message-ID: Hi, I was trying to use some of DT functionality (in particular PetscDTGaussTensorQuadrature()) in petsc 3.5.2 and I encountered this error: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Invalid object classid 0 This could happen if you compile with PETSC_HAVE_DYNAMIC_LIBRARIES, but link with static libraries. [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 [0]PETSC ERROR: ./dmdadg.app on a arch-darwin-c-mpi-debug-f2c named nikkan.local by dmay Tue Nov 18 20:34:56 2014 [0]PETSC ERROR: Configure options --download-openmpi=yes --with-debugging=yes --with-fc=0 --download-f2cblaslapck=yes PETSC_ARCH=arch-darwin-c-mpi-debug-f2c [0]PETSC ERROR: #1 PetscClassRegLogGetClass() line 290 in /Users/dmay/software/petsc-3.5.2/src/sys/logging/utils/classlog.c [0]PETSC ERROR: #2 PetscLogObjCreateDefault() line 317 in /Users/dmay/software/petsc-3.5.2/src/sys/logging/utils/classlog.c [0]PETSC ERROR: #3 PetscQuadratureCreate() line 35 in /Users/dmay/software/petsc-3.5.2/src/dm/dt/interface/dt.c [0]PETSC ERROR: #4 PetscDTGaussTensorQuadrature() line 322 in /Users/dmay/software/petsc-3.5.2/src/dm/dt/interface/dt.c [0]PETSC ERROR: #5 DMDGSpace_ex4_quadrature() line 480 in dmdadg.c [0]PETSC ERROR: #6 main() line 503 in dmdadg.c [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 1. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- Snooping around in the source tree, the only place where I can find PetscHeaderCreate being passed PETSC_OBJECT_CLASSID is within src/dm/dt/interface/dt.c in the function PetscQuadratureCreate(). Is this is a bug or is my petsc build broken?? I'm sure I am linking against a dynamic library (or whatever you want to call a dylib on OSX) dmay at nikkan:~/codes/dgspace$ otool -L dmdadg.app dmdadg.app: * /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libpetsc.3.5.dylib (compatibility version 3.5.0, current version 3.5.2)* /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib (compatibility version 1.0.0, current version 1.0.0) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib (compatibility version 1.0.0, current version 1.0.0) /opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current version 10.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1) /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 50.0.0) /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 50.0.0) /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libmpi_cxx.1.dylib (compatibility version 3.0.0, current version 3.3.0) /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0) /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libmpi.1.dylib (compatibility version 7.0.0, current version 7.0.0) Cheers, Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Nov 18 14:24:51 2014 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 18 Nov 2014 15:24:51 -0500 Subject: [petsc-users] Verifying ParMetis used In-Reply-To: <0ECA11BC-0792-4C14-B765-9F6CA8078733@mcs.anl.gov> References: <0ECA11BC-0792-4C14-B765-9F6CA8078733@mcs.anl.gov> Message-ID: Ah yes, -mat_partitioning_view, Thanks, Mark On Mon, Nov 17, 2014 at 11:42 AM, Barry Smith wrote: > > MatPartitioningView() ? > > > > On Nov 17, 2014, at 9:12 AM, Mark Adams wrote: > > > > I have a code that repartitions the grid using PETSc. I would like to > know the simplest instructions that I can give my users (Pat Worley in this > case) to verify that ParMetis is actually called. I had difficulty with > this my self in that -info did not seem to say ParMetis was being used. I > did verify by printing vizing the partitions that partitioning was taking > place. I am thinking I can say use -log_summary and search for > download-parmetis -- is there a better way? > > > > Mark > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 18 15:46:23 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 Nov 2014 15:46:23 -0600 Subject: [petsc-users] PetscQuadratureCreate and PETSC_OBJECT_CLASSID In-Reply-To: References: Message-ID: On Tue, Nov 18, 2014 at 1:49 PM, Dave May wrote: > Hi, > > I was trying to use some of DT functionality (in particular > PetscDTGaussTensorQuadrature()) in petsc 3.5.2 and I encountered this error: > I did not anticipate someone calling that before anything by itself. You need a call to DMInitializePackage() This is called by PetscQuadratureCreate(). Matt > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Invalid object classid 0 > This could happen if you compile with PETSC_HAVE_DYNAMIC_LIBRARIES, but > link with static libraries. > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > [0]PETSC ERROR: ./dmdadg.app on a arch-darwin-c-mpi-debug-f2c named > nikkan.local by dmay Tue Nov 18 20:34:56 2014 > [0]PETSC ERROR: Configure options --download-openmpi=yes > --with-debugging=yes --with-fc=0 --download-f2cblaslapck=yes > PETSC_ARCH=arch-darwin-c-mpi-debug-f2c > [0]PETSC ERROR: #1 PetscClassRegLogGetClass() line 290 in > /Users/dmay/software/petsc-3.5.2/src/sys/logging/utils/classlog.c > [0]PETSC ERROR: #2 PetscLogObjCreateDefault() line 317 in > /Users/dmay/software/petsc-3.5.2/src/sys/logging/utils/classlog.c > [0]PETSC ERROR: #3 PetscQuadratureCreate() line 35 in > /Users/dmay/software/petsc-3.5.2/src/dm/dt/interface/dt.c > [0]PETSC ERROR: #4 PetscDTGaussTensorQuadrature() line 322 in > /Users/dmay/software/petsc-3.5.2/src/dm/dt/interface/dt.c > [0]PETSC ERROR: #5 DMDGSpace_ex4_quadrature() line 480 in dmdadg.c > [0]PETSC ERROR: #6 main() line 503 in dmdadg.c > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 1. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > > Snooping around in the source tree, the only place where I can find > PetscHeaderCreate being passed PETSC_OBJECT_CLASSID is within > src/dm/dt/interface/dt.c in the function PetscQuadratureCreate(). > > Is this is a bug or is my petsc build broken?? > > I'm sure I am linking against a dynamic library (or whatever you want to > call a dylib on OSX) > > dmay at nikkan:~/codes/dgspace$ otool -L dmdadg.app > dmdadg.app: > * > /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libpetsc.3.5.dylib > (compatibility version 3.5.0, current version 3.5.2)* > > /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib > (compatibility version 1.0.0, current version 1.0.0) > > /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib > (compatibility version 1.0.0, current version 1.0.0) > /opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current > version 10.0.0) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current > version 1197.1.1) > /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current > version 50.0.0) > /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current > version 50.0.0) > > /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libmpi_cxx.1.dylib > (compatibility version 3.0.0, current version 3.3.0) > /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version > 120.0.0) > > /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libmpi.1.dylib > (compatibility version 7.0.0, current version 7.0.0) > > > > Cheers, > Dave > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Nov 18 15:51:44 2014 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 18 Nov 2014 22:51:44 +0100 Subject: [petsc-users] PetscQuadratureCreate and PETSC_OBJECT_CLASSID In-Reply-To: References: Message-ID: Okay, thanks Matt. It now makes sense why it failed, but I don't understand your logic. The PetscQuadrature class is so small and doesn't depend on DM (at least not in the definition of the structure described PetscQuadrature). Hence, why should DMInitializePackage() have to be called at all? I realize you how are you using PetscDT, but it should be available for everyone - even if they don't ever create a DM. Cheers, Dave On 18 November 2014 22:46, Matthew Knepley wrote: > On Tue, Nov 18, 2014 at 1:49 PM, Dave May wrote: > >> Hi, >> >> I was trying to use some of DT functionality (in particular >> PetscDTGaussTensorQuadrature()) in petsc 3.5.2 and I encountered this error: >> > > I did not anticipate someone calling that before anything by itself. You > need a call to > > DMInitializePackage() > > This is called by PetscQuadratureCreate(). > > Matt > > >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Invalid argument >> [0]PETSC ERROR: Invalid object classid 0 >> This could happen if you compile with PETSC_HAVE_DYNAMIC_LIBRARIES, but >> link with static libraries. >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >> [0]PETSC ERROR: ./dmdadg.app on a arch-darwin-c-mpi-debug-f2c named >> nikkan.local by dmay Tue Nov 18 20:34:56 2014 >> [0]PETSC ERROR: Configure options --download-openmpi=yes >> --with-debugging=yes --with-fc=0 --download-f2cblaslapck=yes >> PETSC_ARCH=arch-darwin-c-mpi-debug-f2c >> [0]PETSC ERROR: #1 PetscClassRegLogGetClass() line 290 in >> /Users/dmay/software/petsc-3.5.2/src/sys/logging/utils/classlog.c >> [0]PETSC ERROR: #2 PetscLogObjCreateDefault() line 317 in >> /Users/dmay/software/petsc-3.5.2/src/sys/logging/utils/classlog.c >> [0]PETSC ERROR: #3 PetscQuadratureCreate() line 35 in >> /Users/dmay/software/petsc-3.5.2/src/dm/dt/interface/dt.c >> [0]PETSC ERROR: #4 PetscDTGaussTensorQuadrature() line 322 in >> /Users/dmay/software/petsc-3.5.2/src/dm/dt/interface/dt.c >> [0]PETSC ERROR: #5 DMDGSpace_ex4_quadrature() line 480 in dmdadg.c >> [0]PETSC ERROR: #6 main() line 503 in dmdadg.c >> [0]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint at mcs.anl.gov---------- >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 1. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -------------------------------------------------------------------------- >> >> Snooping around in the source tree, the only place where I can find >> PetscHeaderCreate being passed PETSC_OBJECT_CLASSID is within >> src/dm/dt/interface/dt.c in the function PetscQuadratureCreate(). >> >> Is this is a bug or is my petsc build broken?? >> >> I'm sure I am linking against a dynamic library (or whatever you want to >> call a dylib on OSX) >> >> dmay at nikkan:~/codes/dgspace$ otool -L dmdadg.app >> dmdadg.app: >> * >> /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libpetsc.3.5.dylib >> (compatibility version 3.5.0, current version 3.5.2)* >> >> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib >> (compatibility version 1.0.0, current version 1.0.0) >> >> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib >> (compatibility version 1.0.0, current version 1.0.0) >> /opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current >> version 10.0.0) >> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current >> version 1197.1.1) >> /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current >> version 50.0.0) >> /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current >> version 50.0.0) >> >> /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libmpi_cxx.1.dylib >> (compatibility version 3.0.0, current version 3.3.0) >> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version >> 120.0.0) >> >> /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libmpi.1.dylib >> (compatibility version 7.0.0, current version 7.0.0) >> >> >> >> Cheers, >> Dave >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Nov 18 16:09:04 2014 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 18 Nov 2014 23:09:04 +0100 Subject: [petsc-users] PetscQuadratureCreate and PETSC_OBJECT_CLASSID In-Reply-To: References: Message-ID: Okay - maybe you meant PetscSysInitializePackage() needs to get first? In this simple test I was running, I essentially called PetscInitialize PetscDTGaussTensorQuadrature PetscFinalize and thus no one call PetscSysInitializePackage() before PetscDTGaussTensorQuadrature(). Thanks - my test works now. Cheers, Dave On 18 November 2014 22:51, Dave May wrote: > Okay, thanks Matt. It now makes sense why it failed, but I don't > understand your logic. > The PetscQuadrature class is so small and doesn't depend on DM (at least > not in the definition of the structure described PetscQuadrature). Hence, > why should DMInitializePackage() have to be called at all? > I realize you how are you using PetscDT, but it should be available for > everyone - even if they don't ever create a DM. > > > Cheers, > Dave > > On 18 November 2014 22:46, Matthew Knepley wrote: > >> On Tue, Nov 18, 2014 at 1:49 PM, Dave May >> wrote: >> >>> Hi, >>> >>> I was trying to use some of DT functionality (in particular >>> PetscDTGaussTensorQuadrature()) in petsc 3.5.2 and I encountered this error: >>> >> >> I did not anticipate someone calling that before anything by itself. You >> need a call to >> >> DMInitializePackage() >> >> This is called by PetscQuadratureCreate(). >> >> Matt >> >> >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Invalid argument >>> [0]PETSC ERROR: Invalid object classid 0 >>> This could happen if you compile with PETSC_HAVE_DYNAMIC_LIBRARIES, but >>> link with static libraries. >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>> [0]PETSC ERROR: ./dmdadg.app on a arch-darwin-c-mpi-debug-f2c named >>> nikkan.local by dmay Tue Nov 18 20:34:56 2014 >>> [0]PETSC ERROR: Configure options --download-openmpi=yes >>> --with-debugging=yes --with-fc=0 --download-f2cblaslapck=yes >>> PETSC_ARCH=arch-darwin-c-mpi-debug-f2c >>> [0]PETSC ERROR: #1 PetscClassRegLogGetClass() line 290 in >>> /Users/dmay/software/petsc-3.5.2/src/sys/logging/utils/classlog.c >>> [0]PETSC ERROR: #2 PetscLogObjCreateDefault() line 317 in >>> /Users/dmay/software/petsc-3.5.2/src/sys/logging/utils/classlog.c >>> [0]PETSC ERROR: #3 PetscQuadratureCreate() line 35 in >>> /Users/dmay/software/petsc-3.5.2/src/dm/dt/interface/dt.c >>> [0]PETSC ERROR: #4 PetscDTGaussTensorQuadrature() line 322 in >>> /Users/dmay/software/petsc-3.5.2/src/dm/dt/interface/dt.c >>> [0]PETSC ERROR: #5 DMDGSpace_ex4_quadrature() line 480 in dmdadg.c >>> [0]PETSC ERROR: #6 main() line 503 in dmdadg.c >>> [0]PETSC ERROR: ----------------End of Error Message -------send entire >>> error message to petsc-maint at mcs.anl.gov---------- >>> >>> -------------------------------------------------------------------------- >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>> with errorcode 1. >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> You may or may not see output from other processes, depending on >>> exactly when Open MPI kills them. >>> >>> -------------------------------------------------------------------------- >>> >>> Snooping around in the source tree, the only place where I can find >>> PetscHeaderCreate being passed PETSC_OBJECT_CLASSID is within >>> src/dm/dt/interface/dt.c in the function PetscQuadratureCreate(). >>> >>> Is this is a bug or is my petsc build broken?? >>> >>> I'm sure I am linking against a dynamic library (or whatever you want to >>> call a dylib on OSX) >>> >>> dmay at nikkan:~/codes/dgspace$ otool -L dmdadg.app >>> dmdadg.app: >>> * >>> /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libpetsc.3.5.dylib >>> (compatibility version 3.5.0, current version 3.5.2)* >>> >>> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib >>> (compatibility version 1.0.0, current version 1.0.0) >>> >>> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib >>> (compatibility version 1.0.0, current version 1.0.0) >>> /opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current >>> version 10.0.0) >>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current >>> version 1197.1.1) >>> /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current >>> version 50.0.0) >>> /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current >>> version 50.0.0) >>> >>> /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libmpi_cxx.1.dylib >>> (compatibility version 3.0.0, current version 3.3.0) >>> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current >>> version 120.0.0) >>> >>> /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libmpi.1.dylib >>> (compatibility version 7.0.0, current version 7.0.0) >>> >>> >>> >>> Cheers, >>> Dave >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 18 16:54:44 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 Nov 2014 16:54:44 -0600 Subject: [petsc-users] PetscQuadratureCreate and PETSC_OBJECT_CLASSID In-Reply-To: References: Message-ID: On Tue, Nov 18, 2014 at 4:09 PM, Dave May wrote: > Okay - maybe you meant PetscSysInitializePackage() needs to get first? > Yes I meant that. Something has to register our classes. I can put this call into the DT functions. Matt > In this simple test I was running, I essentially called > > PetscInitialize > PetscDTGaussTensorQuadrature > PetscFinalize > > and thus no one call PetscSysInitializePackage() before > PetscDTGaussTensorQuadrature(). > > Thanks - my test works now. > > Cheers, > Dave > > > > On 18 November 2014 22:51, Dave May wrote: > >> Okay, thanks Matt. It now makes sense why it failed, but I don't >> understand your logic. >> The PetscQuadrature class is so small and doesn't depend on DM (at least >> not in the definition of the structure described PetscQuadrature). Hence, >> why should DMInitializePackage() have to be called at all? >> I realize you how are you using PetscDT, but it should be available for >> everyone - even if they don't ever create a DM. >> >> >> Cheers, >> Dave >> >> On 18 November 2014 22:46, Matthew Knepley wrote: >> >>> On Tue, Nov 18, 2014 at 1:49 PM, Dave May >>> wrote: >>> >>>> Hi, >>>> >>>> I was trying to use some of DT functionality (in particular >>>> PetscDTGaussTensorQuadrature()) in petsc 3.5.2 and I encountered this error: >>>> >>> >>> I did not anticipate someone calling that before anything by itself. You >>> need a call to >>> >>> DMInitializePackage() >>> >>> This is called by PetscQuadratureCreate(). >>> >>> Matt >>> >>> >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Invalid argument >>>> [0]PETSC ERROR: Invalid object classid 0 >>>> This could happen if you compile with PETSC_HAVE_DYNAMIC_LIBRARIES, but >>>> link with static libraries. >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>> [0]PETSC ERROR: ./dmdadg.app on a arch-darwin-c-mpi-debug-f2c named >>>> nikkan.local by dmay Tue Nov 18 20:34:56 2014 >>>> [0]PETSC ERROR: Configure options --download-openmpi=yes >>>> --with-debugging=yes --with-fc=0 --download-f2cblaslapck=yes >>>> PETSC_ARCH=arch-darwin-c-mpi-debug-f2c >>>> [0]PETSC ERROR: #1 PetscClassRegLogGetClass() line 290 in >>>> /Users/dmay/software/petsc-3.5.2/src/sys/logging/utils/classlog.c >>>> [0]PETSC ERROR: #2 PetscLogObjCreateDefault() line 317 in >>>> /Users/dmay/software/petsc-3.5.2/src/sys/logging/utils/classlog.c >>>> [0]PETSC ERROR: #3 PetscQuadratureCreate() line 35 in >>>> /Users/dmay/software/petsc-3.5.2/src/dm/dt/interface/dt.c >>>> [0]PETSC ERROR: #4 PetscDTGaussTensorQuadrature() line 322 in >>>> /Users/dmay/software/petsc-3.5.2/src/dm/dt/interface/dt.c >>>> [0]PETSC ERROR: #5 DMDGSpace_ex4_quadrature() line 480 in dmdadg.c >>>> [0]PETSC ERROR: #6 main() line 503 in dmdadg.c >>>> [0]PETSC ERROR: ----------------End of Error Message -------send entire >>>> error message to petsc-maint at mcs.anl.gov---------- >>>> >>>> -------------------------------------------------------------------------- >>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>> with errorcode 1. >>>> >>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>> You may or may not see output from other processes, depending on >>>> exactly when Open MPI kills them. >>>> >>>> -------------------------------------------------------------------------- >>>> >>>> Snooping around in the source tree, the only place where I can find >>>> PetscHeaderCreate being passed PETSC_OBJECT_CLASSID is within >>>> src/dm/dt/interface/dt.c in the function PetscQuadratureCreate(). >>>> >>>> Is this is a bug or is my petsc build broken?? >>>> >>>> I'm sure I am linking against a dynamic library (or whatever you want >>>> to call a dylib on OSX) >>>> >>>> dmay at nikkan:~/codes/dgspace$ otool -L dmdadg.app >>>> dmdadg.app: >>>> * >>>> /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libpetsc.3.5.dylib >>>> (compatibility version 3.5.0, current version 3.5.2)* >>>> >>>> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib >>>> (compatibility version 1.0.0, current version 1.0.0) >>>> >>>> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib >>>> (compatibility version 1.0.0, current version 1.0.0) >>>> /opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current >>>> version 10.0.0) >>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current >>>> version 1197.1.1) >>>> /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current >>>> version 50.0.0) >>>> /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, >>>> current version 50.0.0) >>>> >>>> /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libmpi_cxx.1.dylib >>>> (compatibility version 3.0.0, current version 3.3.0) >>>> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current >>>> version 120.0.0) >>>> >>>> /Users/dmay/software/petsc-3.5.2/arch-darwin-c-mpi-debug-f2c/lib/libmpi.1.dylib >>>> (compatibility version 7.0.0, current version 7.0.0) >>>> >>>> >>>> >>>> Cheers, >>>> Dave >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lb2653 at columbia.edu Tue Nov 18 17:25:14 2014 From: lb2653 at columbia.edu (Luc Berger-Vergiat) Date: Tue, 18 Nov 2014 18:25:14 -0500 Subject: [petsc-users] statistics on matrix passed to mumps Message-ID: <546BD55A.3080103@columbi.edu> Hi, I have an integer memory allocation when I call mumps to solve my linear problems. mumps report a memory error: INFOG(1)=-7 when it allocates IRN and JCN, these two arrays being of size NZ. mumps documents also defines NZ as the numbers of entries being input. I want to know what is the size value of NZ so that I can determine how to partition my system of equations correctly. So far I call: MatGetInfo(Kmat, MAT_LOCAL, info, ierr) in order to get the statistics on my matrix. I want to know if info(MAT_INFO_NZ_ALLOCATED) corresponds to the NZ of mumps? If it is not the case how can I recover that mumps NZ value? -- Best, Luc From knepley at gmail.com Wed Nov 19 06:19:05 2014 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 19 Nov 2014 06:19:05 -0600 Subject: [petsc-users] Roles of ksp_type and pc_type In-Reply-To: <4302011.cIgiPZOLR6@debianxps> References: <2594845.ya19CE4v59@debianxps> <1928038.ts4lkFBGPS@debianxps> <4302011.cIgiPZOLR6@debianxps> Message-ID: On Mon, Nov 17, 2014 at 4:27 PM, Massimiliano Leoni < leoni.massimiliano1 at gmail.com> wrote: > Hi again, > I've been experimenting further with fieldsplit, following your slides, > but > for some reason -pc_fieldsplit_type multiplicative and schur segfault at > KSPSolve. Additive seems to work fine. > It looks to me like you have memory corruption in a different part of the code. I would build a debugging version, and consider running under valgrind. I could not reproduce ex62 as in your slides because I don't have the header > file and the script PetscGenerateFEMQuadrature.py, which I downloaded from > the > github repo, complains that PETSC.FEM module is missing, so I used my code, > which is supopsd to assemble the same matrix, then I call KSPSetOperators > with > Stokes matrix as both the matrix and the preconditioner. > That is no longer necessary. It just runs out of the box. The slides from SIAM CS&E 2013 show this. Here is a sample run cd src/snes/examples/tutorials make ex62 ./ex62 -run_type full -refinement_limit 0.00625 -bc_type dirichlet -interpolate 1 -vel_petscspace_order 2 -pres_petscspace_order 1 -ksp_type fgmres -ksp_gmres_restart 100 -ksp_rtol 1.0e-9 -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type full -fieldsplit_pressure_ksp_rtol 1e-10 -fieldsplit_velocity_ksp_type gmres -fieldsplit_velocity_pc_type lu -fieldsplit_pressure_pc_type jacobi -snes_monitor_short -ksp_monitor_short -snes_converged_reason -ksp_converged_reason -snes_view -show_solution 0 Thanks, Matt I use the same options as in slide 115, the "block jacobi, inexact" version > works, while the "block Gauss-seidl, inexact" doesn't and segfaults. > > I attach backtrace and log files, according to what I was told last time. > > Thanks for any help, > Massimiliano > > > > In data gioved? 13 novembre 2014 15:07:40, Massimiliano Leoni ha scritto: > > In data gioved? 13 novembre 2014 06:39:38, Matthew Knepley ha scritto: > > > On Thu, Nov 13, 2014 at 6:24 AM, Massimiliano Leoni < > > > > > > This is not exactly right because here you are just using additive > > > fieldsplit. > > > > AFAI understood from the user manual, this should be right because my > > preconditioner is block diagonal, so I want to use block Jacobi. > > > > If I used the other one, P = [[C,O],[B,Mp/nu]] then I would need a > > multiplicative fieldsplit because this would be lower triangular. > > > > Is this correct? > > > > > The mass matrix > > > is a good preconditioner for the Schur complement, not the zero matrix. > > > > > Take a look at these slides: > > What do you mean? > > I know from theory that the preconditioner I am using is optimal > [iteration > > number independent of grid size]. > > > > I read the slides and the manual and the Schur complement is what I was > > talking about earlier, so it's not what I wanna do now -- even though > > knowing that it's so easy to implement my be very useful. > > > > Anyway, thanks for the slides, they have been enlightening on the power > of > > command line options! > > > > > Thanks, > > > > > > Matt > > > > Thanks again, > > Massimiliano > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From evanum at gmail.com Wed Nov 19 11:43:55 2014 From: evanum at gmail.com (Evan Um) Date: Wed, 19 Nov 2014 09:43:55 -0800 Subject: [petsc-users] MPI+OpenMP in PETSC Message-ID: Dear PETSC users, I would like to ask a question about using an external library with MPI+OpenMP in PETSC. For example, within PETSC, I want to use MUMPS with MPI+OpenMP. This means that if one node has 12 MPI processes and 24GB, MUMPS uses 4 MPI processes with 6GB and each MPI process has 3 threads. To do this, what should be done from PETSC installation? After using MUMPS in this way, is there a way to convert from MPI+OpenMP to flat MPI? Thank you! Regards, Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Nov 19 13:39:54 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 19 Nov 2014 13:39:54 -0600 Subject: [petsc-users] MPI+OpenMP in PETSC In-Reply-To: References: Message-ID: <59A382F1-3593-4434-B950-32DB357AA4D1@mcs.anl.gov> > On Nov 19, 2014, at 11:43 AM, Evan Um wrote: > > Dear PETSC users, > > I would like to ask a question about using an external library with MPI+OpenMP in PETSC. For example, within PETSC, I want to use MUMPS with MPI+OpenMP. This means that if one node has 12 MPI processes and 24GB, MUMPS uses 4 MPI processes with 6GB and each MPI process has 3 threads. To do this, what should be done from PETSC installation? After using MUMPS in this way, is there a way to convert from MPI+OpenMP to flat MPI? Thank you! You just need to launch your MPI (PETSc) program so that it uses only 4 MPI processes per node (how do this this depends on your system, it may be options to mpirun or it may be options in a batch system). Then you need tell MUMPS to use 3 OpenMP threads (this is probably a OpenMP environmental variable). Note that these two things that need to be set are not really directly related to PETSc and we cannot set them automatically. It is essentially impossible to convert from MPI+OpenMP to flat MPI*. You will need to just use those 4 MPI processes per node for the MPI part. Barry * yes you could use MPI_Spawn and stuff but it won't do what you want. > > Regards, > Evan > > > From salazardetroya at gmail.com Wed Nov 19 14:02:10 2014 From: salazardetroya at gmail.com (Miguel Angel Salazar de Troya) Date: Wed, 19 Nov 2014 14:02:10 -0600 Subject: [petsc-users] PetscFunctionBegin, -malloc_dump and C++ classes with PETSc objects In-Reply-To: <382BFA94-5183-4D5E-ADFE-306FCD43113A@mcs.anl.gov> References: <6256BDCD-19E6-48F5-9872-E5DB97C981EA@mcs.anl.gov> <382BFA94-5183-4D5E-ADFE-306FCD43113A@mcs.anl.gov> Message-ID: You were right. I was forgetting a DMRestoreLocalVector(). Thanks. If we are creating local vectors every time we call functions such as the RHSFunction of a TS implementation, which is called many times in the TS integration, will this be a problem in terms of performance? I've noticed that it might be. This is the way it is implemented in some TS. I wanted to double check with you guys before I figure out a way to create just one local vector that is re used in functions such as RHSFunction. Miguel On Tue, Nov 18, 2014 at 12:54 PM, Barry Smith wrote: > > > On Nov 18, 2014, at 11:54 AM, Miguel Angel Salazar de Troya < > salazardetroya at gmail.com> wrote: > > > > PetscFinalize() is inside of the class destructor (the last line of the > destructor), so when the object goes out of scope, the class destructor is > called and PetscFinalize() as well. Is it better to have PetscFinalize() > outside of the destructor and call the destructor explicitly before? > > It shouldn't really matter. > > My guess is you must be missing calling a destroy or restore on one of > the PETSc objects. > > Barry > > > > > Miguel > > > > On Tue, Nov 18, 2014 at 11:32 AM, Barry Smith > wrote: > > > > > On Nov 18, 2014, at 11:19 AM, Miguel Angel Salazar de Troya < > salazardetroya at gmail.com> wrote: > > > > > > Hi > > > > > > I'm implementing a problem using the TS. Most of my functions are > methods inside of a class, except for the callbacks (to form the RHS and > the TS monitor), which are outside of the class, although in the same .C > file where the class methods are implemented. For these callbacks I > followed the network example: > > > > > > > https://bitbucket.org/petsc/petsc/src/a614f7369d93d476173b8fc6bf2463276dcbdb3a/src/snes/examples/tutorials/network/pflow/pf.c?at=master > > > > > > Therefore, the callbacks have the PetscFunctionBegin at the beginning > and PetscFunctionReturn(0) at the end. My problems come when I run the > program with -malloc_dump and I get a lot of unfreed memory. Inspecting the > output I see that the line of my code where the memory is allocated > corresponds with the line when PetscFunctionBegin is called. > > > > This is normal. We cannot register the exact line the memory > allocated, only the location of the PETScFunctionBegin; > > > > > > > Later in the file, I see that the function DMGetLocalVector() is > called within a petsc internal routine (at the file dmget.c). I also call > this routine in my callback methods few lines after PetscFunctionBegin. The > procedure that I follow to use the local vectors is as the one in the > network example. For vectors that I want to modify this is: > > > > > > ierr = DMGetLocalVector(networkdm,&localX);CHKERRQ(ierr); > > > ierr = > DMGlobalToLocalBegin(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); > > > ierr = > DMGlobalToLocalEnd(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); > > > ierr = VecGetArray(localX,&xarr);CHKERRQ(ierr); > > > > > > Modify values in xarr > > > > > > ierr = VecRestoreArray(localX,&xarr);CHKERRQ(ierr); > > > ierr = > DMLocalToGlobalBegin(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); > > > ierr = > DMLocalToGlobalEnd(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); > > > ierr = DMRestoreLocalVector(networkdm,&localX);CHKERRQ(ierr); > > > > > > One last thing that I think it might be a issue here is how I destroy > the petsc objects. I create the petsc objects within a class. For instance, > the class has a petsc vector that later passes to the TS object to get the > solution. To destroy the petsc objects, I use the class destructor, where > at the end I call PetscFinalize() Inside the class I pass the callbacks to > the TS routines that need them (e.g. TSSetRHSFunction() ) I can compile the > code and run it, but many memory allocations are not freed. What can be the > issue here? Do you know of an example using C++ classes to implement PETSc > methods? Thanks in advance. > > > > Do you call the class destructor yourself explicitly before the > PetscFinalize()? You need to, otherwise the class may not be destroyed > until after PetscFinalize() and hence the PETSc objects won't be freed when > you call PetscFinalize(). > > > > You also need to make sure that you destroy ALL PETSc objects, if you > miss even one PETSc object, since the objects have references to each other > it may be that many PETSc objects do not get freed and hence -malloc_dump > shows many objects still alive. > > > > Barry > > > > > > > > Miguel > > > > > > -- > > > Miguel Angel Salazar de Troya > > > Graduate Research Assistant > > > Department of Mechanical Science and Engineering > > > University of Illinois at Urbana-Champaign > > > (217) 550-2360 > > > salaza11 at illinois.edu > > > > > > > > > > > > > -- > > Miguel Angel Salazar de Troya > > Graduate Research Assistant > > Department of Mechanical Science and Engineering > > University of Illinois at Urbana-Champaign > > (217) 550-2360 > > salaza11 at illinois.edu > > > > -- *Miguel Angel Salazar de Troya* Graduate Research Assistant Department of Mechanical Science and Engineering University of Illinois at Urbana-Champaign (217) 550-2360 salaza11 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Wed Nov 19 14:36:23 2014 From: abhyshr at mcs.anl.gov (Abhyankar, Shrirang G.) Date: Wed, 19 Nov 2014 20:36:23 +0000 Subject: [petsc-users] PetscFunctionBegin, -malloc_dump and C++ classes with PETSc objects In-Reply-To: Message-ID: The DM keeps track of the local vectors created by DMGetLocalVector(). So, if your routine gets called a million times with exactly one DMGetLocalVector() followed by one call to DMRestoreLocalVector(), then only one vector is created. Shri From: Miguel Angel Salazar de Troya > Date: Wed, 19 Nov 2014 14:02:10 -0600 To: barry smith > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] PetscFunctionBegin, -malloc_dump and C++ classes with PETSc objects You were right. I was forgetting a DMRestoreLocalVector(). Thanks. If we are creating local vectors every time we call functions such as the RHSFunction of a TS implementation, which is called many times in the TS integration, will this be a problem in terms of performance? I've noticed that it might be. This is the way it is implemented in some TS. I wanted to double check with you guys before I figure out a way to create just one local vector that is re used in functions such as RHSFunction. Miguel On Tue, Nov 18, 2014 at 12:54 PM, Barry Smith > wrote: > On Nov 18, 2014, at 11:54 AM, Miguel Angel Salazar de Troya > wrote: > > PetscFinalize() is inside of the class destructor (the last line of the destructor), so when the object goes out of scope, the class destructor is called and PetscFinalize() as well. Is it better to have PetscFinalize() outside of the destructor and call the destructor explicitly before? It shouldn't really matter. My guess is you must be missing calling a destroy or restore on one of the PETSc objects. Barry > > Miguel > > On Tue, Nov 18, 2014 at 11:32 AM, Barry Smith > wrote: > > > On Nov 18, 2014, at 11:19 AM, Miguel Angel Salazar de Troya > wrote: > > > > Hi > > > > I'm implementing a problem using the TS. Most of my functions are methods inside of a class, except for the callbacks (to form the RHS and the TS monitor), which are outside of the class, although in the same .C file where the class methods are implemented. For these callbacks I followed the network example: > > > > https://bitbucket.org/petsc/petsc/src/a614f7369d93d476173b8fc6bf2463276dcbdb3a/src/snes/examples/tutorials/network/pflow/pf.c?at=master > > > > Therefore, the callbacks have the PetscFunctionBegin at the beginning and PetscFunctionReturn(0) at the end. My problems come when I run the program with -malloc_dump and I get a lot of unfreed memory. Inspecting the output I see that the line of my code where the memory is allocated corresponds with the line when PetscFunctionBegin is called. > > This is normal. We cannot register the exact line the memory allocated, only the location of the PETScFunctionBegin; > > > > Later in the file, I see that the function DMGetLocalVector() is called within a petsc internal routine (at the file dmget.c). I also call this routine in my callback methods few lines after PetscFunctionBegin. The procedure that I follow to use the local vectors is as the one in the network example. For vectors that I want to modify this is: > > > > ierr = DMGetLocalVector(networkdm,&localX);CHKERRQ(ierr); > > ierr = DMGlobalToLocalBegin(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); > > ierr = DMGlobalToLocalEnd(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); > > ierr = VecGetArray(localX,&xarr);CHKERRQ(ierr); > > > > Modify values in xarr > > > > ierr = VecRestoreArray(localX,&xarr);CHKERRQ(ierr); > > ierr = DMLocalToGlobalBegin(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); > > ierr = DMLocalToGlobalEnd(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); > > ierr = DMRestoreLocalVector(networkdm,&localX);CHKERRQ(ierr); > > > > One last thing that I think it might be a issue here is how I destroy the petsc objects. I create the petsc objects within a class. For instance, the class has a petsc vector that later passes to the TS object to get the solution. To destroy the petsc objects, I use the class destructor, where at the end I call PetscFinalize() Inside the class I pass the callbacks to the TS routines that need them (e.g. TSSetRHSFunction() ) I can compile the code and run it, but many memory allocations are not freed. What can be the issue here? Do you know of an example using C++ classes to implement PETSc methods? Thanks in advance. > > Do you call the class destructor yourself explicitly before the PetscFinalize()? You need to, otherwise the class may not be destroyed until after PetscFinalize() and hence the PETSc objects won't be freed when you call PetscFinalize(). > > You also need to make sure that you destroy ALL PETSc objects, if you miss even one PETSc object, since the objects have references to each other it may be that many PETSc objects do not get freed and hence -malloc_dump shows many objects still alive. > > Barry > > > > > Miguel > > > > -- > > Miguel Angel Salazar de Troya > > Graduate Research Assistant > > Department of Mechanical Science and Engineering > > University of Illinois at Urbana-Champaign > > (217) 550-2360 > > salaza11 at illinois.edu > > > > > > > -- > Miguel Angel Salazar de Troya > Graduate Research Assistant > Department of Mechanical Science and Engineering > University of Illinois at Urbana-Champaign > (217) 550-2360 > salaza11 at illinois.edu > -- Miguel Angel Salazar de Troya Graduate Research Assistant Department of Mechanical Science and Engineering University of Illinois at Urbana-Champaign (217) 550-2360 salaza11 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Nov 19 16:07:58 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 19 Nov 2014 16:07:58 -0600 Subject: [petsc-users] PetscFunctionBegin, -malloc_dump and C++ classes with PETSc objects In-Reply-To: References: Message-ID: <8C604A1B-1E80-492E-9BCC-B46E8697982E@mcs.anl.gov> > On Nov 19, 2014, at 2:36 PM, Abhyankar, Shrirang G. wrote: > > The DM keeps track of the local vectors created by DMGetLocalVector(). So, if your routine gets called a million times with exactly one DMGetLocalVector() followed by one call to DMRestoreLocalVector(), then only one vector is created. In other words the DMGet/RestoreXXVector() are designed EXACTLY to be used when you need work vectors so you do not need to manage keeping the work vectors around yourself. In other words you are using it exactly right. Barry > > Shri > > From: Miguel Angel Salazar de Troya > Date: Wed, 19 Nov 2014 14:02:10 -0600 > To: barry smith > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] PetscFunctionBegin, -malloc_dump and C++ classes with PETSc objects > > You were right. I was forgetting a DMRestoreLocalVector(). Thanks. If we are creating local vectors every time we call functions such as the RHSFunction of a TS implementation, which is called many times in the TS integration, will this be a problem in terms of performance? I've noticed that it might be. This is the way it is implemented in some TS. I wanted to double check with you guys before I figure out a way to create just one local vector that is re used in functions such as RHSFunction. > > Miguel > > On Tue, Nov 18, 2014 at 12:54 PM, Barry Smith wrote: > > > On Nov 18, 2014, at 11:54 AM, Miguel Angel Salazar de Troya wrote: > > > > PetscFinalize() is inside of the class destructor (the last line of the destructor), so when the object goes out of scope, the class destructor is called and PetscFinalize() as well. Is it better to have PetscFinalize() outside of the destructor and call the destructor explicitly before? > > It shouldn't really matter. > > My guess is you must be missing calling a destroy or restore on one of the PETSc objects. > > Barry > > > > > Miguel > > > > On Tue, Nov 18, 2014 at 11:32 AM, Barry Smith wrote: > > > > > On Nov 18, 2014, at 11:19 AM, Miguel Angel Salazar de Troya wrote: > > > > > > Hi > > > > > > I'm implementing a problem using the TS. Most of my functions are methods inside of a class, except for the callbacks (to form the RHS and the TS monitor), which are outside of the class, although in the same .C file where the class methods are implemented. For these callbacks I followed the network example: > > > > > > https://bitbucket.org/petsc/petsc/src/a614f7369d93d476173b8fc6bf2463276dcbdb3a/src/snes/examples/tutorials/network/pflow/pf.c?at=master > > > > > > Therefore, the callbacks have the PetscFunctionBegin at the beginning and PetscFunctionReturn(0) at the end. My problems come when I run the program with -malloc_dump and I get a lot of unfreed memory. Inspecting the output I see that the line of my code where the memory is allocated corresponds with the line when PetscFunctionBegin is called. > > > > This is normal. We cannot register the exact line the memory allocated, only the location of the PETScFunctionBegin; > > > > > > > Later in the file, I see that the function DMGetLocalVector() is called within a petsc internal routine (at the file dmget.c). I also call this routine in my callback methods few lines after PetscFunctionBegin. The procedure that I follow to use the local vectors is as the one in the network example. For vectors that I want to modify this is: > > > > > > ierr = DMGetLocalVector(networkdm,&localX);CHKERRQ(ierr); > > > ierr = DMGlobalToLocalBegin(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); > > > ierr = DMGlobalToLocalEnd(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr); > > > ierr = VecGetArray(localX,&xarr);CHKERRQ(ierr); > > > > > > Modify values in xarr > > > > > > ierr = VecRestoreArray(localX,&xarr);CHKERRQ(ierr); > > > ierr = DMLocalToGlobalBegin(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); > > > ierr = DMLocalToGlobalEnd(networkdm,localX,INSERT_VALUES,X);CHKERRQ(ierr); > > > ierr = DMRestoreLocalVector(networkdm,&localX);CHKERRQ(ierr); > > > > > > One last thing that I think it might be a issue here is how I destroy the petsc objects. I create the petsc objects within a class. For instance, the class has a petsc vector that later passes to the TS object to get the solution. To destroy the petsc objects, I use the class destructor, where at the end I call PetscFinalize() Inside the class I pass the callbacks to the TS routines that need them (e.g. TSSetRHSFunction() ) I can compile the code and run it, but many memory allocations are not freed. What can be the issue here? Do you know of an example using C++ classes to implement PETSc methods? Thanks in advance. > > > > Do you call the class destructor yourself explicitly before the PetscFinalize()? You need to, otherwise the class may not be destroyed until after PetscFinalize() and hence the PETSc objects won't be freed when you call PetscFinalize(). > > > > You also need to make sure that you destroy ALL PETSc objects, if you miss even one PETSc object, since the objects have references to each other it may be that many PETSc objects do not get freed and hence -malloc_dump shows many objects still alive. > > > > Barry > > > > > > > > Miguel > > > > > > -- > > > Miguel Angel Salazar de Troya > > > Graduate Research Assistant > > > Department of Mechanical Science and Engineering > > > University of Illinois at Urbana-Champaign > > > (217) 550-2360 > > > salaza11 at illinois.edu > > > > > > > > > > > > > -- > > Miguel Angel Salazar de Troya > > Graduate Research Assistant > > Department of Mechanical Science and Engineering > > University of Illinois at Urbana-Champaign > > (217) 550-2360 > > salaza11 at illinois.edu > > > > > > > -- > Miguel Angel Salazar de Troya > Graduate Research Assistant > Department of Mechanical Science and Engineering > University of Illinois at Urbana-Champaign > (217) 550-2360 > salaza11 at illinois.edu > From hsahasra at purdue.edu Wed Nov 19 23:01:26 2014 From: hsahasra at purdue.edu (Harshad Sahasrabudhe) Date: Thu, 20 Nov 2014 00:01:26 -0500 Subject: [petsc-users] Custom line search for predictor-corrector Message-ID: Hi, I want to implement a Predictor-Corrector algorithm for solving a coupled Schr?dinger-Poisson system using SNES. I can calculate the next step using my predictor algorithm and don't want PETSc to calculate the next step. I only need PETSc to check for convergence. What is the best way to implement this? Thanks, Harshad -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Nov 19 23:17:10 2014 From: jed at jedbrown.org (Jed Brown) Date: Wed, 19 Nov 2014 23:17:10 -0600 Subject: [petsc-users] Custom line search for predictor-corrector In-Reply-To: References: Message-ID: <87fvdee9ih.fsf@jedbrown.org> Harshad Sahasrabudhe writes: > Hi, > > I want to implement a Predictor-Corrector algorithm for solving a coupled > Schr?dinger-Poisson system using SNES. I can calculate the next step using > my predictor algorithm and don't want PETSc to calculate the next step. I > only need PETSc to check for convergence. What is the best way to implement > this? Probably SNESLineSearchRegister your implementation. The Newton solve gives a correction and you can follow it with anything you want. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From haakon at hakostra.net Thu Nov 20 04:05:45 2014 From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=) Date: Thu, 20 Nov 2014 11:05:45 +0100 Subject: [petsc-users] Pipelined solvers PIPECG etc fail Message-ID: <546DBCF9.6080407@hakostra.net> Hi, First: thanks to the list and Barry for solving my previous problem on loading vectors from HDF5 files. It works great! Now a second problem: I want to try the pipelined KSP solvers, PIPECG in particular. However, my program fails on the first KSPSolve() with the message below. The same happens if I try PIPECR. I have tried this on three MPI implementations, on and two of them fully support MPI 3 (I have read http://www.mcs.anl.gov/petsc/documentation/faq.html#pipelined). I am using latest PETSc from Git repo. I have tried to turn off the preconditioner without any effect. The equation system is the result of a finite-volume discretization of a Poisson-like equation if that helps anything. A normal CG solver works like a charm, converging fast and effective. Are there any tricks to get this working? Anything special I have to set and/or define to get this solver running? Thanks in advance. Best regards, H?kon Strandenes Error message: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Use KSPConvergedSkip() with KSPNormType of KSP_NORM_NONE [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown [0]PETSC ERROR: madns on a arch-linux2-c-debug named hakostra-kontor by hakostra Thu Nov 20 11:04:28 2014 [0]PETSC ERROR: Configure options --prefix=/opt/petsc/git-debug/install --with-shared-libraries --download-hypre --with-64-bit-indices --with-hdf5 --with-hdf5-dir=/opt/HDF5/1.8.13/install --with-debugging --with-debugger=gdb [0]PETSC ERROR: #1 KSPConvergedDefault() line 710 in /opt/petsc/git-debug/source/src/ksp/ksp/interface/iterativ.c [0]PETSC ERROR: #2 KSPSolve_PIPECG() line 99 in /opt/petsc/git-debug/source/src/ksp/ksp/impls/cg/pipecg/pipecg.c [0]PETSC ERROR: #3 KSPSolve() line 459 in /opt/petsc/git-debug/source/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #4 findPressure() line 53 in /opt/MaDNS/source/src/FVM2.cpp From knepley at gmail.com Thu Nov 20 07:17:33 2014 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 20 Nov 2014 07:17:33 -0600 Subject: [petsc-users] Pipelined solvers PIPECG etc fail In-Reply-To: <546DBCF9.6080407@hakostra.net> References: <546DBCF9.6080407@hakostra.net> Message-ID: On Thu, Nov 20, 2014 at 4:05 AM, H?kon Strandenes wrote: > Hi, > > First: thanks to the list and Barry for solving my previous problem on > loading vectors from HDF5 files. It works great! > > Now a second problem: I want to try the pipelined KSP solvers, PIPECG in > particular. However, my program fails on the first KSPSolve() with the > message below. The same happens if I try PIPECR. > These are very new, so there could be bugs lurking. It would help our testing if this is reproducuble on a standard example. Does this happen if you try to solve KSP ex2 or SNES ex5 with PIPECG? Thanks, Matt > I have tried this on three MPI implementations, on and two of them fully > support MPI 3 (I have read http://www.mcs.anl.gov/petsc/ > documentation/faq.html#pipelined). I am using latest PETSc from Git repo. > > I have tried to turn off the preconditioner without any effect. > > The equation system is the result of a finite-volume discretization of a > Poisson-like equation if that helps anything. A normal CG solver works like > a charm, converging fast and effective. > > Are there any tricks to get this working? Anything special I have to set > and/or define to get this solver running? > > Thanks in advance. > > Best regards, > H?kon Strandenes > > > Error message: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Use KSPConvergedSkip() with KSPNormType of KSP_NORM_NONE > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown > [0]PETSC ERROR: madns on a arch-linux2-c-debug named hakostra-kontor by > hakostra Thu Nov 20 11:04:28 2014 > [0]PETSC ERROR: Configure options --prefix=/opt/petsc/git-debug/install > --with-shared-libraries --download-hypre --with-64-bit-indices --with-hdf5 > --with-hdf5-dir=/opt/HDF5/1.8.13/install --with-debugging > --with-debugger=gdb > [0]PETSC ERROR: #1 KSPConvergedDefault() line 710 in > /opt/petsc/git-debug/source/src/ksp/ksp/interface/iterativ.c > [0]PETSC ERROR: #2 KSPSolve_PIPECG() line 99 in > /opt/petsc/git-debug/source/src/ksp/ksp/impls/cg/pipecg/pipecg.c > [0]PETSC ERROR: #3 KSPSolve() line 459 in /opt/petsc/git-debug/source/ > src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #4 findPressure() line 53 in /opt/MaDNS/source/src/FVM2.cpp > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From leoni.massimiliano1 at gmail.com Thu Nov 20 07:37:20 2014 From: leoni.massimiliano1 at gmail.com (Massimiliano Leoni) Date: Thu, 20 Nov 2014 14:37:20 +0100 Subject: [petsc-users] Roles of ksp_type and pc_type In-Reply-To: References: <2594845.ya19CE4v59@debianxps> <4302011.cIgiPZOLR6@debianxps> Message-ID: <1866871.vrluyBkT2a@debianxps> Hi Matt, thanks for your reply. In data mercoled? 19 novembre 2014 06:19:05, Matthew Knepley ha scritto: > It looks to me like you have memory corruption in a different part of the > code. I > would build a debugging version, and consider running under valgrind. I think I got what's happening. I found out this morning that if I set ksp_type multiplicative in the source code, with PCFieldSplitSetType, it works! This suggests that I am messing in the code with the relative positions of {KSP|PC}SetFromOptions, SetUp and the rest. Ksp_view confirms that PETSc actually does what I want, and ksp_monitor shows different iterations, which means that I am actually influencing something. > Here is a sample run > > cd src/snes/examples/tutorials > make ex62 > ./ex62 -run_type full -refinement_limit 0.00625 -bc_type dirichlet > -interpolate 1 -vel_petscspace_order 2 -pres_petscspace_order 1 -ksp_type > fgmres -ksp_gmres_restart 100 -ksp_rtol 1.0e-9 -pc_type fieldsplit > -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type full > -fieldsplit_pressure_ksp_rtol 1e-10 -fieldsplit_velocity_ksp_type gmres > -fieldsplit_velocity_pc_type lu -fieldsplit_pressure_pc_type jacobi > -snes_monitor_short -ksp_monitor_short -snes_converged_reason > -ksp_converged_reason -snes_view -show_solution 0 I can confirm this works for me as well, of course. > Thanks, > > Matt Thanks for the help! Massimiliano From knepley at gmail.com Thu Nov 20 07:45:48 2014 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 20 Nov 2014 07:45:48 -0600 Subject: [petsc-users] Roles of ksp_type and pc_type In-Reply-To: <1866871.vrluyBkT2a@debianxps> References: <2594845.ya19CE4v59@debianxps> <4302011.cIgiPZOLR6@debianxps> <1866871.vrluyBkT2a@debianxps> Message-ID: On Thu, Nov 20, 2014 at 7:37 AM, Massimiliano Leoni < leoni.massimiliano1 at gmail.com> wrote: > Hi Matt, thanks for your reply. > > In data mercoled? 19 novembre 2014 06:19:05, Matthew Knepley ha scritto: > > It looks to me like you have memory corruption in a different part of the > > code. I > > would build a debugging version, and consider running under valgrind. > I think I got what's happening. > I found out this morning that if I set ksp_type multiplicative in the > source > code, with PCFieldSplitSetType, it works! > This suggests that I am messing in the code with the relative positions of > {KSP|PC}SetFromOptions, SetUp and the rest. > > Ksp_view confirms that PETSc actually does what I want, and ksp_monitor > shows > different iterations, which means that I am actually influencing something. > Great, I am glad something is working. This is tricky, and it is why in our own code we have moved almost completely to an options driven configuration, with SetUp() called automatically by the solver. I have not been able to come up with a better option. Thanks, Matt > > Here is a sample run > > > > cd src/snes/examples/tutorials > > make ex62 > > ./ex62 -run_type full -refinement_limit 0.00625 -bc_type dirichlet > > -interpolate 1 -vel_petscspace_order 2 -pres_petscspace_order 1 -ksp_type > > fgmres -ksp_gmres_restart 100 -ksp_rtol 1.0e-9 -pc_type fieldsplit > > -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type full > > -fieldsplit_pressure_ksp_rtol 1e-10 -fieldsplit_velocity_ksp_type gmres > > -fieldsplit_velocity_pc_type lu -fieldsplit_pressure_pc_type jacobi > > -snes_monitor_short -ksp_monitor_short -snes_converged_reason > > -ksp_converged_reason -snes_view -show_solution 0 > > I can confirm this works for me as well, of course. > > > Thanks, > > > > Matt > > Thanks for the help! > Massimiliano > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From haakon at hakostra.net Thu Nov 20 08:38:29 2014 From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=) Date: Thu, 20 Nov 2014 15:38:29 +0100 Subject: [petsc-users] Pipelined solvers PIPECG etc fail In-Reply-To: References: <546DBCF9.6080407@hakostra.net> Message-ID: <546DFCE5.50406@hakostra.net> I just tested KSP ex2, and this error happened if I ran the code with "-ksp_type pipecg" or "-ksp_type pipecr", just as it did with my code. I used an unmodified version of KSP ex2, so I guess I don't have to send you my code... The error message was also the same. As far as I can see from the error message, this error has something to do with the convergence criterion... There must be some ways to stop the iteration process other than running a fixed number of iterations? H?kon On 20. nov. 2014 14:17, Matthew Knepley wrote: > On Thu, Nov 20, 2014 at 4:05 AM, H?kon Strandenes > wrote: > > Hi, > > First: thanks to the list and Barry for solving my previous problem > on loading vectors from HDF5 files. It works great! > > Now a second problem: I want to try the pipelined KSP solvers, > PIPECG in particular. However, my program fails on the first > KSPSolve() with the message below. The same happens if I try PIPECR. > > > These are very new, so there could be bugs lurking. It would help our > testing if this is > reproducuble on a standard example. Does this happen if you try to solve > KSP ex2 > or SNES ex5 with PIPECG? > > Thanks, > > Matt > > I have tried this on three MPI implementations, on and two of them > fully support MPI 3 (I have read > http://www.mcs.anl.gov/petsc/__documentation/faq.html#__pipelined > ). I > am using latest PETSc from Git repo. > > I have tried to turn off the preconditioner without any effect. > > The equation system is the result of a finite-volume discretization > of a Poisson-like equation if that helps anything. A normal CG > solver works like a charm, converging fast and effective. > > Are there any tricks to get this working? Anything special I have to > set and/or define to get this solver running? > > Thanks in advance. > > Best regards, > H?kon Strandenes > > > Error message: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------__------------------------------__-- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Use KSPConvergedSkip() with KSPNormType of KSP_NORM_NONE > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/__documentation/faq.html > for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown > [0]PETSC ERROR: madns on a arch-linux2-c-debug named hakostra-kontor > by hakostra Thu Nov 20 11:04:28 2014 > [0]PETSC ERROR: Configure options > --prefix=/opt/petsc/git-debug/__install --with-shared-libraries > --download-hypre --with-64-bit-indices --with-hdf5 > --with-hdf5-dir=/opt/HDF5/1.8.__13/install --with-debugging > --with-debugger=gdb > [0]PETSC ERROR: #1 KSPConvergedDefault() line 710 in > /opt/petsc/git-debug/source/__src/ksp/ksp/interface/__iterativ.c > [0]PETSC ERROR: #2 KSPSolve_PIPECG() line 99 in > /opt/petsc/git-debug/source/__src/ksp/ksp/impls/cg/pipecg/__pipecg.c > [0]PETSC ERROR: #3 KSPSolve() line 459 in > /opt/petsc/git-debug/source/__src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #4 findPressure() line 53 in > /opt/MaDNS/source/src/FVM2.cpp > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From mail at saschaschnepp.net Thu Nov 20 08:49:58 2014 From: mail at saschaschnepp.net (Sascha Schnepp) Date: Thu, 20 Nov 2014 15:49:58 +0100 Subject: [petsc-users] Pipelined solvers PIPECG etc fail References: Message-ID: <04D69684-F157-4ABA-AAEA-882BBF5E1B52@saschaschnepp.net> Hi, this is an error that occurs when calling ksp->converged and having ksp->normtype == KSP_NORM_NONE In pipecg.c the error occurs here: line99: ierr = (*ksp->converged)(ksp,0,dp,&ksp->reason,ksp->cnvP);CHKERRQ(ierr); /* test for convergence */ It can be fixed (though there might be a nicer way) using this replacement code, wich explicitly calls KSPConvergendSkip for the respective norm: if (ksp->normtype == KSP_NORM_NONE) { ierr = KSPConvergedSkip (ksp,0,dp,&ksp->reason,ksp->cnvP);CHKERRQ(ierr); } else { ierr = (*ksp->converged)(ksp,0,dp,&ksp->reason,ksp->cnvP);CHKERRQ(ierr); } The issue could be related to this code in KSPCreate_PIPECG ierr = KSPSetSupportedNorm(ksp,KSP_NORM_UNPRECONDITIONED,PC_LEFT,2);CHKERRQ(ierr); ierr = KSPSetSupportedNorm(ksp,KSP_NORM_PRECONDITIONED,PC_LEFT,2);CHKERRQ(ierr); ierr = KSPSetSupportedNorm(ksp,KSP_NORM_NATURAL,PC_LEFT,2);CHKERRQ(ierr); ierr = KSPSetSupportedNorm(ksp,KSP_NORM_NONE,PC_LEFT,2);CHKERRQ(ierr); where all norms have the same priority. For the non-pipelined version of CG the unpreconditioned norm has a priority level of 3. The modifications are independent. Choosing different default priorities will still make line 99 fail if type NONE is chosen explicitly. Cheers, Sascha Begin forwarded message: > On Thu, Nov 20, 2014 at 4:05 AM, H?kon Strandenes > wrote: > >> Hi, >> >> First: thanks to the list and Barry for solving my previous problem on >> loading vectors from HDF5 files. It works great! >> >> Now a second problem: I want to try the pipelined KSP solvers, PIPECG in >> particular. However, my program fails on the first KSPSolve() with the >> message below. The same happens if I try PIPECR. >> > > These are very new, so there could be bugs lurking. It would help our > testing if this is > reproducuble on a standard example. Does this happen if you try to solve > KSP ex2 > or SNES ex5 with PIPECG? > > Thanks, > > Matt > > >> I have tried this on three MPI implementations, on and two of them fully >> support MPI 3 (I have read http://www.mcs.anl.gov/petsc/ >> documentation/faq.html#pipelined). I am using latest PETSc from Git repo. >> >> I have tried to turn off the preconditioner without any effect. >> >> The equation system is the result of a finite-volume discretization of a >> Poisson-like equation if that helps anything. A normal CG solver works like >> a charm, converging fast and effective. >> >> Are there any tricks to get this working? Anything special I have to set >> and/or define to get this solver running? >> >> Thanks in advance. >> >> Best regards, >> H?kon Strandenes >> >> >> Error message: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Object is in wrong state >> [0]PETSC ERROR: Use KSPConvergedSkip() with KSPNormType of KSP_NORM_NONE >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown >> [0]PETSC ERROR: madns on a arch-linux2-c-debug named hakostra-kontor by >> hakostra Thu Nov 20 11:04:28 2014 >> [0]PETSC ERROR: Configure options --prefix=/opt/petsc/git-debug/install >> --with-shared-libraries --download-hypre --with-64-bit-indices --with-hdf5 >> --with-hdf5-dir=/opt/HDF5/1.8.13/install --with-debugging >> --with-debugger=gdb >> [0]PETSC ERROR: #1 KSPConvergedDefault() line 710 in >> /opt/petsc/git-debug/source/src/ksp/ksp/interface/iterativ.c >> [0]PETSC ERROR: #2 KSPSolve_PIPECG() line 99 in >> /opt/petsc/git-debug/source/src/ksp/ksp/impls/cg/pipecg/pipecg.c >> [0]PETSC ERROR: #3 KSPSolve() line 459 in /opt/petsc/git-debug/source/ >> src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: #4 findPressure() line 53 in /opt/MaDNS/source/src/FVM2.cpp >> > From haakon at hakostra.net Thu Nov 20 09:06:12 2014 From: haakon at hakostra.net (=?windows-1252?Q?H=E5kon_Strandenes?=) Date: Thu, 20 Nov 2014 16:06:12 +0100 Subject: [petsc-users] Pipelined solvers PIPECG etc fail In-Reply-To: <04D69684-F157-4ABA-AAEA-882BBF5E1B52@saschaschnepp.net> References: <04D69684-F157-4ABA-AAEA-882BBF5E1B52@saschaschnepp.net> Message-ID: <546E0364.8060702@hakostra.net> Ok: If I run KSP ex 2 with parameters "-ksp_type pipecg -ksp_norm_type natural", it works. If I try *any* other norm, it fails, but with a different error message, for example if I try "preconditioned": [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Scalar value must be same on all processes, argument # 2 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown [0]PETSC ERROR: ./ex2 on a arch-linux2-c-debug named hakostra-kontor by hakostra Thu Nov 20 16:04:37 2014 [0]PETSC ERROR: Configure options --prefix=/opt/petsc/git-debug/install --with-shared-libraries --download-hypre --with-64-bit-indices --with-hdf5 --with-hdf5-dir=/opt/HDF5/1.8.13/install --with-debugging --with-debugger=gdb [0]PETSC ERROR: #1 VecAXPY() line 634 in /opt/petsc/git-debug/source/src/vec/vec/interface/rvector.c [0]PETSC ERROR: #2 KSPSolve_PIPECG() line 153 in /opt/petsc/git-debug/source/src/ksp/ksp/impls/cg/pipecg/pipecg.c [0]PETSC ERROR: #3 KSPSolve() line 459 in /opt/petsc/git-debug/source/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #4 main() line 75 in /home/hakostra/Dropbox/KSP-ex2/ex2.c H?kon On 20. nov. 2014 15:49, Sascha Schnepp wrote: > Hi, > > this is an error that occurs when calling ksp->converged and having > ksp->normtype == KSP_NORM_NONE > > In pipecg.c the error occurs here: > line99: ierr = (*ksp->converged)(ksp,0,dp,&ksp->reason,ksp->cnvP);CHKERRQ(ierr); /* test for convergence */ > > It can be fixed (though there might be a nicer way) using this replacement code, wich explicitly calls KSPConvergendSkip for the respective norm: > if (ksp->normtype == KSP_NORM_NONE) { > ierr = KSPConvergedSkip (ksp,0,dp,&ksp->reason,ksp->cnvP);CHKERRQ(ierr); > } else { > ierr = (*ksp->converged)(ksp,0,dp,&ksp->reason,ksp->cnvP);CHKERRQ(ierr); > } > > > The issue could be related to this code in > KSPCreate_PIPECG > ierr = KSPSetSupportedNorm(ksp,KSP_NORM_UNPRECONDITIONED,PC_LEFT,2);CHKERRQ(ierr); > ierr = KSPSetSupportedNorm(ksp,KSP_NORM_PRECONDITIONED,PC_LEFT,2);CHKERRQ(ierr); > ierr = KSPSetSupportedNorm(ksp,KSP_NORM_NATURAL,PC_LEFT,2);CHKERRQ(ierr); > ierr = KSPSetSupportedNorm(ksp,KSP_NORM_NONE,PC_LEFT,2);CHKERRQ(ierr); > where all norms have the same priority. > > For the non-pipelined version of CG the unpreconditioned norm has a priority level of 3. > > The modifications are independent. Choosing different default priorities will still make line 99 fail if type NONE is chosen explicitly. > > Cheers, > Sascha > > > Begin forwarded message: > >> On Thu, Nov 20, 2014 at 4:05 AM, H?kon Strandenes >> wrote: >> >>> Hi, >>> >>> First: thanks to the list and Barry for solving my previous problem on >>> loading vectors from HDF5 files. It works great! >>> >>> Now a second problem: I want to try the pipelined KSP solvers, PIPECG in >>> particular. However, my program fails on the first KSPSolve() with the >>> message below. The same happens if I try PIPECR. >>> >> >> These are very new, so there could be bugs lurking. It would help our >> testing if this is >> reproducuble on a standard example. Does this happen if you try to solve >> KSP ex2 >> or SNES ex5 with PIPECG? >> >> Thanks, >> >> Matt >> >> >>> I have tried this on three MPI implementations, on and two of them fully >>> support MPI 3 (I have read http://www.mcs.anl.gov/petsc/ >>> documentation/faq.html#pipelined). I am using latest PETSc from Git repo. >>> >>> I have tried to turn off the preconditioner without any effect. >>> >>> The equation system is the result of a finite-volume discretization of a >>> Poisson-like equation if that helps anything. A normal CG solver works like >>> a charm, converging fast and effective. >>> >>> Are there any tricks to get this working? Anything special I have to set >>> and/or define to get this solver running? >>> >>> Thanks in advance. >>> >>> Best regards, >>> H?kon Strandenes >>> >>> >>> Error message: >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Object is in wrong state >>> [0]PETSC ERROR: Use KSPConvergedSkip() with KSPNormType of KSP_NORM_NONE >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown >>> [0]PETSC ERROR: madns on a arch-linux2-c-debug named hakostra-kontor by >>> hakostra Thu Nov 20 11:04:28 2014 >>> [0]PETSC ERROR: Configure options --prefix=/opt/petsc/git-debug/install >>> --with-shared-libraries --download-hypre --with-64-bit-indices --with-hdf5 >>> --with-hdf5-dir=/opt/HDF5/1.8.13/install --with-debugging >>> --with-debugger=gdb >>> [0]PETSC ERROR: #1 KSPConvergedDefault() line 710 in >>> /opt/petsc/git-debug/source/src/ksp/ksp/interface/iterativ.c >>> [0]PETSC ERROR: #2 KSPSolve_PIPECG() line 99 in >>> /opt/petsc/git-debug/source/src/ksp/ksp/impls/cg/pipecg/pipecg.c >>> [0]PETSC ERROR: #3 KSPSolve() line 459 in /opt/petsc/git-debug/source/ >>> src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: #4 findPressure() line 53 in /opt/MaDNS/source/src/FVM2.cpp >>> >> > > From jed at jedbrown.org Thu Nov 20 07:22:28 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 20 Nov 2014 07:22:28 -0600 Subject: [petsc-users] Pipelined solvers PIPECG etc fail In-Reply-To: References: <546DBCF9.6080407@hakostra.net> Message-ID: <87zjbmc8h7.fsf@jedbrown.org> Matthew Knepley writes: > These are very new, so there could be bugs lurking. It would help our > testing if this is > reproducuble on a standard example. Does this happen if you try to solve > KSP ex2 > or SNES ex5 with PIPECG? I can reproduce. PIPECG has evidently been broken recently. I'll debug today. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Thu Nov 20 09:09:19 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 20 Nov 2014 09:09:19 -0600 Subject: [petsc-users] Pipelined solvers PIPECG etc fail In-Reply-To: <04D69684-F157-4ABA-AAEA-882BBF5E1B52@saschaschnepp.net> References: <04D69684-F157-4ABA-AAEA-882BBF5E1B52@saschaschnepp.net> Message-ID: <87wq6pdi3k.fsf@jedbrown.org> Sascha Schnepp writes: > The issue could be related to this code in > KSPCreate_PIPECG > ierr = KSPSetSupportedNorm(ksp,KSP_NORM_UNPRECONDITIONED,PC_LEFT,2);CHKERRQ(ierr); > ierr = KSPSetSupportedNorm(ksp,KSP_NORM_PRECONDITIONED,PC_LEFT,2);CHKERRQ(ierr); > ierr = KSPSetSupportedNorm(ksp,KSP_NORM_NATURAL,PC_LEFT,2);CHKERRQ(ierr); > ierr = KSPSetSupportedNorm(ksp,KSP_NORM_NONE,PC_LEFT,2);CHKERRQ(ierr); > where all norms have the same priority. Yes, this commit was buggy, but somehow when we merged the PIPECG implementation, it didn't come with any tests, so we didn't notice. I'll fix it after my meeting this morning and we'll get it into the upcoming petsc-3.5.3. https://bitbucket.org/petsc/petsc/commits/04d66c6e00809e52f621f72696de30b33e3deda5 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From haakon at hakostra.net Thu Nov 20 09:56:11 2014 From: haakon at hakostra.net (=?windows-1252?Q?H=E5kon_Strandenes?=) Date: Thu, 20 Nov 2014 16:56:11 +0100 Subject: [petsc-users] Pipelined solvers PIPECG etc fail In-Reply-To: <87wq6pdi3k.fsf@jedbrown.org> References: <04D69684-F157-4ABA-AAEA-882BBF5E1B52@saschaschnepp.net> <87wq6pdi3k.fsf@jedbrown.org> Message-ID: <546E0F1B.6090203@hakostra.net> Another thing one might look at is to update some documentation related to KSPSetNormType: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetNormType.html KSP_NORM_NATURAL works for example also with KSPPIPECG (actually it's the only norm working ATM, see my last e-mail to the list)... H?kon On 20. nov. 2014 16:09, Jed Brown wrote: > Sascha Schnepp writes: > >> The issue could be related to this code in >> KSPCreate_PIPECG >> ierr = KSPSetSupportedNorm(ksp,KSP_NORM_UNPRECONDITIONED,PC_LEFT,2);CHKERRQ(ierr); >> ierr = KSPSetSupportedNorm(ksp,KSP_NORM_PRECONDITIONED,PC_LEFT,2);CHKERRQ(ierr); >> ierr = KSPSetSupportedNorm(ksp,KSP_NORM_NATURAL,PC_LEFT,2);CHKERRQ(ierr); >> ierr = KSPSetSupportedNorm(ksp,KSP_NORM_NONE,PC_LEFT,2);CHKERRQ(ierr); >> where all norms have the same priority. > > Yes, this commit was buggy, but somehow when we merged the PIPECG > implementation, it didn't come with any tests, so we didn't notice. > I'll fix it after my meeting this morning and we'll get it into the > upcoming petsc-3.5.3. > > https://bitbucket.org/petsc/petsc/commits/04d66c6e00809e52f621f72696de30b33e3deda5 > From mail at saschaschnepp.net Thu Nov 20 12:10:03 2014 From: mail at saschaschnepp.net (Sascha Schnepp) Date: Thu, 20 Nov 2014 19:10:03 +0100 Subject: [petsc-users] Pipelined solvers PIPECG etc fail In-Reply-To: <546E0364.8060702@hakostra.net> References: <04D69684-F157-4ABA-AAEA-882BBF5E1B52@saschaschnepp.net> <546E0364.8060702@hakostra.net> Message-ID: <05B6DD7C-C926-4927-8FDC-F6D876B23E76@saschaschnepp.net> Hi, I cannot reproduce this on my system. All norm types work for me. If you use KSP_NORM_NONE you have to add something like -ksp_max_it 30 but this works too. I have seen this error though. For me, it occurred when I messed up the memory space by allocating too few memory in some place and overwriting other memory in consequence. My guess is that the actual error occurs (long) before line 153 in pipecg.c Sascha On 20 Nov 2014, at 16:06, H?kon Strandenes wrote: > Ok: > > If I run KSP ex 2 with parameters "-ksp_type pipecg -ksp_norm_type natural", it works. > > If I try *any* other norm, it fails, but with a different error message, for example if I try "preconditioned": > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 2 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown > [0]PETSC ERROR: ./ex2 on a arch-linux2-c-debug named hakostra-kontor by hakostra Thu Nov 20 16:04:37 2014 > [0]PETSC ERROR: Configure options --prefix=/opt/petsc/git-debug/install --with-shared-libraries --download-hypre --with-64-bit-indices --with-hdf5 --with-hdf5-dir=/opt/HDF5/1.8.13/install --with-debugging --with-debugger=gdb > [0]PETSC ERROR: #1 VecAXPY() line 634 in /opt/petsc/git-debug/source/src/vec/vec/interface/rvector.c > [0]PETSC ERROR: #2 KSPSolve_PIPECG() line 153 in /opt/petsc/git-debug/source/src/ksp/ksp/impls/cg/pipecg/pipecg.c > [0]PETSC ERROR: #3 KSPSolve() line 459 in /opt/petsc/git-debug/source/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #4 main() line 75 in /home/hakostra/Dropbox/KSP-ex2/ex2.c > > H?kon > > > On 20. nov. 2014 15:49, Sascha Schnepp wrote: >> Hi, >> >> this is an error that occurs when calling ksp->converged and having >> ksp->normtype == KSP_NORM_NONE >> >> In pipecg.c the error occurs here: >> line99: ierr = (*ksp->converged)(ksp,0,dp,&ksp->reason,ksp->cnvP);CHKERRQ(ierr); /* test for convergence */ >> >> It can be fixed (though there might be a nicer way) using this replacement code, wich explicitly calls KSPConvergendSkip for the respective norm: >> if (ksp->normtype == KSP_NORM_NONE) { >> ierr = KSPConvergedSkip (ksp,0,dp,&ksp->reason,ksp->cnvP);CHKERRQ(ierr); >> } else { >> ierr = (*ksp->converged)(ksp,0,dp,&ksp->reason,ksp->cnvP);CHKERRQ(ierr); >> } >> >> >> The issue could be related to this code in >> KSPCreate_PIPECG >> ierr = KSPSetSupportedNorm(ksp,KSP_NORM_UNPRECONDITIONED,PC_LEFT,2);CHKERRQ(ierr); >> ierr = KSPSetSupportedNorm(ksp,KSP_NORM_PRECONDITIONED,PC_LEFT,2);CHKERRQ(ierr); >> ierr = KSPSetSupportedNorm(ksp,KSP_NORM_NATURAL,PC_LEFT,2);CHKERRQ(ierr); >> ierr = KSPSetSupportedNorm(ksp,KSP_NORM_NONE,PC_LEFT,2);CHKERRQ(ierr); >> where all norms have the same priority. >> >> For the non-pipelined version of CG the unpreconditioned norm has a priority level of 3. >> >> The modifications are independent. Choosing different default priorities will still make line 99 fail if type NONE is chosen explicitly. >> >> Cheers, >> Sascha >> >> >> Begin forwarded message: >> >>> On Thu, Nov 20, 2014 at 4:05 AM, H?kon Strandenes >>> wrote: >>> >>>> Hi, >>>> >>>> First: thanks to the list and Barry for solving my previous problem on >>>> loading vectors from HDF5 files. It works great! >>>> >>>> Now a second problem: I want to try the pipelined KSP solvers, PIPECG in >>>> particular. However, my program fails on the first KSPSolve() with the >>>> message below. The same happens if I try PIPECR. >>>> >>> >>> These are very new, so there could be bugs lurking. It would help our >>> testing if this is >>> reproducuble on a standard example. Does this happen if you try to solve >>> KSP ex2 >>> or SNES ex5 with PIPECG? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> I have tried this on three MPI implementations, on and two of them fully >>>> support MPI 3 (I have read http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html#pipelined). I am using latest PETSc from Git repo. >>>> >>>> I have tried to turn off the preconditioner without any effect. >>>> >>>> The equation system is the result of a finite-volume discretization of a >>>> Poisson-like equation if that helps anything. A normal CG solver works like >>>> a charm, converging fast and effective. >>>> >>>> Are there any tricks to get this working? Anything special I have to set >>>> and/or define to get this solver running? >>>> >>>> Thanks in advance. >>>> >>>> Best regards, >>>> H?kon Strandenes >>>> >>>> >>>> Error message: >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Object is in wrong state >>>> [0]PETSC ERROR: Use KSPConvergedSkip() with KSPNormType of KSP_NORM_NONE >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown >>>> [0]PETSC ERROR: madns on a arch-linux2-c-debug named hakostra-kontor by >>>> hakostra Thu Nov 20 11:04:28 2014 >>>> [0]PETSC ERROR: Configure options --prefix=/opt/petsc/git-debug/install >>>> --with-shared-libraries --download-hypre --with-64-bit-indices --with-hdf5 >>>> --with-hdf5-dir=/opt/HDF5/1.8.13/install --with-debugging >>>> --with-debugger=gdb >>>> [0]PETSC ERROR: #1 KSPConvergedDefault() line 710 in >>>> /opt/petsc/git-debug/source/src/ksp/ksp/interface/iterativ.c >>>> [0]PETSC ERROR: #2 KSPSolve_PIPECG() line 99 in >>>> /opt/petsc/git-debug/source/src/ksp/ksp/impls/cg/pipecg/pipecg.c >>>> [0]PETSC ERROR: #3 KSPSolve() line 459 in /opt/petsc/git-debug/source/ >>>> src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #4 findPressure() line 53 in /opt/MaDNS/source/src/FVM2.cpp >>>> >>> >> >> From graziosov at libero.it Thu Nov 20 21:19:39 2014 From: graziosov at libero.it (Valerio Grazioso) Date: Fri, 21 Nov 2014 04:19:39 +0100 Subject: [petsc-users] Fortran 77 VecGetArray problems Message-ID: <73DA0A26-8A55-45B3-BDF6-6F56DBC3A50E@libero.it> Dear Petsc Users, I?m trying to write a simple parallel linear solver (A*x=q) in fortran 77 with Petsc. I managed to Setup a MPIAIJ matrix A and a VecMPI qvec then the KSP and the solution. Everything seems fine (I?ve checked both A and q with MatView and VecView) but I?m having problems getting the solution in a ?normal? local double precision fortran vector. To be more than sure I?ve tried to repeat the necessary steps in order to achieve this result on qvec : After creating and calling a VecScatter in order to bring a sequential version on all the processors (as described in http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#mpi-vec-to-mpi-vec) I?ve tried to use VecGetValue as described in the User manual in the Fortran users chapter (pag. 147-148) It seems to fail in both the local xx_v vector (that has wrong values) and in calculating the i_x index offset that has a very large negative value that, when it is used, generates an obvious ?Aut of bound? error. I?ve tried to use VecGetValue also on qvec directly : same result! There must be something wrong that I?m doing. Can anybody point me in the right direction? Thanks Regards Valerio Grazioso Here it is the relevant code: #define xx_a(ib) xx_v(i_x + (ib)) ???????.. Vec xSEQ PetscScalar xx_v(1) PetscOffset i_x ????????.. ??????.. write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecCreate" call VecCreate(comm,qvec,ierr); CHKERRQ(ierr) write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecSetSizes" call VecSetSizes(qvec,PETSC_DECIDE,NCELL,ierr); CHKERRQ(ierr) write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecSetFromOptions" call VecSetFromOptions(qvec,ierr); CHKERRQ(ierr) write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecDuplicate" call VecDuplicate(qvec,xvec,ierr); CHKERRQ(ierr) write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecGetOwnershipRange" call VecGetOwnershipRange(qvec,Istart,Iend,ierr); CHKERRQ(ierr) write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> Istart,Iend (Vet)=", Istart,Iend do Iproc = 0,(mpi_size-1) if (Iproc == mpi_rank) then do I= (Istart+1), (Iend) C write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> MatSetValues",I,I call VecSetValues(qvec,1,I-1,SU(I),INSERT_VALUES,ierr); CHKERRQ(ierr) call VecSetValues(xvec,1,I-1,0.E0,INSERT_VALUES,ierr); CHKERRQ(ierr) enddo endif enddo write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecAssemblyBegin" call VecAssemblyBegin(qvec,ierr); CHKERRQ(ierr) call VecAssemblyEnd(qvec,ierr); CHKERRQ(ierr) call VecAssemblyBegin(xvec,ierr); CHKERRQ(ierr) call VecAssemblyEnd(xvec,ierr); CHKERRQ(ierr) write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecAssemblyEnd" C --------------------------------- DEBUG ------------------------------------- if (1 == 1) then call VecScatterCreateToAll(qvec,ctx,xSEQ,ierr); CHKERRQ(ierr) call VecScatterBegin(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) CHKERRQ(ierr) call VecScatterEnd(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) CHKERRQ(ierr) call VecScatterDestroy(ctx,ierr); CHKERRQ(ierr) C call PetscViewerCreate(comm, view, ierr); CHKERRQ(ierr) C call PetscViewerSetType(view, PETSCVIEWERASCII, ierr); CHKERRQ(ierr) C call PetscViewerSetFormat(view, PETSC_VIEWER_ASCII_INDEX, ierr) C CHKERRQ(ierr) C call PetscViewerFileSetName(view, 'qvec.m', ierr) C call VecView(xSEQ, view, ierr); CHKERRQ(ierr) C call PetscViewerDestroy(view, ierr); CHKERRQ(ierr) call VecGetArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) write(6, *) (xx_a(I), I=1,NCELL) write(6,'(a,i20)')"i_x = ", i_x call VecRestoreArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) call PetscFinalize(ierr); CHKERRQ(ierr) STOP endif C ---------------------------------- END DEBUG ------------------------------ From knepley at gmail.com Fri Nov 21 05:18:22 2014 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 21 Nov 2014 05:18:22 -0600 Subject: [petsc-users] Fortran 77 VecGetArray problems In-Reply-To: <73DA0A26-8A55-45B3-BDF6-6F56DBC3A50E@libero.it> References: <73DA0A26-8A55-45B3-BDF6-6F56DBC3A50E@libero.it> Message-ID: On Thu, Nov 20, 2014 at 9:19 PM, Valerio Grazioso wrote: > Dear Petsc Users, > I?m trying to write a simple parallel linear solver (A*x=q) in fortran 77 > with Petsc. > I managed to Setup a MPIAIJ matrix A and a VecMPI qvec > then the KSP and the solution. > Everything seems fine (I?ve checked both A and q with MatView and VecView) > but I?m having problems getting the solution in a ?normal? local double > precision fortran vector. > > To be more than sure I?ve tried to repeat the necessary steps in order to > achieve this result on qvec : > > After creating and calling a VecScatter in order to bring a sequential > version on all the processors (as described in > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#mpi-vec-to-mpi-vec > ) > I?ve tried to use VecGetValue as described in the User manual in the > Fortran users chapter (pag. 147-148) > It seems to fail in both the local xx_v vector (that has wrong values) and > in calculating the i_x index offset that has a very large negative value > that, when it is used, generates an obvious ?Aut of bound? error. > I?ve tried to use VecGetValue also on qvec directly : same result! > > There must be something wrong that I?m doing. > Can anybody point me in the right direction? > The code looks correct to me. As a first step, can you try running an example, to make sure its not a compiler problem. This example http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/examples/tutorials/ex4f.F.html should be enough to check. Thanks, Matt > Thanks > > Regards > Valerio Grazioso > > > Here it is the relevant code: > > > > #define xx_a(ib) xx_v(i_x + (ib)) > > ???????.. > > Vec xSEQ > PetscScalar xx_v(1) > PetscOffset i_x > > ????????.. > ??????.. > > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> > VecCreate" > call VecCreate(comm,qvec,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> > VecSetSizes" > call VecSetSizes(qvec,PETSC_DECIDE,NCELL,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> > VecSetFromOptions" > call VecSetFromOptions(qvec,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> > VecDuplicate" > call VecDuplicate(qvec,xvec,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> > VecGetOwnershipRange" > call VecGetOwnershipRange(qvec,Istart,Iend,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> > Istart,Iend (Vet)=", Istart,Iend > > do Iproc = 0,(mpi_size-1) > if (Iproc == mpi_rank) then > do I= (Istart+1), (Iend) > C write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve > ---> MatSetValues",I,I > call VecSetValues(qvec,1,I-1,SU(I),INSERT_VALUES,ierr); > CHKERRQ(ierr) > call VecSetValues(xvec,1,I-1,0.E0,INSERT_VALUES,ierr); > CHKERRQ(ierr) > enddo > endif > enddo > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> > VecAssemblyBegin" > call VecAssemblyBegin(qvec,ierr); CHKERRQ(ierr) > call VecAssemblyEnd(qvec,ierr); CHKERRQ(ierr) > call VecAssemblyBegin(xvec,ierr); CHKERRQ(ierr) > call VecAssemblyEnd(xvec,ierr); CHKERRQ(ierr) > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> > VecAssemblyEnd" > > C --------------------------------- DEBUG > ------------------------------------- > if (1 == 1) then > > call VecScatterCreateToAll(qvec,ctx,xSEQ,ierr); CHKERRQ(ierr) > call > VecScatterBegin(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) > CHKERRQ(ierr) > call > VecScatterEnd(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) > CHKERRQ(ierr) > call VecScatterDestroy(ctx,ierr); CHKERRQ(ierr) > > C call PetscViewerCreate(comm, view, ierr); CHKERRQ(ierr) > C call PetscViewerSetType(view, PETSCVIEWERASCII, ierr); > CHKERRQ(ierr) > C call PetscViewerSetFormat(view, PETSC_VIEWER_ASCII_INDEX, ierr) > C CHKERRQ(ierr) > C call PetscViewerFileSetName(view, 'qvec.m', ierr) > C call VecView(xSEQ, view, ierr); CHKERRQ(ierr) > C call PetscViewerDestroy(view, ierr); CHKERRQ(ierr) > > call VecGetArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) > > write(6, *) (xx_a(I), I=1,NCELL) > write(6,'(a,i20)')"i_x = ", i_x > > > call VecRestoreArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) > > call PetscFinalize(ierr); CHKERRQ(ierr) > STOP > endif > C ---------------------------------- END DEBUG > ------------------------------ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Nov 21 07:18:00 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 21 Nov 2014 07:18:00 -0600 Subject: [petsc-users] Fortran 77 VecGetArray problems In-Reply-To: <73DA0A26-8A55-45B3-BDF6-6F56DBC3A50E@libero.it> References: <73DA0A26-8A55-45B3-BDF6-6F56DBC3A50E@libero.it> Message-ID: <6D5C07E8-4693-4454-81A0-379DAA6B5395@mcs.anl.gov> > On Nov 20, 2014, at 9:19 PM, Valerio Grazioso wrote: > > Dear Petsc Users, > I?m trying to write a simple parallel linear solver (A*x=q) in fortran 77 with Petsc. > I managed to Setup a MPIAIJ matrix A and a VecMPI qvec > then the KSP and the solution. > Everything seems fine (I?ve checked both A and q with MatView and VecView) but I?m having problems getting the solution in a ?normal? local double precision fortran vector. > > To be more than sure I?ve tried to repeat the necessary steps in order to achieve this result on qvec : > > After creating and calling a VecScatter in order to bring a sequential version on all the processors (as described in > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#mpi-vec-to-mpi-vec) > I?ve tried to use VecGetValue as described in the User manual in the Fortran users chapter (pag. 147-148) > It seems to fail in both the local xx_v vector (that has wrong values) and in calculating the i_x index offset that has a very large negative value > that, when it is used, generates an obvious ?Aut of bound? error. It is completely possible that i_x is a very large negative value, in fact it usually is. From the users manual: Note: If using VecGetArray(), MatDenseGetArray(), or ISGetIndices(), from Fortran, the user must not compile the Fortran code with options to check for ?array entries out of bounds? (e.g., on the IBM RS/6000 this is done with the -C compiler option, so never use the -C option with this). My guess is that your Fortran compiler is checking for array out of bounds and (of course) finding them since xx_v() is declared as xx_v(1). You need to turn of this array bounds checking when compiling. Or you can use VecGetArrayF90(). Barry > I?ve tried to use VecGetValue also on qvec directly : same result! > > There must be something wrong that I?m doing. > Can anybody point me in the right direction? > Thanks > > Regards > Valerio Grazioso > > > Here it is the relevant code: > > > > #define xx_a(ib) xx_v(i_x + (ib)) > > ???????.. > > Vec xSEQ > PetscScalar xx_v(1) > PetscOffset i_x > > ????????.. > ??????.. > > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecCreate" > call VecCreate(comm,qvec,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecSetSizes" > call VecSetSizes(qvec,PETSC_DECIDE,NCELL,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecSetFromOptions" > call VecSetFromOptions(qvec,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecDuplicate" > call VecDuplicate(qvec,xvec,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecGetOwnershipRange" > call VecGetOwnershipRange(qvec,Istart,Iend,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> Istart,Iend (Vet)=", Istart,Iend > > do Iproc = 0,(mpi_size-1) > if (Iproc == mpi_rank) then > do I= (Istart+1), (Iend) > C write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> MatSetValues",I,I > call VecSetValues(qvec,1,I-1,SU(I),INSERT_VALUES,ierr); CHKERRQ(ierr) > call VecSetValues(xvec,1,I-1,0.E0,INSERT_VALUES,ierr); CHKERRQ(ierr) > enddo > endif > enddo > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecAssemblyBegin" > call VecAssemblyBegin(qvec,ierr); CHKERRQ(ierr) > call VecAssemblyEnd(qvec,ierr); CHKERRQ(ierr) > call VecAssemblyBegin(xvec,ierr); CHKERRQ(ierr) > call VecAssemblyEnd(xvec,ierr); CHKERRQ(ierr) > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecAssemblyEnd" > > C --------------------------------- DEBUG ------------------------------------- > if (1 == 1) then > > call VecScatterCreateToAll(qvec,ctx,xSEQ,ierr); CHKERRQ(ierr) > call VecScatterBegin(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) > CHKERRQ(ierr) > call VecScatterEnd(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) > CHKERRQ(ierr) > call VecScatterDestroy(ctx,ierr); CHKERRQ(ierr) > > C call PetscViewerCreate(comm, view, ierr); CHKERRQ(ierr) > C call PetscViewerSetType(view, PETSCVIEWERASCII, ierr); CHKERRQ(ierr) > C call PetscViewerSetFormat(view, PETSC_VIEWER_ASCII_INDEX, ierr) > C CHKERRQ(ierr) > C call PetscViewerFileSetName(view, 'qvec.m', ierr) > C call VecView(xSEQ, view, ierr); CHKERRQ(ierr) > C call PetscViewerDestroy(view, ierr); CHKERRQ(ierr) > > call VecGetArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) > > write(6, *) (xx_a(I), I=1,NCELL) > write(6,'(a,i20)')"i_x = ", i_x > > > call VecRestoreArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) > > call PetscFinalize(ierr); CHKERRQ(ierr) > STOP > endif > C ---------------------------------- END DEBUG ------------------------------ > > From paulhuaizhang at gmail.com Fri Nov 21 08:18:58 2014 From: paulhuaizhang at gmail.com (huaibao zhang) Date: Fri, 21 Nov 2014 09:18:58 -0500 Subject: [petsc-users] the specific preconditioner Message-ID: Hi Jed, I have a specific question regarding the attached my initialization function. Previously I used KSPSetType(ksp,KSPFGMRES) solver, and I would like to see if there is some improvement using the other choices like KSPBCGS. It seems hoverer, the PC //PCSetType(pc,PCBJACOBI) does not work with me. And I am not quite sure why. By the way, can you please give some suggestions on my code? I am not quite sure if I am doing the things rightly. Many thanks, Paul void petsc_init(void) { vector::iterator cit; vector::iterator it; //Create nonlinear solver context KSPCreate(PETSC_COMM_WORLD,&ksp); VecCreateMPI(PETSC_COMM_WORLD,grid[gid].cellCount*nVars,grid[gid].globalCellCount*nVars,&RHS); VecSetFromOptions(RHS); VecDuplicate(RHS,&deltaX); VecSet(RHS,0.); VecSet(deltaX,0.); vector diagonal_nonzeros, off_diagonal_nonzeros; int nextCellCount; // Calculate space necessary for matrix memory allocation for (cit=grid[gid].cell.begin();cit!=grid[gid].cell.end();cit++) { nextCellCount=0; for (it=(*cit).faces.begin();it!=(*cit).faces.end();it++) { if (grid[gid].face[*it].bc==INTERNAL_FACE) { nextCellCount++; } } for (int i=0;i From lb2653 at columbia.edu Fri Nov 21 09:55:57 2014 From: lb2653 at columbia.edu (Luc Berger-Vergiat) Date: Fri, 21 Nov 2014 10:55:57 -0500 Subject: [petsc-users] Error while calling a fieldsplit preconditioner containing a mg/gamg preconditioner Message-ID: <546F608D.2080606@columbi.edu> Hi all, I am using a DMShell to create to use a fieldsplit preconditioner. I would like to try some of petsc's multigrid options: mg and gamg. When I call my preconditioner: -ksp_type gmres -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type full -pc_fieldsplit_schur_precondition selfp -pc_fieldsplit_0_fields 2,3 -pc_fieldsplit_1_fields 0,1 -fieldsplit_0_ksp_type preonly -fieldsplit_0_pc_type hypre -fieldsplit_0_pc_hypre_type euclid -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type mg -malloc_log mlog -log_summary time.log -ksp_view it returns me the following error message and ksp_view: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: Must call DMShellSetGlobalVector() or DMShellSetCreateGlobalVector() [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 [0]PETSC ERROR: /home/luc/research/feap_repo/ShearBands/parfeap/feap on a arch-opt named euler by luc Fri Nov 21 10:12:53 2014 [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-debugging=0 --with-shared-libraries=0 --download-fblaslapack --download-mpich --download-parmetis --download-metis --download-ml=yes --download-hypre --download-superlu_dist --download-mumps --download-scalapack [0]PETSC ERROR: #259 DMCreateGlobalVector_Shell() line 245 in /home/luc/research/petsc-3.5.2/src/dm/impls/shell/dmshell.c [0]PETSC ERROR: #260 DMCreateGlobalVector() line 681 in /home/luc/research/petsc-3.5.2/src/dm/interface/dm.c [0]PETSC ERROR: #261 DMGetGlobalVector() line 154 in /home/luc/research/petsc-3.5.2/src/dm/interface/dmget.c KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-08, absolute=1e-16, divergence=1e+16 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: fieldsplit FieldSplit with Schur preconditioner, factorization FULL Preconditioner for the Schur complement formed from Sp, an assembled approximation to S, which uses (the lumped) A00's diagonal's inverse Split info: Split number 0 Defined by IS Split number 1 Defined by IS KSP solver for A00 block KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE Euclid preconditioning HYPRE Euclid: number of levels 1 linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=2000, cols=2000 total: nonzeros=40000, allocated nonzeros=40000 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 400 nodes, limit used is 5 KSP solver for S = A11 - A10 inv(A00) A01 KSP Object: (fieldsplit_1_) 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (fieldsplit_1_) 1 MPI processes type: mg MG: type is MULTIPLICATIVE, levels=1 cycles=v Cycles per PCApply=1 Not using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (fieldsplit_1_mg_levels_0_) 1 MPI processes type: chebyshev Chebyshev: eigenvalue estimates: min = 1.70057, max = 18.7063 Chebyshev: estimated using: [0 0.1; 0 1.1] KSP Object: (fieldsplit_1_mg_levels_0_est_) 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_1_mg_levels_0_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix followed by preconditioner matrix: Mat Object: (fieldsplit_1_) 1 MPI processes type: schurcomplement rows=330, cols=330 Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_1_) 1 MPI processes type: seqaij rows=330, cols=330 total: nonzeros=7642, allocated nonzeros=7642 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 A10 Mat Object: 1 MPI processes type: seqaij rows=330, cols=2000 total: nonzeros=22800, allocated nonzeros=22800 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 KSP of A00 KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE Euclid preconditioning HYPRE Euclid: number of levels 1 linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=2000, cols=2000 total: nonzeros=40000, allocated nonzeros=40000 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 400 nodes, limit used is 5 A01 Mat Object: 1 MPI processes type: seqaij rows=2000, cols=330 total: nonzeros=22800, allocated nonzeros=22800 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 400 nodes, limit used is 5 Mat Object: 1 MPI processes type: seqaij rows=330, cols=330 total: nonzeros=7642, allocated nonzeros=7642 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_1_mg_levels_0_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix followed by preconditioner matrix: Mat Object: (fieldsplit_1_) 1 MPI processes type: schurcomplement rows=330, cols=330 Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_1_) 1 MPI processes type: seqaij rows=330, cols=330 total: nonzeros=7642, allocated nonzeros=7642 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 A10 Mat Object: 1 MPI processes type: seqaij rows=330, cols=2000 total: nonzeros=22800, allocated nonzeros=22800 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 KSP of A00 KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE Euclid preconditioning HYPRE Euclid: number of levels 1 linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=2000, cols=2000 total: nonzeros=40000, allocated nonzeros=40000 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 400 nodes, limit used is 5 A01 Mat Object: 1 MPI processes type: seqaij rows=2000, cols=330 total: nonzeros=22800, allocated nonzeros=22800 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 400 nodes, limit used is 5 Mat Object: 1 MPI processes type: seqaij rows=330, cols=330 total: nonzeros=7642, allocated nonzeros=7642 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 linear system matrix followed by preconditioner matrix: Mat Object: (fieldsplit_1_) 1 MPI processes type: schurcomplement rows=330, cols=330 Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_1_) 1 MPI processes type: seqaij rows=330, cols=330 total: nonzeros=7642, allocated nonzeros=7642 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 A10 Mat Object: 1 MPI processes type: seqaij rows=330, cols=2000 total: nonzeros=22800, allocated nonzeros=22800 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 KSP of A00 KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE Euclid preconditioning HYPRE Euclid: number of levels 1 linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=2000, cols=2000 total: nonzeros=40000, allocated nonzeros=40000 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 400 nodes, limit used is 5 A01 Mat Object: 1 MPI processes type: seqaij rows=2000, cols=330 total: nonzeros=22800, allocated nonzeros=22800 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 400 nodes, limit used is 5 Mat Object: 1 MPI processes type: seqaij rows=330, cols=330 total: nonzeros=7642, allocated nonzeros=7642 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=2330, cols=2330 total: nonzeros=93242, allocated nonzeros=93242 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 521 nodes, limit used is 5 I am not completely surprised by this since the multigrid algorithm might want to know the structure of my vector in order to do a good job at restrictions and prolongations, etc... I am just not sure when I should call: DMShellSetGlobalVector() or DMShellSetCreateGlobalVector(). If I call DMShellSetGlobalVector() I assume that I need to generate my vector somehow before hand but I don't know what is required to make it a valid "template" vector? If I call DMShellSetCreateGlobalVector() I need to pass it a functions that computes a vector but what kind of vector? A residual or a "template"? This is not very clear to me... -- Best, Luc -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Nov 21 10:14:08 2014 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 21 Nov 2014 10:14:08 -0600 Subject: [petsc-users] Error while calling a fieldsplit preconditioner containing a mg/gamg preconditioner In-Reply-To: <546F608D.2080606@columbi.edu> References: <546F608D.2080606@columbi.edu> Message-ID: On Fri, Nov 21, 2014 at 9:55 AM, Luc Berger-Vergiat wrote: > Hi all, > I am using a DMShell to create to use a fieldsplit preconditioner. > I would like to try some of petsc's multigrid options: mg and gamg. > Lets start with gamg, since that does not need anything else from the user. If you want to use mg, then you will need to supply the interpolation/restriction operators since we cannot calculate them without an idea of the discretization. Thanks, Matt > When I call my preconditioner: > > -ksp_type gmres -pc_type fieldsplit -pc_fieldsplit_type schur > -pc_fieldsplit_schur_factorization_type full > -pc_fieldsplit_schur_precondition selfp -pc_fieldsplit_0_fields 2,3 > -pc_fieldsplit_1_fields 0,1 -fieldsplit_0_ksp_type preonly > -fieldsplit_0_pc_type hypre -fieldsplit_0_pc_hypre_type euclid > -fieldsplit_1_ksp_type gmres -fieldsplit_1_pc_type mg -malloc_log mlog > -log_summary time.log -ksp_view > > it returns me the following error message and ksp_view: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or > DMShellSetCreateGlobalVector() > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > [0]PETSC ERROR: /home/luc/research/feap_repo/ShearBands/parfeap/feap on a > arch-opt named euler by luc Fri Nov 21 10:12:53 2014 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --with-cxx=g++ --with-debugging=0 --with-shared-libraries=0 > --download-fblaslapack --download-mpich --download-parmetis > --download-metis --download-ml=yes --download-hypre --download-superlu_dist > --download-mumps --download-scalapack > [0]PETSC ERROR: #259 DMCreateGlobalVector_Shell() line 245 in > /home/luc/research/petsc-3.5.2/src/dm/impls/shell/dmshell.c > [0]PETSC ERROR: #260 DMCreateGlobalVector() line 681 in > /home/luc/research/petsc-3.5.2/src/dm/interface/dm.c > [0]PETSC ERROR: #261 DMGetGlobalVector() line 154 in > /home/luc/research/petsc-3.5.2/src/dm/interface/dmget.c > KSP Object: 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-08, absolute=1e-16, divergence=1e+16 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, factorization FULL > Preconditioner for the Schur complement formed from Sp, an assembled > approximation to S, which uses (the lumped) A00's diagonal's inverse > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: hypre > HYPRE Euclid preconditioning > HYPRE Euclid: number of levels 1 > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=2000, cols=2000 > total: nonzeros=40000, allocated nonzeros=40000 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 400 nodes, limit used is 5 > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (fieldsplit_1_) 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (fieldsplit_1_) 1 MPI processes > type: mg > MG: type is MULTIPLICATIVE, levels=1 cycles=v > Cycles per PCApply=1 > Not using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (fieldsplit_1_mg_levels_0_) 1 MPI > processes > type: chebyshev > Chebyshev: eigenvalue estimates: min = 1.70057, max = > 18.7063 > Chebyshev: estimated using: [0 0.1; 0 1.1] > KSP Object: > (fieldsplit_1_mg_levels_0_est_) 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) > Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: > (fieldsplit_1_mg_levels_0_) 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local > iterations = 1, omega = 1 > linear system matrix followed by preconditioner matrix: > Mat Object: (fieldsplit_1_) > 1 MPI processes > type: schurcomplement > rows=330, cols=330 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: > (fieldsplit_1_) 1 MPI processes > type: seqaij > rows=330, cols=330 > total: nonzeros=7642, allocated nonzeros=7642 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 121 nodes, limit > used is 5 > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=330, cols=2000 > total: nonzeros=22800, allocated nonzeros=22800 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 121 nodes, limit > used is 5 > KSP of A00 > KSP Object: > (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: > (fieldsplit_0_) 1 MPI processes > type: hypre > HYPRE Euclid preconditioning > HYPRE Euclid: number of levels 1 > linear system matrix = precond matrix: > Mat Object: > (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=2000, cols=2000 > total: nonzeros=40000, allocated nonzeros=40000 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 400 nodes, limit > used is 5 > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=2000, cols=330 > total: nonzeros=22800, allocated nonzeros=22800 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 400 nodes, limit > used is 5 > Mat Object: 1 MPI processes > type: seqaij > rows=330, cols=330 > total: nonzeros=7642, allocated nonzeros=7642 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 121 nodes, limit used is 5 > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_mg_levels_0_) 1 MPI > processes > type: sor > SOR: type = local_symmetric, iterations = 1, local > iterations = 1, omega = 1 > linear system matrix followed by preconditioner matrix: > Mat Object: (fieldsplit_1_) 1 MPI > processes > type: schurcomplement > rows=330, cols=330 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: > (fieldsplit_1_) 1 MPI processes > type: seqaij > rows=330, cols=330 > total: nonzeros=7642, allocated nonzeros=7642 > total number of mallocs used during MatSetValues calls > =0 > using I-node routines: found 121 nodes, limit used > is 5 > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=330, cols=2000 > total: nonzeros=22800, allocated nonzeros=22800 > total number of mallocs used during MatSetValues calls > =0 > using I-node routines: found 121 nodes, limit used > is 5 > KSP of A00 > KSP Object: > (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: > (fieldsplit_0_) 1 MPI processes > type: hypre > HYPRE Euclid preconditioning > HYPRE Euclid: number of levels 1 > linear system matrix = precond matrix: > Mat Object: > (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=2000, cols=2000 > total: nonzeros=40000, allocated nonzeros=40000 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 400 nodes, limit used > is 5 > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=2000, cols=330 > total: nonzeros=22800, allocated nonzeros=22800 > total number of mallocs used during MatSetValues calls > =0 > using I-node routines: found 400 nodes, limit used > is 5 > Mat Object: 1 MPI processes > type: seqaij > rows=330, cols=330 > total: nonzeros=7642, allocated nonzeros=7642 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 121 nodes, limit used is 5 > linear system matrix followed by preconditioner matrix: > Mat Object: (fieldsplit_1_) 1 MPI processes > type: schurcomplement > rows=330, cols=330 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (fieldsplit_1_) 1 MPI > processes > type: seqaij > rows=330, cols=330 > total: nonzeros=7642, allocated nonzeros=7642 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 121 nodes, limit used is 5 > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=330, cols=2000 > total: nonzeros=22800, allocated nonzeros=22800 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 121 nodes, limit used is 5 > KSP of A00 > KSP Object: (fieldsplit_0_) 1 MPI > processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI > processes > type: hypre > HYPRE Euclid preconditioning > HYPRE Euclid: number of levels 1 > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) > 1 MPI processes > type: seqaij > rows=2000, cols=2000 > total: nonzeros=40000, allocated nonzeros=40000 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 400 nodes, limit used is 5 > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=2000, cols=330 > total: nonzeros=22800, allocated nonzeros=22800 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 400 nodes, limit used is 5 > Mat Object: 1 MPI processes > type: seqaij > rows=330, cols=330 > total: nonzeros=7642, allocated nonzeros=7642 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 121 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=2330, cols=2330 > total: nonzeros=93242, allocated nonzeros=93242 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 521 nodes, limit used is 5 > > I am not completely surprised by this since the multigrid algorithm might > want to know the structure of my vector in order to do a good job at > restrictions and prolongations, etc... I am just not sure when I should > call: DMShellSetGlobalVector() or DMShellSetCreateGlobalVector(). > If I call DMShellSetGlobalVector() I assume that I need to generate my > vector somehow before hand but I don't know what is required to make it a > valid "template" vector? > If I call DMShellSetCreateGlobalVector() I need to pass it a functions > that computes a vector but what kind of vector? A residual or a "template"? > This is not very clear to me... > > -- > Best, > Luc > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lb2653 at columbia.edu Fri Nov 21 10:55:37 2014 From: lb2653 at columbia.edu (Luc Berger-Vergiat) Date: Fri, 21 Nov 2014 11:55:37 -0500 Subject: [petsc-users] Error while calling a fieldsplit preconditioner containing a mg/gamg preconditioner In-Reply-To: References: <546F608D.2080606@columbi.edu> Message-ID: <546F6E89.8000501@columbi.edu> Sure, so with gamg (which was my first choice too), I get the following: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: Must call DMShellSetGlobalVector() or DMShellSetCreateGlobalVector() [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 [0]PETSC ERROR: /home/luc/research/feap_repo/ShearBands/parfeap/feap on a arch-opt named euler by luc Thu Nov 20 16:19:29 2014 [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-debugging=0 --with-shared-libraries=0 --download-fblaslapack --download-mpich --download-parmetis --download-metis --download-ml=yes --download-hypre --download-superlu_dist --download-mumps --download-scalapack [0]PETSC ERROR: #145 DMCreateGlobalVector_Shell() line 245 in /home/luc/research/petsc-3.5.2/src/dm/impls/shell/dmshell.c [0]PETSC ERROR: #146 DMCreateGlobalVector() line 681 in /home/luc/research/petsc-3.5.2/src/dm/interface/dm.c [0]PETSC ERROR: #147 DMGetGlobalVector() line 154 in /home/luc/research/petsc-3.5.2/src/dm/interface/dmget.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: Must call DMShellSetGlobalVector() or DMShellSetCreateGlobalVector() [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 [0]PETSC ERROR: /home/luc/research/feap_repo/ShearBands/parfeap/feap on a arch-opt named euler by luc Thu Nov 20 16:19:29 2014 [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-debugging=0 --with-shared-libraries=0 --download-fblaslapack --download-mpich --download-parmetis --download-metis --download-ml=yes --download-hypre --download-superlu_dist --download-mumps --download-scalapack [0]PETSC ERROR: #148 DMCreateGlobalVector_Shell() line 245 in /home/luc/research/petsc-3.5.2/src/dm/impls/shell/dmshell.c [0]PETSC ERROR: #149 DMCreateGlobalVector() line 681 in /home/luc/research/petsc-3.5.2/src/dm/interface/dm.c [0]PETSC ERROR: #150 DMGetGlobalVector() line 154 in /home/luc/research/petsc-3.5.2/src/dm/interface/dmget.c KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-08, absolute=1e-16, divergence=1e+16 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: fieldsplit FieldSplit with Schur preconditioner, factorization FULL Preconditioner for the Schur complement formed from Sp, an assembled approximation to S, which uses (the lumped) A00's diagonal's inverse Split info: Split number 0 Defined by IS Split number 1 Defined by IS KSP solver for A00 block KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE Euclid preconditioning HYPRE Euclid: number of levels 1 linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=2000, cols=2000 total: nonzeros=40000, allocated nonzeros=40000 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 400 nodes, limit used is 5 KSP solver for S = A11 - A10 inv(A00) A01 KSP Object: (fieldsplit_1_) 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (fieldsplit_1_) 1 MPI processes type: gamg MG: type is MULTIPLICATIVE, levels=2 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (fieldsplit_1_mg_coarse_) 1 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_1_mg_coarse_) 1 MPI processes type: bjacobi block Jacobi: number of blocks = 1 Local solve is same for all blocks, in the following KSP and PC objects: KSP Object: (fieldsplit_1_mg_coarse_sub_) 1 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_1_mg_coarse_sub_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 using diagonal shift on blocks to prevent zero pivot [INBLOCKS] matrix ordering: nd factor fill ratio given 5, needed 1.22785 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=13, cols=13 package used to perform factorization: petsc total: nonzeros=97, allocated nonzeros=97 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 10 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=13, cols=13 total: nonzeros=79, allocated nonzeros=79 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=13, cols=13 total: nonzeros=79, allocated nonzeros=79 total number of mallocs used during MatSetValues calls =0 not using I-node routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (fieldsplit_1_mg_levels_1_) 1 MPI processes type: chebyshev Chebyshev: eigenvalue estimates: min = 0.0979919, max = 2.05783 maximum iterations=2 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (fieldsplit_1_mg_levels_1_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix followed by preconditioner matrix: Mat Object: (fieldsplit_1_) 1 MPI processes type: schurcomplement rows=330, cols=330 Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_1_) 1 MPI processes type: seqaij rows=330, cols=330 total: nonzeros=7642, allocated nonzeros=7642 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 A10 Mat Object: 1 MPI processes type: seqaij rows=330, cols=2000 total: nonzeros=22800, allocated nonzeros=22800 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 KSP of A00 KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE Euclid preconditioning HYPRE Euclid: number of levels 1 linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=2000, cols=2000 total: nonzeros=40000, allocated nonzeros=40000 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 400 nodes, limit used is 5 A01 Mat Object: 1 MPI processes type: seqaij rows=2000, cols=330 total: nonzeros=22800, allocated nonzeros=22800 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 400 nodes, limit used is 5 Mat Object: 1 MPI processes type: seqaij rows=330, cols=330 total: nonzeros=7642, allocated nonzeros=7642 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix followed by preconditioner matrix: Mat Object: (fieldsplit_1_) 1 MPI processes type: schurcomplement rows=330, cols=330 Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_1_) 1 MPI processes type: seqaij rows=330, cols=330 total: nonzeros=7642, allocated nonzeros=7642 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 A10 Mat Object: 1 MPI processes type: seqaij rows=330, cols=2000 total: nonzeros=22800, allocated nonzeros=22800 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 KSP of A00 KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: hypre HYPRE Euclid preconditioning HYPRE Euclid: number of levels 1 linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=2000, cols=2000 total: nonzeros=40000, allocated nonzeros=40000 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 400 nodes, limit used is 5 A01 Mat Object: 1 MPI processes type: seqaij rows=2000, cols=330 total: nonzeros=22800, allocated nonzeros=22800 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 400 nodes, limit used is 5 Mat Object: 1 MPI processes type: seqaij rows=330, cols=330 total: nonzeros=7642, allocated nonzeros=7642 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 121 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=2330, cols=2330 total: nonzeros=93242, allocated nonzeros=93242 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 521 nodes, limit used is 5 I am still missing the same call to DMShellSetCreateGlobalVector()/DMShellSetGlobalVector(), the only difference is that I get this error message twice : ) FYI this how I set my DMShell, KSP and PC: c Set up the number of fields, the section dofs and the fields sections dofs call PetscSectionCreate(PETSC_COMM_WORLD, FSSection, ierr) call PetscSectionSetNumFields(FSSection, numFields, ierr) call PetscSectionSetChart(FSSection, 1, numpeq+1, ierr) call PetscSectionGetChart(FSSection, NF, NE, ierr) do some loop over nodes and dofs call PetscSectionSetDof(FSSection, kk, 1, ierr) call PetscSectionSetFieldDof(FSSection, kk, jj-1,1, ierr) enddo call PetscSectionSetUp(FSSection, ierr) c Create the DM and set the default section if(DM_flg) then call DMShellCreate(PETSC_COMM_WORLD,FSDM, ierr) call DMSetDefaultSection(FSDM, FSSection, ierr) call DMSetUp(FSDM, ierr) endif call KSPCreate (PETSC_COMM_WORLD, kspsol ,ierr) call PetscOptionsGetString(PETSC_NULL_CHARACTER,'-pc_type', & pctype,chk,ierr) c Check for fieldsplit preconditioner to assign a DM to the KSP if (chk .eqv. PETSC_TRUE) then if(pctype.eq.'fieldsplit') then call KSPSetDM(kspsol, FSDM, ierr) call KSPSetDMActive(kspsol, PETSC_FALSE, ierr) endif endif c call KSPSetOperators (kspsol, Kmat, Kmat, c & DIFFERENT_NONZERO_PATTERN ,ierr) call KSPSetOperators(kspsol, Kmat, Kmat, ierr) c Options from command line call KSPSetFromOptions(kspsol,ierr) call KSPGetPC (kspsol, pc , ierr) call PCSetCoordinates(pc,ndm,numpn,hr(np(43)),ierr) Best, Luc On 11/21/2014 11:14 AM, Matthew Knepley wrote: > On Fri, Nov 21, 2014 at 9:55 AM, Luc Berger-Vergiat > > wrote: > > Hi all, > I am using a DMShell to create to use a fieldsplit preconditioner. > I would like to try some of petsc's multigrid options: mg and gamg. > > > Lets start with gamg, since that does not need anything else from the > user. If you want to use mg, > then you will need to supply the interpolation/restriction operators > since we cannot calculate them > without an idea of the discretization. > > Thanks, > > Matt > > When I call my preconditioner: > > -ksp_type gmres -pc_type fieldsplit -pc_fieldsplit_type schur > -pc_fieldsplit_schur_factorization_type full > -pc_fieldsplit_schur_precondition selfp > -pc_fieldsplit_0_fields 2,3 -pc_fieldsplit_1_fields 0,1 > -fieldsplit_0_ksp_type preonly -fieldsplit_0_pc_type hypre > -fieldsplit_0_pc_hypre_type euclid -fieldsplit_1_ksp_type > gmres -fieldsplit_1_pc_type mg -malloc_log mlog -log_summary > time.log -ksp_view > > it returns me the following error message and ksp_view: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or > DMShellSetCreateGlobalVector() > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > [0]PETSC ERROR: > /home/luc/research/feap_repo/ShearBands/parfeap/feap on a arch-opt > named euler by luc Fri Nov 21 10:12:53 2014 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --with-cxx=g++ --with-debugging=0 --with-shared-libraries=0 > --download-fblaslapack --download-mpich --download-parmetis > --download-metis --download-ml=yes --download-hypre > --download-superlu_dist --download-mumps --download-scalapack > [0]PETSC ERROR: #259 DMCreateGlobalVector_Shell() line 245 in > /home/luc/research/petsc-3.5.2/src/dm/impls/shell/dmshell.c > [0]PETSC ERROR: #260 DMCreateGlobalVector() line 681 in > /home/luc/research/petsc-3.5.2/src/dm/interface/dm.c > [0]PETSC ERROR: #261 DMGetGlobalVector() line 154 in > /home/luc/research/petsc-3.5.2/src/dm/interface/dmget.c > KSP Object: 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-08, absolute=1e-16, divergence=1e+16 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, factorization FULL > Preconditioner for the Schur complement formed from Sp, an > assembled approximation to S, which uses (the lumped) A00's > diagonal's inverse > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: hypre > HYPRE Euclid preconditioning > HYPRE Euclid: number of levels 1 > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=2000, cols=2000 > total: nonzeros=40000, allocated nonzeros=40000 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 400 nodes, limit used is 5 > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (fieldsplit_1_) 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) > Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (fieldsplit_1_) 1 MPI processes > type: mg > MG: type is MULTIPLICATIVE, levels=1 cycles=v > Cycles per PCApply=1 > Not using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (fieldsplit_1_mg_levels_0_) 1 MPI > processes > type: chebyshev > Chebyshev: eigenvalue estimates: min = 1.70057, max > = 18.7063 > Chebyshev: estimated using: [0 0.1; 0 1.1] > KSP Object: > (fieldsplit_1_mg_levels_0_est_) 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) > Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_mg_levels_0_) > 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, > local iterations = 1, omega = 1 > linear system matrix followed by preconditioner > matrix: > Mat Object: (fieldsplit_1_) 1 MPI > processes > type: schurcomplement > rows=330, cols=330 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: > (fieldsplit_1_) 1 MPI processes > type: seqaij > rows=330, cols=330 > total: nonzeros=7642, allocated nonzeros=7642 > total number of mallocs used during > MatSetValues calls =0 > using I-node routines: found 121 nodes, > limit used is 5 > A10 > Mat Object: 1 MPI > processes > type: seqaij > rows=330, cols=2000 > total: nonzeros=22800, allocated > nonzeros=22800 > total number of mallocs used during > MatSetValues calls =0 > using I-node routines: found 121 nodes, > limit used is 5 > KSP of A00 > KSP Object: > (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is > zero > tolerances: relative=1e-05, > absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: > (fieldsplit_0_) 1 MPI processes > type: hypre > HYPRE Euclid preconditioning > HYPRE Euclid: number of levels 1 > linear system matrix = precond matrix: > Mat Object: > (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=2000, cols=2000 > total: nonzeros=40000, allocated > nonzeros=40000 > total number of mallocs used during > MatSetValues calls =0 > using I-node routines: found 400 > nodes, limit used is 5 > A01 > Mat Object: 1 MPI > processes > type: seqaij > rows=2000, cols=330 > total: nonzeros=22800, allocated > nonzeros=22800 > total number of mallocs used during > MatSetValues calls =0 > using I-node routines: found 400 nodes, > limit used is 5 > Mat Object: 1 MPI processes > type: seqaij > rows=330, cols=330 > total: nonzeros=7642, allocated nonzeros=7642 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 121 nodes, limit > used is 5 > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_1_mg_levels_0_) 1 MPI > processes > type: sor > SOR: type = local_symmetric, iterations = 1, local > iterations = 1, omega = 1 > linear system matrix followed by preconditioner matrix: > Mat Object: (fieldsplit_1_) 1 MPI processes > type: schurcomplement > rows=330, cols=330 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (fieldsplit_1_) 1 > MPI processes > type: seqaij > rows=330, cols=330 > total: nonzeros=7642, allocated nonzeros=7642 > total number of mallocs used during > MatSetValues calls =0 > using I-node routines: found 121 nodes, > limit used is 5 > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=330, cols=2000 > total: nonzeros=22800, allocated nonzeros=22800 > total number of mallocs used during > MatSetValues calls =0 > using I-node routines: found 121 nodes, > limit used is 5 > KSP of A00 > KSP Object: (fieldsplit_0_) 1 > MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_0_) 1 > MPI processes > type: hypre > HYPRE Euclid preconditioning > HYPRE Euclid: number of levels 1 > linear system matrix = precond matrix: > Mat Object: > (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=2000, cols=2000 > total: nonzeros=40000, allocated nonzeros=40000 > total number of mallocs used during > MatSetValues calls =0 > using I-node routines: found 400 nodes, > limit used is 5 > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=2000, cols=330 > total: nonzeros=22800, allocated nonzeros=22800 > total number of mallocs used during > MatSetValues calls =0 > using I-node routines: found 400 nodes, > limit used is 5 > Mat Object: 1 MPI processes > type: seqaij > rows=330, cols=330 > total: nonzeros=7642, allocated nonzeros=7642 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 121 nodes, limit used > is 5 > linear system matrix followed by preconditioner matrix: > Mat Object: (fieldsplit_1_) 1 MPI processes > type: schurcomplement > rows=330, cols=330 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (fieldsplit_1_) 1 MPI > processes > type: seqaij > rows=330, cols=330 > total: nonzeros=7642, allocated nonzeros=7642 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 121 nodes, limit > used is 5 > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=330, cols=2000 > total: nonzeros=22800, allocated nonzeros=22800 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 121 nodes, limit > used is 5 > KSP of A00 > KSP Object: (fieldsplit_0_) 1 MPI > processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: hypre > HYPRE Euclid preconditioning > HYPRE Euclid: number of levels 1 > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI > processes > type: seqaij > rows=2000, cols=2000 > total: nonzeros=40000, allocated nonzeros=40000 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 400 nodes, limit > used is 5 > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=2000, cols=330 > total: nonzeros=22800, allocated nonzeros=22800 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 400 nodes, limit > used is 5 > Mat Object: 1 MPI processes > type: seqaij > rows=330, cols=330 > total: nonzeros=7642, allocated nonzeros=7642 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 121 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=2330, cols=2330 > total: nonzeros=93242, allocated nonzeros=93242 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 521 nodes, limit used is 5 > > I am not completely surprised by this since the multigrid > algorithm might want to know the structure of my vector in order > to do a good job at restrictions and prolongations, etc... I am > just not sure when I should call: DMShellSetGlobalVector() or > DMShellSetCreateGlobalVector(). > If I call DMShellSetGlobalVector() I assume that I need to > generate my vector somehow before hand but I don't know what is > required to make it a valid "template" vector? > If I call DMShellSetCreateGlobalVector() I need to pass it a > functions that computes a vector but what kind of vector? A > residual or a "template"? This is not very clear to me... > > -- > Best, > Luc > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Fri Nov 21 11:08:51 2014 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Fri, 21 Nov 2014 17:08:51 +0000 Subject: [petsc-users] Error while calling a fieldsplit preconditioner containing a mg/gamg preconditioner In-Reply-To: <546F6E89.8000501@columbi.edu> References: <546F608D.2080606@columbi.edu> <546F6E89.8000501@columbi.edu> Message-ID: > On 21 Nov 2014, at 16:55, Luc Berger-Vergiat wrote: > > Sure, so with gamg (which was my first choice too), I get the following: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or DMShellSetCreateGlobalVector() > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > [0]PETSC ERROR: /home/luc/research/feap_repo/ShearBands/parfeap/feap on a arch-opt named euler by I wonder if this is the same problem described here https://bitbucket.org/petsc/petsc/pull-request/221 Where the ksp contains an inactive dmshell that doesn't know how to create vectors Lawrence -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.antonio.magri at gmail.com Fri Nov 21 11:23:03 2014 From: victor.antonio.magri at gmail.com (Victor Magri) Date: Fri, 21 Nov 2014 15:23:03 -0200 Subject: [petsc-users] Why does CHOLMOD solution time differs between PETSc and MATLAB? Message-ID: Hi, I'm solving a linear system which comes from the discretization of an elliptic equation. The mesh has 1 million control volumes and there are two more constraints for the system. First, I've solved it in MATLAB using backslash command, which in turn calls CHOLMOD once the matrix is SPD. Using tic and toc funtions, I could see that it takes about 2s for the system to be solved (See attached for more details). I wrote both matrix and rhs in PETSc binary format and solved the linear system using the example problem defined in ksp/examples/tutorials/ex10.c. The command line options were: mpiexec -n -1 ./ex10 -f A1mi.dat -rhs rhs1mi.dat -log_summary -ksp_monitos -ksp_view -ksp_type preonly -pc_type cholesky -pc_factor_mat_solver_package cholmod The configuration parameters for CHOLMOD in PETSc and MATLAB seems the same, so I was expecting a similar computation time, but it took about 13s to solve in PETSc. (See attached for more details). Why does CHOLMOD solution time differs between them? If someone wants to try for yourself, here are the binaries for the matrix and the rhs, respectively. https://www.dropbox.com/s/q9s6mlrmv1qdxon/A1mi.dat?dl=0 https://www.dropbox.com/s/2zadcbjg5d9bycy/rhs1mi.dat?dl=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PetscOutput.dat Type: application/octet-stream Size: 14006 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MatlabOutput.dat Type: application/octet-stream Size: 2158 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri Nov 21 11:39:29 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 21 Nov 2014 11:39:29 -0600 Subject: [petsc-users] the specific preconditioner In-Reply-To: References: Message-ID: You need to call KSPGetPC(ksp,&pc); before setting the pc Note that since you call KSPSetFromOptions(ksp); you can control the KSP's and PC's from the command line and don't need to edit the code and recompile it for each change to the KSP or PC. For example -ksp_type bcgs -pc_type jacobi Barry > On Nov 21, 2014, at 8:18 AM, huaibao zhang wrote: > > Hi Jed, > > I have a specific question regarding the attached my initialization function. Previously I used KSPSetType(ksp,KSPFGMRES) solver, and I would like to see if there is some improvement using the other choices like KSPBCGS. It seems hoverer, the PC //PCSetType(pc,PCBJACOBI) does not work with me. And I am not quite sure why. > > By the way, can you please give some suggestions on my code? I am not quite sure if I am doing the things rightly. > > > Many thanks, > Paul > > > > > void petsc_init(void) { > > vector::iterator cit; > vector::iterator it; > > //Create nonlinear solver context > KSPCreate(PETSC_COMM_WORLD,&ksp); > > VecCreateMPI(PETSC_COMM_WORLD,grid[gid].cellCount*nVars,grid[gid].globalCellCount*nVars,&RHS); > VecSetFromOptions(RHS); > VecDuplicate(RHS,&deltaX); > > > VecSet(RHS,0.); > VecSet(deltaX,0.); > > vector diagonal_nonzeros, off_diagonal_nonzeros; > int nextCellCount; > > // Calculate space necessary for matrix memory allocation > for (cit=grid[gid].cell.begin();cit!=grid[gid].cell.end();cit++) { > nextCellCount=0; > for (it=(*cit).faces.begin();it!=(*cit).faces.end();it++) { > if (grid[gid].face[*it].bc==INTERNAL_FACE) { > nextCellCount++; > } > } > for (int i=0;i diagonal_nonzeros.push_back( (nextCellCount+1)*nVars); > off_diagonal_nonzeros.push_back( ((*cit).ghosts.size())*nVars); > } > } > > MatCreateMPIAIJ( > PETSC_COMM_WORLD, > grid[gid].cellCount*nVars, > grid[gid].cellCount*nVars, > grid[gid].globalCellCount*nVars, > grid[gid].globalCellCount*nVars, > 0,&diagonal_nonzeros[0], > 0,&off_diagonal_nonzeros[0], > &impOP); > > KSPSetOperators(ksp,impOP,impOP,SAME_NONZERO_PATTERN); > > KSPSetTolerances(ksp,rtol,abstol,1.e15,maxits); > KSPSetType(ksp, KSPBCGS); > //PCSetType(pc,PCBJACOBI); > KSPGMRESSetRestart(ksp,30); > KSPSetFromOptions(ksp); > > return; > } From bsmith at mcs.anl.gov Fri Nov 21 12:49:26 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 21 Nov 2014 12:49:26 -0600 Subject: [petsc-users] Why does CHOLMOD solution time differs between PETSc and MATLAB? In-Reply-To: References: Message-ID: <6BF6951C-282C-4321-B086-EAE204180F77@mcs.anl.gov> Victor, Here is my conjecture (more than a guess, but less than a full statement of fact). CHOLMOD uses a "supernodel" algorithm which means that it works on dense blocks of the factored matrix and can take advantage of BLAS 2 and 3 operations (unlike almost everything in PETSc). You have installed PETSc to use the --download-fblaslapack which is just the reference implementation of BLAS/LAPACK in Fortran. Meanwhile MATLAB uses the Intel MKL libraries for its BLAS/LAPACK 1) the MKL BLAS/LAPACK are likely much faster than the Fortran versions 2) it is possible the MKL BLAS/LAPACK used by MATLAB use multiple threads to take advantage of multiple cores on the hardware. So to improve your performance of the PETSc version you need to switch to MKL (and possibly turn on multithreaded version of MKL). No MKL installed on your machine? If you have MATLAB on the machine then you have MKL, for example on my Mac it is at /Applications/MATLAB_R2014b.app/bin/maci64/mkl.dylib and /Applications/MATLAB_R2014b.app/bin/maci64/mklcompat.dylib Also note that MATLAB's MKL uses 64 bit integers so you need to configure PETSc with the additional option --known-64-bit-blas-indices I think you are likely to get close to the MATLAB performance of CHOLMOD if you switch to using MKL blas/lapack with multithreading. Barry > On Nov 21, 2014, at 11:23 AM, Victor Magri wrote: > > Hi, > > I'm solving a linear system which comes from the discretization of an elliptic equation. The mesh has 1 million control volumes and there are two more constraints for the system. > > First, I've solved it in MATLAB using backslash command, which in turn calls CHOLMOD once the matrix is SPD. Using tic and toc funtions, I could see that it takes about 2s for the system to be solved (See attached for more details). > > I wrote both matrix and rhs in PETSc binary format and solved the linear system using the example problem defined in ksp/examples/tutorials/ex10.c. The command line options were: > > mpiexec -n -1 ./ex10 -f A1mi.dat -rhs rhs1mi.dat -log_summary -ksp_monitos -ksp_view -ksp_type preonly -pc_type cholesky -pc_factor_mat_solver_package cholmod > > The configuration parameters for CHOLMOD in PETSc and MATLAB seems the same, so I was expecting a similar computation time, but it took about 13s to solve in PETSc. (See attached for more details). > > Why does CHOLMOD solution time differs between them? > > If someone wants to try for yourself, here are the binaries for the matrix and the rhs, respectively. > https://www.dropbox.com/s/q9s6mlrmv1qdxon/A1mi.dat?dl=0 > https://www.dropbox.com/s/2zadcbjg5d9bycy/rhs1mi.dat?dl=0 > > > > From abhyshr at mcs.anl.gov Fri Nov 21 13:46:24 2014 From: abhyshr at mcs.anl.gov (Abhyankar, Shrirang G.) Date: Fri, 21 Nov 2014 19:46:24 +0000 Subject: [petsc-users] Why does CHOLMOD solution time differs between PETSc and MATLAB? In-Reply-To: References: Message-ID: <98D4398D-7858-4CDA-A534-7A38F3E5BB1B@anl.gov> I think this may have to do with the matrix reodering scheme. I think CHOLMOD, used by MATLAB backslash operator, uses AMD (approximate min. degree) ordering scheme by default. PETSc's default reordering scheme is nested dissection. For PETSc + MATLAB, use the option -pc_factor_mat_ordering_type qmd Shri On Nov 21, 2014, at 11:23 AM, "Victor Magri" > wrote: Hi, I'm solving a linear system which comes from the discretization of an elliptic equation. The mesh has 1 million control volumes and there are two more constraints for the system. First, I've solved it in MATLAB using backslash command, which in turn calls CHOLMOD once the matrix is SPD. Using tic and toc funtions, I could see that it takes about 2s for the system to be solved (See attached for more details). I wrote both matrix and rhs in PETSc binary format and solved the linear system using the example problem defined in ksp/examples/tutorials/ex10.c. The command line options were: mpiexec -n -1 ./ex10 -f A1mi.dat -rhs rhs1mi.dat -log_summary -ksp_monitos -ksp_view -ksp_type preonly -pc_type cholesky -pc_factor_mat_solver_package cholmod The configuration parameters for CHOLMOD in PETSc and MATLAB seems the same, so I was expecting a similar computation time, but it took about 13s to solve in PETSc. (See attached for more details). Why does CHOLMOD solution time differs between them? If someone wants to try for yourself, here are the binaries for the matrix and the rhs, respectively. https://www.dropbox.com/s/q9s6mlrmv1qdxon/A1mi.dat?dl=0 https://www.dropbox.com/s/2zadcbjg5d9bycy/rhs1mi.dat?dl=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Nov 21 14:34:58 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 21 Nov 2014 14:34:58 -0600 Subject: [petsc-users] Why does CHOLMOD solution time differs between PETSc and MATLAB? In-Reply-To: <98D4398D-7858-4CDA-A534-7A38F3E5BB1B@anl.gov> References: <98D4398D-7858-4CDA-A534-7A38F3E5BB1B@anl.gov> Message-ID: > On Nov 21, 2014, at 1:46 PM, Abhyankar, Shrirang G. wrote: > > I think this may have to do with the matrix reodering scheme. I think CHOLMOD, used by MATLAB backslash operator, uses AMD (approximate min. degree) ordering scheme by default. PETSc's default reordering scheme is nested dissection. For PETSc + MATLAB, use the option -pc_factor_mat_ordering_type qmd This is possible and definitely worth trying with different orders, but I checked the nonzeros in the matrix and the L factor for both approaches and they are very similar PETSc call: Common.fl 1.81877e+10 (flop count from most recent analysis) Common.lnz 4.46748e+07 (fundamental nz in L) MatLab call: flop count: 1.8188e+10 nnz(L): 4.4675e+07 Unless I am misunderstanding something both factors likely look very similar. Barry > > Shri > > On Nov 21, 2014, at 11:23 AM, "Victor Magri" wrote: > >> Hi, >> >> I'm solving a linear system which comes from the discretization of an elliptic equation. The mesh has 1 million control volumes and there are two more constraints for the system. >> >> First, I've solved it in MATLAB using backslash command, which in turn calls CHOLMOD once the matrix is SPD. Using tic and toc funtions, I could see that it takes about 2s for the system to be solved (See attached for more details). >> >> I wrote both matrix and rhs in PETSc binary format and solved the linear system using the example problem defined in ksp/examples/tutorials/ex10.c. The command line options were: >> >> mpiexec -n -1 ./ex10 -f A1mi.dat -rhs rhs1mi.dat -log_summary -ksp_monitos -ksp_view -ksp_type preonly -pc_type cholesky -pc_factor_mat_solver_package cholmod >> >> The configuration parameters for CHOLMOD in PETSc and MATLAB seems the same, so I was expecting a similar computation time, but it took about 13s to solve in PETSc. (See attached for more details). >> >> Why does CHOLMOD solution time differs between them? >> >> If someone wants to try for yourself, here are the binaries for the matrix and the rhs, respectively. >> https://www.dropbox.com/s/q9s6mlrmv1qdxon/A1mi.dat?dl=0 >> https://www.dropbox.com/s/2zadcbjg5d9bycy/rhs1mi.dat?dl=0 >> >> >> >> >> From lb2653 at columbia.edu Fri Nov 21 15:12:52 2014 From: lb2653 at columbia.edu (Luc Berger-Vergiat) Date: Fri, 21 Nov 2014 16:12:52 -0500 Subject: [petsc-users] Error while calling a fieldsplit preconditioner containing a mg/gamg preconditioner In-Reply-To: References: <546F608D.2080606@columbi.edu> <546F6E89.8000501@columbi.edu> Message-ID: <546FAAD4.9000906@columbi.edu> Yes good call, I pulled master and got the patch. I don't have errors anymore. Do this mean that when a dm is active, the dm is assumed to create hold the information about the vectors (residual, solution and internal work vectors) for the problem, whereas when the dm is not active or is not used, the ksp is assumed to hold the information about vectors format? Best, Luc On 11/21/2014 12:08 PM, Lawrence Mitchell wrote: > > On 21 Nov 2014, at 16:55, Luc Berger-Vergiat > wrote: > >> Sure, so with gamg (which was my first choice too), I get the following: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: >> [0]PETSC ERROR: Must call DMShellSetGlobalVector() or >> DMShellSetCreateGlobalVector() >> [0]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >> [0]PETSC ERROR: /home/luc/research/feap_repo/ShearBands/parfeap/feap >> on a arch-opt named euler by > > I wonder if this is the same problem described here > https://bitbucket.org/petsc/petsc/pull-request/221 > > Where the ksp contains an inactive dmshell that doesn't know how to > create vectors > > Lawrence -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Nov 21 15:16:10 2014 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 21 Nov 2014 15:16:10 -0600 Subject: [petsc-users] Error while calling a fieldsplit preconditioner containing a mg/gamg preconditioner In-Reply-To: <546FAAD4.9000906@columbi.edu> References: <546F608D.2080606@columbi.edu> <546F6E89.8000501@columbi.edu> <546FAAD4.9000906@columbi.edu> Message-ID: On Fri, Nov 21, 2014 at 3:12 PM, Luc Berger-Vergiat wrote: > Yes good call, I pulled master and got the patch. > I don't have errors anymore. > +1 Lawrence > Do this mean that when a dm is active, the dm is assumed to create hold > the information about the vectors (residual, solution and internal work > vectors) for the problem, whereas when the dm is not active or is not used, > the ksp is assumed to hold the information about vectors format? > Yes, exactly. We wanted a way to have a DM associated with a problem, and yet have the Vec/Mat passed in define it. We were trying to force the DM organization down the throat of users. Thanks, Matt > Best, > Luc > > On 11/21/2014 12:08 PM, Lawrence Mitchell wrote: > > > On 21 Nov 2014, at 16:55, Luc Berger-Vergiat wrote: > > Sure, so with gamg (which was my first choice too), I get the following: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: Must call DMShellSetGlobalVector() or > DMShellSetCreateGlobalVector() > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > [0]PETSC ERROR: /home/luc/research/feap_repo/ShearBands/parfeap/feap on a > arch-opt named euler by > > > I wonder if this is the same problem described here > https://bitbucket.org/petsc/petsc/pull-request/221 > > Where the ksp contains an inactive dmshell that doesn't know how to > create vectors > > Lawrence > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Nov 21 15:38:38 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 21 Nov 2014 15:38:38 -0600 Subject: [petsc-users] the specific preconditioner In-Reply-To: References: Message-ID: > On Nov 21, 2014, at 2:53 PM, paul zhang wrote: > > Hi Barry, > > Thanks for your checking. I know it for the first problem. > For the second, I am not quite sure how to use those flags though. I have compile all of my source code to a binary, say CFDsolver. Then as I use it, I do > > $ mpirun -np 16 ../CFDsolver case_file mpirun -np 16 ../CFDsolver case_file -ksp_type bcgs -pc_type jacobi -ksp_view for example > > What am I supposed to do? > > Thanks, > Paul > > > > > > > > > > > On Fri, Nov 21, 2014 at 12:39 PM, Barry Smith wrote: > > You need to call KSPGetPC(ksp,&pc); before setting the pc > > Note that since you call KSPSetFromOptions(ksp); you can control the KSP's and PC's from the command line and don't need to edit the code and recompile it for each change to the KSP or PC. For example > > -ksp_type bcgs -pc_type jacobi > > Barry > > > On Nov 21, 2014, at 8:18 AM, huaibao zhang wrote: > > > > Hi Jed, > > > > I have a specific question regarding the attached my initialization function. Previously I used KSPSetType(ksp,KSPFGMRES) solver, and I would like to see if there is some improvement using the other choices like KSPBCGS. It seems hoverer, the PC //PCSetType(pc,PCBJACOBI) does not work with me. And I am not quite sure why. > > > > By the way, can you please give some suggestions on my code? I am not quite sure if I am doing the things rightly. > > > > > > Many thanks, > > Paul > > > > > > > > > > void petsc_init(void) { > > > > vector::iterator cit; > > vector::iterator it; > > > > //Create nonlinear solver context > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > > > VecCreateMPI(PETSC_COMM_WORLD,grid[gid].cellCount*nVars,grid[gid].globalCellCount*nVars,&RHS); > > VecSetFromOptions(RHS); > > VecDuplicate(RHS,&deltaX); > > > > > > VecSet(RHS,0.); > > VecSet(deltaX,0.); > > > > vector diagonal_nonzeros, off_diagonal_nonzeros; > > int nextCellCount; > > > > // Calculate space necessary for matrix memory allocation > > for (cit=grid[gid].cell.begin();cit!=grid[gid].cell.end();cit++) { > > nextCellCount=0; > > for (it=(*cit).faces.begin();it!=(*cit).faces.end();it++) { > > if (grid[gid].face[*it].bc==INTERNAL_FACE) { > > nextCellCount++; > > } > > } > > for (int i=0;i > diagonal_nonzeros.push_back( (nextCellCount+1)*nVars); > > off_diagonal_nonzeros.push_back( ((*cit).ghosts.size())*nVars); > > } > > } > > > > MatCreateMPIAIJ( > > PETSC_COMM_WORLD, > > grid[gid].cellCount*nVars, > > grid[gid].cellCount*nVars, > > grid[gid].globalCellCount*nVars, > > grid[gid].globalCellCount*nVars, > > 0,&diagonal_nonzeros[0], > > 0,&off_diagonal_nonzeros[0], > > &impOP); > > > > KSPSetOperators(ksp,impOP,impOP,SAME_NONZERO_PATTERN); > > > > KSPSetTolerances(ksp,rtol,abstol,1.e15,maxits); > > KSPSetType(ksp, KSPBCGS); > > //PCSetType(pc,PCBJACOBI); > > KSPGMRESSetRestart(ksp,30); > > KSPSetFromOptions(ksp); > > > > return; > > } > > > > > -- > Huaibao (Paul) Zhang > Gas Surface Interactions Lab > Department of Mechanical Engineering > University of Kentucky, > Lexington, > KY, 40506-0503 > Office: 216 Ralph G. Anderson Building > Web:gsil.engineering.uky.edu From k.anush at gmail.com Fri Nov 21 17:09:27 2014 From: k.anush at gmail.com (Anush Krishnan) Date: Fri, 21 Nov 2014 18:09:27 -0500 Subject: [petsc-users] Indefinite PC Message-ID: Hello petsc-users, I've been running some CFD simulations, and I ran into one case where the system (which is a coupled system for pressure and body forces) did not converge and the reason was KSP_DIVERGED_INDEFINITE_PC. The options I'm using are -pc_gamg_agg_nsmooths 1 -pc_type gamg -pc_gamg_type agg with a conjugate gradient solver. I saw in example ksp/pc/examples/tutorials/ex2.c that I should use the flag -pc_factor_shift_positive_definite to avoid this. I have a few questions regarding this: 1. What does it mean that the preconditioner is indefinite? What can cause this to happen? And what does the above flag do? 2. The error occurs midway through the simulation. Is there any reason why this might be the case? The left-hand side matrix does not change during the simulation. 3. Do both -pc_factor_shift_positive_definite and -pc_factor_shift_type POSITIVE_DEFINITE do the same thing? Thank you, Anush -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Nov 21 17:23:56 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 21 Nov 2014 17:23:56 -0600 Subject: [petsc-users] Indefinite PC In-Reply-To: References: Message-ID: > On Nov 21, 2014, at 5:09 PM, Anush Krishnan wrote: > > Hello petsc-users, > > I've been running some CFD simulations, and I ran into one case where the system (which is a coupled system for pressure and body forces) did not converge and the reason was KSP_DIVERGED_INDEFINITE_PC. The options I'm using are -pc_gamg_agg_nsmooths 1 -pc_type gamg -pc_gamg_type agg with a conjugate gradient solver. We've heard reports of this happening before. You should increase the number of smoothing steps for the multigrid or change to a "stronger" smoother. What do you get with -ksp_view for exact solver you are using? Are you using PETSc's SNES? TS? Or just KSP? > > I saw in example ksp/pc/examples/tutorials/ex2.c that I should use the flag -pc_factor_shift_positive_definite to avoid this. This flag is only for factorization based preconditioners not gamg so you should not use it. > I have a few questions regarding this: > ? What does it mean that the preconditioner is indefinite? It means that the preconditioner generated by GAMG has both negative and positive eigenvalues. CG cannot handle this, CG requires the preconditioner have all eigenvalues of the same sign. > What can cause this to happen? And what does the above flag do? > ? The error occurs midway through the simulation. Is there any reason why this might be the case? The left-hand side matrix does not change during the simulation. Huh? If the matrix is not changing then the preconditioner should not be changing and hence the preconditioner should not be rebuilt and hence you should not see this message "midway through the simulation". Are you sure that the matrix is not changing?? > ? Do both -pc_factor_shift_positive_definite and -pc_factor_shift_type POSITIVE_DEFINITE do the same thing? Yes, they are from different versions of PETSc, but neither are for gamg. Barry > Thank you, > > Anush > From k.anush at gmail.com Fri Nov 21 23:05:54 2014 From: k.anush at gmail.com (Anush Krishnan) Date: Sat, 22 Nov 2014 00:05:54 -0500 Subject: [petsc-users] Indefinite PC In-Reply-To: References: Message-ID: On 21 November 2014 18:23, Barry Smith wrote: > > > On Nov 21, 2014, at 5:09 PM, Anush Krishnan wrote: > > > > Hello petsc-users, > > > > I've been running some CFD simulations, and I ran into one case where > the system (which is a coupled system for pressure and body forces) did not > converge and the reason was KSP_DIVERGED_INDEFINITE_PC. The options I'm > using are -pc_gamg_agg_nsmooths 1 -pc_type gamg -pc_gamg_type agg with a > conjugate gradient solver. > > We've heard reports of this happening before. You should increase the > number of smoothing steps for the multigrid or change to a "stronger" > smoother. > > What do you get with -ksp_view for exact solver you are using? > I've attached a copy of the output with -ksp_view. > Are you using PETSc's SNES? TS? Or just KSP? > Just KSP. > > > > > I saw in example ksp/pc/examples/tutorials/ex2.c that I should use the > flag -pc_factor_shift_positive_definite to avoid this. > > This flag is only for factorization based preconditioners not gamg so > you should not use it. > > > I have a few questions regarding this: > > ? What does it mean that the preconditioner is indefinite? > > It means that the preconditioner generated by GAMG has both negative > and positive eigenvalues. CG cannot handle this, CG requires the > preconditioner have all eigenvalues of the same sign. > > > What can cause this to happen? And what does the above flag do? > > ? The error occurs midway through the simulation. Is there any > reason why this might be the case? The left-hand side matrix does not > change during the simulation. > > Huh? If the matrix is not changing then the preconditioner should not > be changing and hence the preconditioner should not be rebuilt and hence > you should not see this message "midway through the simulation". Are you > sure that the matrix is not changing?? > Yes, I'm sure that the matrix is not changing. Only the right hand side vector changes every time step. I have heard that CG sometimes converges even if the matrix is not positive definite - do you think that might be happening? Also, is there a way to check if a given matrix is positive definite using PETSc? And is it possible to generate an indefinite preconditioner from a matrix that is symmetric positive definite (I remember seeing a thread about this recently)? > > > > ? Do both -pc_factor_shift_positive_definite and > -pc_factor_shift_type POSITIVE_DEFINITE do the same thing? > > Yes, they are from different versions of PETSc, but neither are for > gamg. > > Barry > > > Thank you, > > > > Anush > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- KSP Object:(sys2_) 32 MPI processes type: cg maximum iterations=20000 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(sys2_) 32 MPI processes type: gamg MG: type is MULTIPLICATIVE, levels=4 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (sys2_mg_coarse_) 32 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (sys2_mg_coarse_) 32 MPI processes type: bjacobi block Jacobi: number of blocks = 32 Local solve is same for all blocks, in the following KSP and PC objects: KSP Object: (sys2_mg_coarse_sub_) 1 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (sys2_mg_coarse_sub_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 using diagonal shift on blocks to prevent zero pivot [INBLOCKS] matrix ordering: nd factor fill ratio given 5, needed 4.07919 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=465, cols=465 package used to perform factorization: petsc total: nonzeros=103387, allocated nonzeros=103387 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=465, cols=465 total: nonzeros=25345, allocated nonzeros=25345 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 32 MPI processes type: mpiaij rows=465, cols=465 total: nonzeros=25345, allocated nonzeros=25345 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (sys2_mg_levels_1_) 32 MPI processes type: chebyshev Chebyshev: eigenvalue estimates: min = 0.0670626, max = 1.40831 maximum iterations=2 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (sys2_mg_levels_1_) 32 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Mat Object: 32 MPI processes type: mpiaij rows=59990, cols=59990 total: nonzeros=3.98622e+06, allocated nonzeros=3.98622e+06 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (sys2_mg_levels_2_) 32 MPI processes type: chebyshev Chebyshev: eigenvalue estimates: min = 0.125337, max = 2.63208 maximum iterations=2 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (sys2_mg_levels_2_) 32 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Mat Object: 32 MPI processes type: mpiaij rows=4204607, cols=4204607 total: nonzeros=1.40085e+08, allocated nonzeros=1.40085e+08 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (sys2_mg_levels_3_) 32 MPI processes type: chebyshev Chebyshev: eigenvalue estimates: min = 0.211442, max = 4.44027 maximum iterations=2 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (sys2_mg_levels_3_) 32 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Mat Object: 32 MPI processes type: mpiaij rows=46923044, cols=46923044 total: nonzeros=3.29933e+08, allocated nonzeros=3.29933e+08 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Mat Object: 32 MPI processes type: mpiaij rows=46923044, cols=46923044 total: nonzeros=3.29933e+08, allocated nonzeros=3.29933e+08 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines From graziosov at libero.it Sat Nov 22 06:02:42 2014 From: graziosov at libero.it (Valerio Grazioso) Date: Sat, 22 Nov 2014 13:02:42 +0100 Subject: [petsc-users] Fortran 77 VecGetArray problems In-Reply-To: <6D5C07E8-4693-4454-81A0-379DAA6B5395@mcs.anl.gov> References: <73DA0A26-8A55-45B3-BDF6-6F56DBC3A50E@libero.it> <6D5C07E8-4693-4454-81A0-379DAA6B5395@mcs.anl.gov> Message-ID: <5D3DC451-D7D8-4031-BF21-F40E4809A037@libero.it> Ehi Barry, thanks a lot? that was the problem: I had the -fbounds-check flag of gfortran in my Makefile?removing it everything is ok ! (Of course ?it was in the manual? !! :) ) Valerio On 21/nov/2014, at 14:18, Barry Smith wrote: > >> On Nov 20, 2014, at 9:19 PM, Valerio Grazioso wrote: >> >> Dear Petsc Users, >> I?m trying to write a simple parallel linear solver (A*x=q) in fortran 77 with Petsc. >> I managed to Setup a MPIAIJ matrix A and a VecMPI qvec >> then the KSP and the solution. >> Everything seems fine (I?ve checked both A and q with MatView and VecView) but I?m having problems getting the solution in a ?normal? local double precision fortran vector. >> >> To be more than sure I?ve tried to repeat the necessary steps in order to achieve this result on qvec : >> >> After creating and calling a VecScatter in order to bring a sequential version on all the processors (as described in >> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#mpi-vec-to-mpi-vec) >> I?ve tried to use VecGetValue as described in the User manual in the Fortran users chapter (pag. 147-148) >> It seems to fail in both the local xx_v vector (that has wrong values) and in calculating the i_x index offset that has a very large negative value >> that, when it is used, generates an obvious ?Aut of bound? error. > > It is completely possible that i_x is a very large negative value, in fact it usually is. > > From the users manual: Note: If using VecGetArray(), MatDenseGetArray(), or ISGetIndices(), from Fortran, the user must not compile the Fortran code with options to check for ?array entries out of bounds? (e.g., on the IBM RS/6000 this is done with the -C compiler option, so never use the -C option with this). > > My guess is that your Fortran compiler is checking for array out of bounds and (of course) finding them since xx_v() is declared as xx_v(1). You need to turn of this array bounds checking when compiling. Or you can use VecGetArrayF90(). > > Barry > > > >> I?ve tried to use VecGetValue also on qvec directly : same result! >> >> There must be something wrong that I?m doing. >> Can anybody point me in the right direction? >> Thanks >> >> Regards >> Valerio Grazioso >> >> >> Here it is the relevant code: >> >> >> >> #define xx_a(ib) xx_v(i_x + (ib)) >> >> ???????.. >> >> Vec xSEQ >> PetscScalar xx_v(1) >> PetscOffset i_x >> >> ????????.. >> ??????.. >> >> >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecCreate" >> call VecCreate(comm,qvec,ierr); CHKERRQ(ierr) >> >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecSetSizes" >> call VecSetSizes(qvec,PETSC_DECIDE,NCELL,ierr); CHKERRQ(ierr) >> >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecSetFromOptions" >> call VecSetFromOptions(qvec,ierr); CHKERRQ(ierr) >> >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecDuplicate" >> call VecDuplicate(qvec,xvec,ierr); CHKERRQ(ierr) >> >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecGetOwnershipRange" >> call VecGetOwnershipRange(qvec,Istart,Iend,ierr); CHKERRQ(ierr) >> >> write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> Istart,Iend (Vet)=", Istart,Iend >> >> do Iproc = 0,(mpi_size-1) >> if (Iproc == mpi_rank) then >> do I= (Istart+1), (Iend) >> C write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> MatSetValues",I,I >> call VecSetValues(qvec,1,I-1,SU(I),INSERT_VALUES,ierr); CHKERRQ(ierr) >> call VecSetValues(xvec,1,I-1,0.E0,INSERT_VALUES,ierr); CHKERRQ(ierr) >> enddo >> endif >> enddo >> >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecAssemblyBegin" >> call VecAssemblyBegin(qvec,ierr); CHKERRQ(ierr) >> call VecAssemblyEnd(qvec,ierr); CHKERRQ(ierr) >> call VecAssemblyBegin(xvec,ierr); CHKERRQ(ierr) >> call VecAssemblyEnd(xvec,ierr); CHKERRQ(ierr) >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecAssemblyEnd" >> >> C --------------------------------- DEBUG ------------------------------------- >> if (1 == 1) then >> >> call VecScatterCreateToAll(qvec,ctx,xSEQ,ierr); CHKERRQ(ierr) >> call VecScatterBegin(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) >> CHKERRQ(ierr) >> call VecScatterEnd(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) >> CHKERRQ(ierr) >> call VecScatterDestroy(ctx,ierr); CHKERRQ(ierr) >> >> C call PetscViewerCreate(comm, view, ierr); CHKERRQ(ierr) >> C call PetscViewerSetType(view, PETSCVIEWERASCII, ierr); CHKERRQ(ierr) >> C call PetscViewerSetFormat(view, PETSC_VIEWER_ASCII_INDEX, ierr) >> C CHKERRQ(ierr) >> C call PetscViewerFileSetName(view, 'qvec.m', ierr) >> C call VecView(xSEQ, view, ierr); CHKERRQ(ierr) >> C call PetscViewerDestroy(view, ierr); CHKERRQ(ierr) >> >> call VecGetArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) >> >> write(6, *) (xx_a(I), I=1,NCELL) >> write(6,'(a,i20)')"i_x = ", i_x >> >> >> call VecRestoreArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) >> >> call PetscFinalize(ierr); CHKERRQ(ierr) >> STOP >> endif >> C ---------------------------------- END DEBUG ------------------------------ >> >> > From graziosov at libero.it Sat Nov 22 06:07:14 2014 From: graziosov at libero.it (Valerio Grazioso) Date: Sat, 22 Nov 2014 13:07:14 +0100 Subject: [petsc-users] Fortran 77 VecGetArray problems In-Reply-To: References: <73DA0A26-8A55-45B3-BDF6-6F56DBC3A50E@libero.it> Message-ID: <999378B8-7169-4EC0-8416-F5FB9C3F4BB6@libero.it> Hi Matt, thanks for the reply. I've solved the problem thanks to Barry Smith?s suggestion? Cheers Valerio On 21/nov/2014, at 12:18, Matthew Knepley wrote: > On Thu, Nov 20, 2014 at 9:19 PM, Valerio Grazioso wrote: > Dear Petsc Users, > I?m trying to write a simple parallel linear solver (A*x=q) in fortran 77 with Petsc. > I managed to Setup a MPIAIJ matrix A and a VecMPI qvec > then the KSP and the solution. > Everything seems fine (I?ve checked both A and q with MatView and VecView) but I?m having problems getting the solution in a ?normal? local double precision fortran vector. > > To be more than sure I?ve tried to repeat the necessary steps in order to achieve this result on qvec : > > After creating and calling a VecScatter in order to bring a sequential version on all the processors (as described in > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#mpi-vec-to-mpi-vec) > I?ve tried to use VecGetValue as described in the User manual in the Fortran users chapter (pag. 147-148) > It seems to fail in both the local xx_v vector (that has wrong values) and in calculating the i_x index offset that has a very large negative value > that, when it is used, generates an obvious ?Aut of bound? error. > I?ve tried to use VecGetValue also on qvec directly : same result! > > There must be something wrong that I?m doing. > Can anybody point me in the right direction? > > The code looks correct to me. As a first step, can you try running an example, to make sure its > not a compiler problem. This example > > http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/examples/tutorials/ex4f.F.html > > should be enough to check. > > Thanks, > > Matt > > Thanks > > Regards > Valerio Grazioso > > > Here it is the relevant code: > > > > #define xx_a(ib) xx_v(i_x + (ib)) > > ???????.. > > Vec xSEQ > PetscScalar xx_v(1) > PetscOffset i_x > > ????????.. > ??????.. > > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecCreate" > call VecCreate(comm,qvec,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecSetSizes" > call VecSetSizes(qvec,PETSC_DECIDE,NCELL,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecSetFromOptions" > call VecSetFromOptions(qvec,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecDuplicate" > call VecDuplicate(qvec,xvec,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecGetOwnershipRange" > call VecGetOwnershipRange(qvec,Istart,Iend,ierr); CHKERRQ(ierr) > > write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> Istart,Iend (Vet)=", Istart,Iend > > do Iproc = 0,(mpi_size-1) > if (Iproc == mpi_rank) then > do I= (Istart+1), (Iend) > C write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> MatSetValues",I,I > call VecSetValues(qvec,1,I-1,SU(I),INSERT_VALUES,ierr); CHKERRQ(ierr) > call VecSetValues(xvec,1,I-1,0.E0,INSERT_VALUES,ierr); CHKERRQ(ierr) > enddo > endif > enddo > > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecAssemblyBegin" > call VecAssemblyBegin(qvec,ierr); CHKERRQ(ierr) > call VecAssemblyEnd(qvec,ierr); CHKERRQ(ierr) > call VecAssemblyBegin(xvec,ierr); CHKERRQ(ierr) > call VecAssemblyEnd(xvec,ierr); CHKERRQ(ierr) > write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecAssemblyEnd" > > C --------------------------------- DEBUG ------------------------------------- > if (1 == 1) then > > call VecScatterCreateToAll(qvec,ctx,xSEQ,ierr); CHKERRQ(ierr) > call VecScatterBegin(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) > CHKERRQ(ierr) > call VecScatterEnd(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) > CHKERRQ(ierr) > call VecScatterDestroy(ctx,ierr); CHKERRQ(ierr) > > C call PetscViewerCreate(comm, view, ierr); CHKERRQ(ierr) > C call PetscViewerSetType(view, PETSCVIEWERASCII, ierr); CHKERRQ(ierr) > C call PetscViewerSetFormat(view, PETSC_VIEWER_ASCII_INDEX, ierr) > C CHKERRQ(ierr) > C call PetscViewerFileSetName(view, 'qvec.m', ierr) > C call VecView(xSEQ, view, ierr); CHKERRQ(ierr) > C call PetscViewerDestroy(view, ierr); CHKERRQ(ierr) > > call VecGetArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) > > write(6, *) (xx_a(I), I=1,NCELL) > write(6,'(a,i20)')"i_x = ", i_x > > > call VecRestoreArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) > > call PetscFinalize(ierr); CHKERRQ(ierr) > STOP > endif > C ---------------------------------- END DEBUG ------------------------------ > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sat Nov 22 09:08:20 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 22 Nov 2014 09:08:20 -0600 Subject: [petsc-users] Fortran 77 VecGetArray problems In-Reply-To: <5D3DC451-D7D8-4031-BF21-F40E4809A037@libero.it> References: <73DA0A26-8A55-45B3-BDF6-6F56DBC3A50E@libero.it> <6D5C07E8-4693-4454-81A0-379DAA6B5395@mcs.anl.gov> <5D3DC451-D7D8-4031-BF21-F40E4809A037@libero.it> Message-ID: Great! BTW: since most compilers (including gfortran) now support F90 - its best to use VecGetArrayF90 - instead of VecGetArray() [eventhough the rest of the code is F77] Satish On Sat, 22 Nov 2014, Valerio Grazioso wrote: > Ehi Barry, > thanks a lot? that was the problem: I had the -fbounds-check flag of gfortran in my Makefile?removing it everything is ok ! > (Of course ?it was in the manual? !! :) ) > > Valerio > > > > > On 21/nov/2014, at 14:18, Barry Smith wrote: > > > > >> On Nov 20, 2014, at 9:19 PM, Valerio Grazioso wrote: > >> > >> Dear Petsc Users, > >> I?m trying to write a simple parallel linear solver (A*x=q) in fortran 77 with Petsc. > >> I managed to Setup a MPIAIJ matrix A and a VecMPI qvec > >> then the KSP and the solution. > >> Everything seems fine (I?ve checked both A and q with MatView and VecView) but I?m having problems getting the solution in a ?normal? local double precision fortran vector. > >> > >> To be more than sure I?ve tried to repeat the necessary steps in order to achieve this result on qvec : > >> > >> After creating and calling a VecScatter in order to bring a sequential version on all the processors (as described in > >> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#mpi-vec-to-mpi-vec) > >> I?ve tried to use VecGetValue as described in the User manual in the Fortran users chapter (pag. 147-148) > >> It seems to fail in both the local xx_v vector (that has wrong values) and in calculating the i_x index offset that has a very large negative value > >> that, when it is used, generates an obvious ?Aut of bound? error. > > > > It is completely possible that i_x is a very large negative value, in fact it usually is. > > > > From the users manual: Note: If using VecGetArray(), MatDenseGetArray(), or ISGetIndices(), from Fortran, the user must not compile the Fortran code with options to check for ?array entries out of bounds? (e.g., on the IBM RS/6000 this is done with the -C compiler option, so never use the -C option with this). > > > > My guess is that your Fortran compiler is checking for array out of bounds and (of course) finding them since xx_v() is declared as xx_v(1). You need to turn of this array bounds checking when compiling. Or you can use VecGetArrayF90(). > > > > Barry > > > > > > > >> I?ve tried to use VecGetValue also on qvec directly : same result! > >> > >> There must be something wrong that I?m doing. > >> Can anybody point me in the right direction? > >> Thanks > >> > >> Regards > >> Valerio Grazioso > >> > >> > >> Here it is the relevant code: > >> > >> > >> > >> #define xx_a(ib) xx_v(i_x + (ib)) > >> > >> ???????.. > >> > >> Vec xSEQ > >> PetscScalar xx_v(1) > >> PetscOffset i_x > >> > >> ????????.. > >> ??????.. > >> > >> > >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecCreate" > >> call VecCreate(comm,qvec,ierr); CHKERRQ(ierr) > >> > >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecSetSizes" > >> call VecSetSizes(qvec,PETSC_DECIDE,NCELL,ierr); CHKERRQ(ierr) > >> > >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecSetFromOptions" > >> call VecSetFromOptions(qvec,ierr); CHKERRQ(ierr) > >> > >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecDuplicate" > >> call VecDuplicate(qvec,xvec,ierr); CHKERRQ(ierr) > >> > >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecGetOwnershipRange" > >> call VecGetOwnershipRange(qvec,Istart,Iend,ierr); CHKERRQ(ierr) > >> > >> write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> Istart,Iend (Vet)=", Istart,Iend > >> > >> do Iproc = 0,(mpi_size-1) > >> if (Iproc == mpi_rank) then > >> do I= (Istart+1), (Iend) > >> C write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> MatSetValues",I,I > >> call VecSetValues(qvec,1,I-1,SU(I),INSERT_VALUES,ierr); CHKERRQ(ierr) > >> call VecSetValues(xvec,1,I-1,0.E0,INSERT_VALUES,ierr); CHKERRQ(ierr) > >> enddo > >> endif > >> enddo > >> > >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecAssemblyBegin" > >> call VecAssemblyBegin(qvec,ierr); CHKERRQ(ierr) > >> call VecAssemblyEnd(qvec,ierr); CHKERRQ(ierr) > >> call VecAssemblyBegin(xvec,ierr); CHKERRQ(ierr) > >> call VecAssemblyEnd(xvec,ierr); CHKERRQ(ierr) > >> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecAssemblyEnd" > >> > >> C --------------------------------- DEBUG ------------------------------------- > >> if (1 == 1) then > >> > >> call VecScatterCreateToAll(qvec,ctx,xSEQ,ierr); CHKERRQ(ierr) > >> call VecScatterBegin(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) > >> CHKERRQ(ierr) > >> call VecScatterEnd(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) > >> CHKERRQ(ierr) > >> call VecScatterDestroy(ctx,ierr); CHKERRQ(ierr) > >> > >> C call PetscViewerCreate(comm, view, ierr); CHKERRQ(ierr) > >> C call PetscViewerSetType(view, PETSCVIEWERASCII, ierr); CHKERRQ(ierr) > >> C call PetscViewerSetFormat(view, PETSC_VIEWER_ASCII_INDEX, ierr) > >> C CHKERRQ(ierr) > >> C call PetscViewerFileSetName(view, 'qvec.m', ierr) > >> C call VecView(xSEQ, view, ierr); CHKERRQ(ierr) > >> C call PetscViewerDestroy(view, ierr); CHKERRQ(ierr) > >> > >> call VecGetArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) > >> > >> write(6, *) (xx_a(I), I=1,NCELL) > >> write(6,'(a,i20)')"i_x = ", i_x > >> > >> > >> call VecRestoreArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) > >> > >> call PetscFinalize(ierr); CHKERRQ(ierr) > >> STOP > >> endif > >> C ---------------------------------- END DEBUG ------------------------------ > >> > >> > > > > From knepley at gmail.com Sat Nov 22 09:43:26 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 22 Nov 2014 09:43:26 -0600 Subject: [petsc-users] Indefinite PC In-Reply-To: References: Message-ID: On Fri, Nov 21, 2014 at 11:05 PM, Anush Krishnan wrote: > > > On 21 November 2014 18:23, Barry Smith wrote: > >> >> > On Nov 21, 2014, at 5:09 PM, Anush Krishnan wrote: >> > >> > Hello petsc-users, >> > >> > I've been running some CFD simulations, and I ran into one case where >> the system (which is a coupled system for pressure and body forces) did not >> converge and the reason was KSP_DIVERGED_INDEFINITE_PC. The options I'm >> using are -pc_gamg_agg_nsmooths 1 -pc_type gamg -pc_gamg_type agg with a >> conjugate gradient solver. >> >> We've heard reports of this happening before. You should increase the >> number of smoothing steps for the multigrid or change to a "stronger" >> smoother. >> >> What do you get with -ksp_view for exact solver you are using? >> > > I've attached a copy of the output with -ksp_view. > > >> Are you using PETSc's SNES? TS? Or just KSP? >> > > Just KSP. > > >> >> > >> > I saw in example ksp/pc/examples/tutorials/ex2.c that I should use the >> flag -pc_factor_shift_positive_definite to avoid this. >> >> This flag is only for factorization based preconditioners not gamg so >> you should not use it. >> >> > I have a few questions regarding this: >> > ? What does it mean that the preconditioner is indefinite? >> >> It means that the preconditioner generated by GAMG has both negative >> and positive eigenvalues. CG cannot handle this, CG requires the >> preconditioner have all eigenvalues of the same sign. >> >> > What can cause this to happen? And what does the above flag do? >> > ? The error occurs midway through the simulation. Is there any >> reason why this might be the case? The left-hand side matrix does not >> change during the simulation. >> >> Huh? If the matrix is not changing then the preconditioner should not >> be changing and hence the preconditioner should not be rebuilt and hence >> you should not see this message "midway through the simulation". Are you >> sure that the matrix is not changing?? >> > > Yes, I'm sure that the matrix is not changing. Only the right hand side > vector changes every time step. > > I have heard that CG sometimes converges even if the matrix is not > positive definite - do you think that might be happening? Also, is there a > way to check if a given matrix is positive definite using PETSc? And is it > possible to > That is possible. It is much more expensive to verify indefiniteness than to solve the system. I recommend you increase the smoothing as Barry suggested. Thanks, Matt > generate an indefinite preconditioner from a matrix that is symmetric > positive definite (I remember seeing a thread about this recently)? > > >> >> >> > ? Do both -pc_factor_shift_positive_definite and >> -pc_factor_shift_type POSITIVE_DEFINITE do the same thing? >> >> Yes, they are from different versions of PETSc, but neither are for >> gamg. >> >> Barry >> >> > Thank you, >> > >> > Anush >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From graziosov at libero.it Sat Nov 22 10:43:21 2014 From: graziosov at libero.it (Valerio Grazioso) Date: Sat, 22 Nov 2014 17:43:21 +0100 Subject: [petsc-users] Fortran 77 VecGetArray problems In-Reply-To: References: <73DA0A26-8A55-45B3-BDF6-6F56DBC3A50E@libero.it> <6D5C07E8-4693-4454-81A0-379DAA6B5395@mcs.anl.gov> <5D3DC451-D7D8-4031-BF21-F40E4809A037@libero.it> Message-ID: Ok, I?ll use VecGetArrayF90. Thanks for the heads up Valerio On 22/nov/2014, at 16:08, Satish Balay wrote: > Great! > > BTW: since most compilers (including gfortran) now support F90 - its > best to use VecGetArrayF90 - instead of VecGetArray() [eventhough the > rest of the code is F77] > > Satish > > On Sat, 22 Nov 2014, Valerio Grazioso wrote: > >> Ehi Barry, >> thanks a lot? that was the problem: I had the -fbounds-check flag of gfortran in my Makefile?removing it everything is ok ! >> (Of course ?it was in the manual? !! :) ) >> >> Valerio >> >> >> >> >> On 21/nov/2014, at 14:18, Barry Smith wrote: >> >>> >>>> On Nov 20, 2014, at 9:19 PM, Valerio Grazioso wrote: >>>> >>>> Dear Petsc Users, >>>> I?m trying to write a simple parallel linear solver (A*x=q) in fortran 77 with Petsc. >>>> I managed to Setup a MPIAIJ matrix A and a VecMPI qvec >>>> then the KSP and the solution. >>>> Everything seems fine (I?ve checked both A and q with MatView and VecView) but I?m having problems getting the solution in a ?normal? local double precision fortran vector. >>>> >>>> To be more than sure I?ve tried to repeat the necessary steps in order to achieve this result on qvec : >>>> >>>> After creating and calling a VecScatter in order to bring a sequential version on all the processors (as described in >>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#mpi-vec-to-mpi-vec) >>>> I?ve tried to use VecGetValue as described in the User manual in the Fortran users chapter (pag. 147-148) >>>> It seems to fail in both the local xx_v vector (that has wrong values) and in calculating the i_x index offset that has a very large negative value >>>> that, when it is used, generates an obvious ?Aut of bound? error. >>> >>> It is completely possible that i_x is a very large negative value, in fact it usually is. >>> >>> From the users manual: Note: If using VecGetArray(), MatDenseGetArray(), or ISGetIndices(), from Fortran, the user must not compile the Fortran code with options to check for ?array entries out of bounds? (e.g., on the IBM RS/6000 this is done with the -C compiler option, so never use the -C option with this). >>> >>> My guess is that your Fortran compiler is checking for array out of bounds and (of course) finding them since xx_v() is declared as xx_v(1). You need to turn of this array bounds checking when compiling. Or you can use VecGetArrayF90(). >>> >>> Barry >>> >>> >>> >>>> I?ve tried to use VecGetValue also on qvec directly : same result! >>>> >>>> There must be something wrong that I?m doing. >>>> Can anybody point me in the right direction? >>>> Thanks >>>> >>>> Regards >>>> Valerio Grazioso >>>> >>>> >>>> Here it is the relevant code: >>>> >>>> >>>> >>>> #define xx_a(ib) xx_v(i_x + (ib)) >>>> >>>> ???????.. >>>> >>>> Vec xSEQ >>>> PetscScalar xx_v(1) >>>> PetscOffset i_x >>>> >>>> ????????.. >>>> ??????.. >>>> >>>> >>>> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecCreate" >>>> call VecCreate(comm,qvec,ierr); CHKERRQ(ierr) >>>> >>>> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecSetSizes" >>>> call VecSetSizes(qvec,PETSC_DECIDE,NCELL,ierr); CHKERRQ(ierr) >>>> >>>> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecSetFromOptions" >>>> call VecSetFromOptions(qvec,ierr); CHKERRQ(ierr) >>>> >>>> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecDuplicate" >>>> call VecDuplicate(qvec,xvec,ierr); CHKERRQ(ierr) >>>> >>>> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecGetOwnershipRange" >>>> call VecGetOwnershipRange(qvec,Istart,Iend,ierr); CHKERRQ(ierr) >>>> >>>> write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> Istart,Iend (Vet)=", Istart,Iend >>>> >>>> do Iproc = 0,(mpi_size-1) >>>> if (Iproc == mpi_rank) then >>>> do I= (Istart+1), (Iend) >>>> C write(6,'(a,i2,a,2i8)'),"Processor ",mpi_rank,": petsc_solve ---> MatSetValues",I,I >>>> call VecSetValues(qvec,1,I-1,SU(I),INSERT_VALUES,ierr); CHKERRQ(ierr) >>>> call VecSetValues(xvec,1,I-1,0.E0,INSERT_VALUES,ierr); CHKERRQ(ierr) >>>> enddo >>>> endif >>>> enddo >>>> >>>> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecAssemblyBegin" >>>> call VecAssemblyBegin(qvec,ierr); CHKERRQ(ierr) >>>> call VecAssemblyEnd(qvec,ierr); CHKERRQ(ierr) >>>> call VecAssemblyBegin(xvec,ierr); CHKERRQ(ierr) >>>> call VecAssemblyEnd(xvec,ierr); CHKERRQ(ierr) >>>> write(6,'(a,i2,a)'),"Processor ",mpi_rank,": petsc_solve ---> VecAssemblyEnd" >>>> >>>> C --------------------------------- DEBUG ------------------------------------- >>>> if (1 == 1) then >>>> >>>> call VecScatterCreateToAll(qvec,ctx,xSEQ,ierr); CHKERRQ(ierr) >>>> call VecScatterBegin(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) >>>> CHKERRQ(ierr) >>>> call VecScatterEnd(ctx,qvec,xSEQ,INSERT_VALUES,SCATTER_FORWARD,ierr) >>>> CHKERRQ(ierr) >>>> call VecScatterDestroy(ctx,ierr); CHKERRQ(ierr) >>>> >>>> C call PetscViewerCreate(comm, view, ierr); CHKERRQ(ierr) >>>> C call PetscViewerSetType(view, PETSCVIEWERASCII, ierr); CHKERRQ(ierr) >>>> C call PetscViewerSetFormat(view, PETSC_VIEWER_ASCII_INDEX, ierr) >>>> C CHKERRQ(ierr) >>>> C call PetscViewerFileSetName(view, 'qvec.m', ierr) >>>> C call VecView(xSEQ, view, ierr); CHKERRQ(ierr) >>>> C call PetscViewerDestroy(view, ierr); CHKERRQ(ierr) >>>> >>>> call VecGetArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) >>>> >>>> write(6, *) (xx_a(I), I=1,NCELL) >>>> write(6,'(a,i20)')"i_x = ", i_x >>>> >>>> >>>> call VecRestoreArray(xSEQ,xx_v,i_x,ierr); CHKERRQ(ierr) >>>> >>>> call PetscFinalize(ierr); CHKERRQ(ierr) >>>> STOP >>>> endif >>>> C ---------------------------------- END DEBUG ------------------------------ >>>> >>>> >>> >> >> From bsmith at mcs.anl.gov Sat Nov 22 11:00:58 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 22 Nov 2014 11:00:58 -0600 Subject: [petsc-users] Indefinite PC In-Reply-To: References: Message-ID: You could do -sys2_mg_levels_ksp_max_it 3 and/or -sys2_mg_coarse_pc_type redundant Or if you configured PETSc with a parallel direct solver like superlu_dist you could use -sys2_mg_coarse_pc_type pc -sys2_mg_coarse_pc_mat_factor_solver_package superlu_dist Or you could keep the current PC gamg options and switch to -ksp_type bcgs Barry > On Nov 21, 2014, at 11:05 PM, Anush Krishnan wrote: > > > > On 21 November 2014 18:23, Barry Smith wrote: > > > On Nov 21, 2014, at 5:09 PM, Anush Krishnan wrote: > > > > Hello petsc-users, > > > > I've been running some CFD simulations, and I ran into one case where the system (which is a coupled system for pressure and body forces) did not converge and the reason was KSP_DIVERGED_INDEFINITE_PC. The options I'm using are -pc_gamg_agg_nsmooths 1 -pc_type gamg -pc_gamg_type agg with a conjugate gradient solver. > > We've heard reports of this happening before. You should increase the number of smoothing steps for the multigrid or change to a "stronger" smoother. > > What do you get with -ksp_view for exact solver you are using? > > I've attached a copy of the output with -ksp_view. > > > Are you using PETSc's SNES? TS? Or just KSP? > > Just KSP. > > > > > > I saw in example ksp/pc/examples/tutorials/ex2.c that I should use the flag -pc_factor_shift_positive_definite to avoid this. > > This flag is only for factorization based preconditioners not gamg so you should not use it. > > > I have a few questions regarding this: > > ? What does it mean that the preconditioner is indefinite? > > It means that the preconditioner generated by GAMG has both negative and positive eigenvalues. CG cannot handle this, CG requires the preconditioner have all eigenvalues of the same sign. > > > What can cause this to happen? And what does the above flag do? > > ? The error occurs midway through the simulation. Is there any reason why this might be the case? The left-hand side matrix does not change during the simulation. > > Huh? If the matrix is not changing then the preconditioner should not be changing and hence the preconditioner should not be rebuilt and hence you should not see this message "midway through the simulation". Are you sure that the matrix is not changing?? > > Yes, I'm sure that the matrix is not changing. Only the right hand side vector changes every time step. > > I have heard that CG sometimes converges even if the matrix is not positive definite - do you think that might be happening? Also, is there a way to check if a given matrix is positive definite using PETSc? And is it possible to generate an indefinite preconditioner from a matrix that is symmetric positive definite (I remember seeing a thread about this recently)? > > > > > ? Do both -pc_factor_shift_positive_definite and -pc_factor_shift_type POSITIVE_DEFINITE do the same thing? > > Yes, they are from different versions of PETSc, but neither are for gamg. > > Barry > > > Thank you, > > > > Anush > > > > > From k.anush at gmail.com Sat Nov 22 11:25:13 2014 From: k.anush at gmail.com (Anush Krishnan) Date: Sat, 22 Nov 2014 12:25:13 -0500 Subject: [petsc-users] Indefinite PC In-Reply-To: References: Message-ID: On 22 November 2014 at 12:00, Barry Smith wrote: > > You could do -sys2_mg_levels_ksp_max_it 3 and/or > -sys2_mg_coarse_pc_type redundant > > Or if you configured PETSc with a parallel direct solver like > superlu_dist you could use -sys2_mg_coarse_pc_type pc > -sys2_mg_coarse_pc_mat_factor_solver_package superlu_dist > > > > Or you could keep the current PC gamg options and switch to -ksp_type > bcgs > Thank you. I'll try those options and get back. > > Barry > > > > > On Nov 21, 2014, at 11:05 PM, Anush Krishnan wrote: > > > > > > > > On 21 November 2014 18:23, Barry Smith wrote: > > > > > On Nov 21, 2014, at 5:09 PM, Anush Krishnan wrote: > > > > > > Hello petsc-users, > > > > > > I've been running some CFD simulations, and I ran into one case where > the system (which is a coupled system for pressure and body forces) did not > converge and the reason was KSP_DIVERGED_INDEFINITE_PC. The options I'm > using are -pc_gamg_agg_nsmooths 1 -pc_type gamg -pc_gamg_type agg with a > conjugate gradient solver. > > > > We've heard reports of this happening before. You should increase the > number of smoothing steps for the multigrid or change to a "stronger" > smoother. > > > > What do you get with -ksp_view for exact solver you are using? > > > > I've attached a copy of the output with -ksp_view. > > > > > > Are you using PETSc's SNES? TS? Or just KSP? > > > > Just KSP. > > > > > > > > > > I saw in example ksp/pc/examples/tutorials/ex2.c that I should use the > flag -pc_factor_shift_positive_definite to avoid this. > > > > This flag is only for factorization based preconditioners not gamg so > you should not use it. > > > > > I have a few questions regarding this: > > > ? What does it mean that the preconditioner is indefinite? > > > > It means that the preconditioner generated by GAMG has both negative > and positive eigenvalues. CG cannot handle this, CG requires the > preconditioner have all eigenvalues of the same sign. > > > > > What can cause this to happen? And what does the above flag do? > > > ? The error occurs midway through the simulation. Is there any > reason why this might be the case? The left-hand side matrix does not > change during the simulation. > > > > Huh? If the matrix is not changing then the preconditioner should > not be changing and hence the preconditioner should not be rebuilt and > hence you should not see this message "midway through the simulation". Are > you sure that the matrix is not changing?? > > > > Yes, I'm sure that the matrix is not changing. Only the right hand side > vector changes every time step. > > > > I have heard that CG sometimes converges even if the matrix is not > positive definite - do you think that might be happening? Also, is there a > way to check if a given matrix is positive definite using PETSc? And is it > possible to generate an indefinite preconditioner from a matrix that is > symmetric positive definite (I remember seeing a thread about this > recently)? > > > > > > > > > ? Do both -pc_factor_shift_positive_definite and > -pc_factor_shift_type POSITIVE_DEFINITE do the same thing? > > > > Yes, they are from different versions of PETSc, but neither are for > gamg. > > > > Barry > > > > > Thank you, > > > > > > Anush > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alpkalpalp at gmail.com Sat Nov 22 16:30:26 2014 From: alpkalpalp at gmail.com (Alp Kalpalp) Date: Sun, 23 Nov 2014 00:30:26 +0200 Subject: [petsc-users] AIJ asembly from AIJs Message-ID: Hi, I am a newby in PetSc but I could not find a solution to my problem. Let me describbe it to you.I have an SeqAIJ matrix on each processor. These substructures stiffness matrices should be assembled on a MPIAIJ and they have overlapping dofs (on boundaries). A is Eigen::SparseMatrix and my code was as follows; ierr = MatCreateSeqAIJWithArrays(PETSC_COMM_SELF,m,n,A.outerIndexPtr(),A.innerIndexPtr(),A.valuePtr(),&subK);CHKERRQ(ierr); ierr = MatCreateMPIAIJSumSeqAIJ(PETSC_COMM_WORLD,subK,PETSC_DECIDE,PETSC_DECIDE,MAT_INITIAL_MATRIX,&K);CHKERRQ(ierr); Suppose I have 2 procs. first have 18x18 sparse mat, second hase 12x12 sparse matrix. Assembled system should be 24x24 for this specific problem. Above code is wenting to infinite loops. debugger reports that there is a deadlock on the progman. So, I searched for a workaroud and I found: MatCreateMPIAIJWithArrays and this function seems fails when the matrices has overlapping parts. So that it asembles 30x30 matrix'!! I check MatCreateMPIAIJ and it seems so costly to form d_nnz etc. Finally, I tried the MatCreate, MatSetValues procedure, however in here MatSetValues requires a coordinate list of values not CSR format arrays!!! So my question is how may I set and MPIAIJ from my (distributed) overlapping SeqAIJ matrices or preferebly using my CSR arrays directly? Thanks in advance -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Nov 22 17:17:26 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 22 Nov 2014 17:17:26 -0600 Subject: [petsc-users] AIJ asembly from AIJs In-Reply-To: References: Message-ID: On Sat, Nov 22, 2014 at 4:30 PM, Alp Kalpalp wrote: > Hi, > > I am a newby in PetSc but I could not find a solution to my problem. Let > me describbe it to you.I have an SeqAIJ matrix on each processor. These > substructures stiffness matrices should be assembled on a MPIAIJ and they > have overlapping dofs (on boundaries). A is Eigen::SparseMatrix and > my code was as follows; > > ierr = > MatCreateSeqAIJWithArrays(PETSC_COMM_SELF,m,n,A.outerIndexPtr(),A.innerIndexPtr(),A.valuePtr(),&subK);CHKERRQ(ierr); > ierr = > MatCreateMPIAIJSumSeqAIJ(PETSC_COMM_WORLD,subK,PETSC_DECIDE,PETSC_DECIDE,MAT_INITIAL_MATRIX,&K);CHKERRQ(ierr); > > Suppose I have 2 procs. first have 18x18 sparse mat, second hase 12x12 > sparse matrix. Assembled system should be 24x24 for this specific problem. > Above code is wenting to infinite loops. debugger reports that there is a > deadlock on the progman. > This function does not do this job: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJSumSeqAIJ.html It clearly says "The dimensions of the sequential matrix in each processor MUST be the same." So, I searched for a workaroud and I found: > > MatCreateMPIAIJWithArrays and this function seems fails when the matrices > has overlapping parts. So that it asembles 30x30 matrix'!! > > I check MatCreateMPIAIJ and it seems so costly to form d_nnz etc. > > Finally, I tried the MatCreate, MatSetValues procedure, however in here > MatSetValues requires a coordinate list of values not CSR format arrays!!! > > So my question is how may I set and MPIAIJ from my (distributed) > overlapping SeqAIJ matrices or preferebly using my CSR arrays directly? > 1) I think the best option is to put your element matrices directly into the MPIAIJ matrix, since each elemMat can be set with a single call to MatSetValues(). 2) If that is too much reorganization, then I would a) Use MatCreateMPIAIJWithArrays() for the non-overlapping rows (remove the ghost rows) b) Call MatSetValues() on each ghost row 3) If you cannot remove the overlapping rows, I would just call MatSetValues() on each row. It is unlikely that the insertion process is a bottleneck in your computation (unless you are not preallocating correctly) Thanks, Matt > Thanks in advance > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Sat Nov 22 17:39:23 2014 From: dave.mayhem23 at gmail.com (Dave May) Date: Sun, 23 Nov 2014 00:39:23 +0100 Subject: [petsc-users] AIJ asembly from AIJs In-Reply-To: References: Message-ID: An alternative would be this. Assuming you can map each sub-domain row,col index into a global index for your coupled problem, you could You create your matrix of type MATMPIAIJ Then call MatSeqAIJGetArray() MatGetRowIJ() Then for each row of your sub-domain matrix, 1) convert your local row index into a global row index 2) traverse through the CSR array and copy the local column indices into a temporary array 3) convert the local indices in the temporary array into global column indices 4) Call MatSetValues() (with ADD_VALUES) and add the entries from each sub-domain into the parallel MPIAIJ matrix for the current row. You won't need to copy the aij entries, just index pointer returned by MatSeqAIJGetArray() so that it is points to the beginning of the entries for the current row. Call the appropriate restore functions MatRestoreRowIJ() MatSeqAIJRestoreArray() PETSc team, seems the webpages http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSeqAIJGetArray.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSeqAIJRestoreArray.html both contain a typo. The input matrix should be of type MATSEQAIJ Cheers, Dave On 23 November 2014 at 00:17, Matthew Knepley wrote: > On Sat, Nov 22, 2014 at 4:30 PM, Alp Kalpalp wrote: > >> Hi, >> >> I am a newby in PetSc but I could not find a solution to my problem. Let >> me describbe it to you.I have an SeqAIJ matrix on each processor. These >> substructures stiffness matrices should be assembled on a MPIAIJ and they >> have overlapping dofs (on boundaries). A is Eigen::SparseMatrix and >> my code was as follows; >> >> ierr = >> MatCreateSeqAIJWithArrays(PETSC_COMM_SELF,m,n,A.outerIndexPtr(),A.innerIndexPtr(),A.valuePtr(),&subK);CHKERRQ(ierr); >> ierr = >> MatCreateMPIAIJSumSeqAIJ(PETSC_COMM_WORLD,subK,PETSC_DECIDE,PETSC_DECIDE,MAT_INITIAL_MATRIX,&K);CHKERRQ(ierr); >> >> Suppose I have 2 procs. first have 18x18 sparse mat, second hase 12x12 >> sparse matrix. Assembled system should be 24x24 for this specific problem. >> Above code is wenting to infinite loops. debugger reports that there is a >> deadlock on the progman. >> > > This function does not do this job: > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJSumSeqAIJ.html > > It clearly says "The dimensions of the sequential matrix in each processor > MUST be the same." > > So, I searched for a workaroud and I found: >> >> MatCreateMPIAIJWithArrays and this function seems fails when the matrices >> has overlapping parts. So that it asembles 30x30 matrix'!! >> >> I check MatCreateMPIAIJ and it seems so costly to form d_nnz etc. >> >> Finally, I tried the MatCreate, MatSetValues procedure, however in here >> MatSetValues requires a coordinate list of values not CSR format arrays!!! >> >> So my question is how may I set and MPIAIJ from my (distributed) >> overlapping SeqAIJ matrices or preferebly using my CSR arrays directly? >> > > 1) I think the best option is to put your element matrices directly into > the MPIAIJ matrix, since each elemMat can be set with > a single call to MatSetValues(). > > 2) If that is too much reorganization, then I would > > a) Use MatCreateMPIAIJWithArrays() for the non-overlapping rows (remove > the ghost rows) > > b) Call MatSetValues() on each ghost row > > 3) If you cannot remove the overlapping rows, I would just call > MatSetValues() on each row. It is unlikely that > the insertion process is a bottleneck in your computation (unless you > are not preallocating correctly) > > Thanks, > > Matt > > >> Thanks in advance >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Sat Nov 22 18:54:17 2014 From: hus003 at ucsd.edu (Sun, Hui) Date: Sun, 23 Nov 2014 00:54:17 +0000 Subject: [petsc-users] matrix memory preallocation Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010B53F1@XMAIL-MBX-BH1.AD.UCSD.EDU> Dear Petsc users, I have a DM object by claiming DMDACreate3d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_BOX,32,32,32,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,3,5,0,0,0,&da); There is a problem with it, because it uses DMDA_STENCIL_BOX, and the stencil width is 5, quite big. However, in fact, I don't need such big stencils everywhere, I only need it at a sparse number of locations. I'm wondering how to allocate memory only in those sparse number of locations? Thank you very much! Best, Hui -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Nov 22 20:05:07 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 22 Nov 2014 20:05:07 -0600 Subject: [petsc-users] matrix memory preallocation In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010B53F1@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010B53F1@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: On Sat, Nov 22, 2014 at 6:54 PM, Sun, Hui wrote: > Dear Petsc users, > > I have a DM object by claiming > > DMDACreate3d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_BOX, > 32,32,32,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,3,5,0,0,0,&da); > > There is a problem with it, because it uses DMDA_STENCIL_BOX, and the > stencil width is 5, quite big. > > However, in fact, I don't need such big stencils everywhere, I only need > it at a sparse number of locations. I'm wondering how to allocate memory > only in those sparse number of locations? > DMDA is not flexible enough to support that. You could use a VecScatter to support that communication, but it would be complex to construct. Thanks, Matt > Thank you very much! > > Best, > Hui > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.antonio.magri at gmail.com Sun Nov 23 11:52:49 2014 From: victor.antonio.magri at gmail.com (Victor Magri) Date: Sun, 23 Nov 2014 15:52:49 -0200 Subject: [petsc-users] Why does CHOLMOD solution time differs between PETSc and MATLAB? In-Reply-To: References: <98D4398D-7858-4CDA-A534-7A38F3E5BB1B@anl.gov> Message-ID: On Fri, Nov 21, 2014 at 4:49 PM, Barry Smith wrote: > Here is my conjecture (more than a guess, but less than a full statement > of fact). CHOLMOD uses a "supernodel" algorithm which means that it works > on dense blocks of the factored matrix and can take advantage of BLAS 2 and > 3 operations (unlike almost everything in PETSc). I though that PETSc would take more advantage from BLAS 2. 1) the MKL BLAS/LAPACK are likely much faster than the Fortran versions Your conjecture was right! I compiled PETSc to use mkl.so from my Matlab folder and run the tests again. The computational time was pretty the same in comparison with MATLAB (See attached). 2) it is possible the MKL BLAS/LAPACK used by MATLAB use multiple threads > to take advantage of multiple cores on the hardware. At least in my computer, no. I opened the system monitor to see if MATLAB would use more than one thread and it didn't. I think CHOLMOD, used by MATLAB backslash operator, uses AMD (approximate > min. degree) ordering scheme by default Yes, I searched about that and it really does. For PETSc + MATLAB, use the option -pc_factor_mat_ordering_type qmd The computation time got bigger! I was thinking that reordering would make it lower. Maybe it doesn't because this matrix comes from a 5-point discretization stencil... I tried also RCM and ND but the best time came from natural. Thank you so much for helping, Barry and Abhyankar. On Fri, Nov 21, 2014 at 6:34 PM, Barry Smith wrote: > > > On Nov 21, 2014, at 1:46 PM, Abhyankar, Shrirang G. > wrote: > > > > I think this may have to do with the matrix reodering scheme. I think > CHOLMOD, used by MATLAB backslash operator, uses AMD (approximate min. > degree) ordering scheme by default. PETSc's default reordering scheme is > nested dissection. For PETSc + MATLAB, use the option > -pc_factor_mat_ordering_type qmd > > This is possible and definitely worth trying with different orders, but > I checked the nonzeros in the matrix and the L factor for both approaches > and they are very similar > > PETSc call: > Common.fl 1.81877e+10 (flop count from most recent > analysis) > Common.lnz 4.46748e+07 (fundamental nz in L) > > MatLab call: > flop count: 1.8188e+10 > nnz(L): 4.4675e+07 > > Unless I am misunderstanding something both factors likely look very > similar. > > Barry > > > > > > Shri > > > > On Nov 21, 2014, at 11:23 AM, "Victor Magri" < > victor.antonio.magri at gmail.com> wrote: > > > >> Hi, > >> > >> I'm solving a linear system which comes from the discretization of an > elliptic equation. The mesh has 1 million control volumes and there are two > more constraints for the system. > >> > >> First, I've solved it in MATLAB using backslash command, which in turn > calls CHOLMOD once the matrix is SPD. Using tic and toc funtions, I could > see that it takes about 2s for the system to be solved (See attached for > more details). > >> > >> I wrote both matrix and rhs in PETSc binary format and solved the > linear system using the example problem defined in > ksp/examples/tutorials/ex10.c. The command line options were: > >> > >> mpiexec -n -1 ./ex10 -f A1mi.dat -rhs rhs1mi.dat -log_summary > -ksp_monitos -ksp_view -ksp_type preonly -pc_type cholesky > -pc_factor_mat_solver_package cholmod > >> > >> The configuration parameters for CHOLMOD in PETSc and MATLAB seems the > same, so I was expecting a similar computation time, but it took about 13s > to solve in PETSc. (See attached for more details). > >> > >> Why does CHOLMOD solution time differs between them? > >> > >> If someone wants to try for yourself, here are the binaries for the > matrix and the rhs, respectively. > >> https://www.dropbox.com/s/q9s6mlrmv1qdxon/A1mi.dat?dl=0 > >> https://www.dropbox.com/s/2zadcbjg5d9bycy/rhs1mi.dat?dl=0 > >> > >> > >> > >> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: rcm.dat Type: application/octet-stream Size: 15590 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: qmd.dat Type: application/octet-stream Size: 15590 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: nd.dat Type: application/octet-stream Size: 15588 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: natural.dat Type: application/octet-stream Size: 15598 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Nov 23 12:12:03 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 23 Nov 2014 12:12:03 -0600 Subject: [petsc-users] Why does CHOLMOD solution time differs between PETSc and MATLAB? In-Reply-To: References: <98D4398D-7858-4CDA-A534-7A38F3E5BB1B@anl.gov> Message-ID: <30CD8E75-9944-435F-B441-BCF753978685@mcs.anl.gov> > On Nov 23, 2014, at 11:52 AM, Victor Magri wrote: > > On Fri, Nov 21, 2014 at 4:49 PM, Barry Smith wrote: > Here is my conjecture (more than a guess, but less than a full statement of fact). CHOLMOD uses a "supernodel" algorithm which means that it works on dense blocks of the factored matrix and can take advantage of BLAS 2 and 3 operations (unlike almost everything in PETSc). > > I though that PETSc would take more advantage from BLAS 2. Generally not because BLAS 2 is dense matrix times vector while PETSc usually doesn't use dense matrices > > 1) the MKL BLAS/LAPACK are likely much faster than the Fortran versions > > Your conjecture was right! I compiled PETSc to use mkl.so from my Matlab folder and run the tests again. The computational time was pretty the same in comparison with MATLAB (See attached). Great. Performance for the various orderings can be slightly "counter" intuitive if all you count is flops. Here is the info for the default ordering Common.fl 1.81877e+10 (flop count from most recent analysis) Common.lnz 4.46748e+07 (fundamental nz in L) MatSolve 1 1.0 1.3265e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 6 0 0 0 0 7 0 0 0 0 0 MatCholFctrSym 1 1.0 6.5606e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 32 0 0 0 0 34 0 0 0 0 0 MatCholFctrNum 1 1.0 1.1193e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 54 0 0 0 0 57 0 0 0 0 0 MatGetOrdering 1 1.0 3.8588e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 Here is the info for nested dissection ordering Common.fl 1.09806e+10 (flop count from most recent analysis) Common.lnz 3.31035e+07 (fundamental nz in L) MatSolve 1 1.0 1.1116e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 0 MatCholFctrSym 1 1.0 6.3994e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 27 0 0 0 0 29 0 0 0 0 0 MatCholFctrNum 1 1.0 8.5032e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 36 0 0 0 0 38 0 0 0 0 0 MatGetOrdering 1 1.0 5.2249e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 22 0 0 0 0 23 0 0 0 0 0 Note that nested dissection requires 1.09806e+10/1.81877e+10 = 60% of the flops needed but the total time for ordering, factoring and solving is higher since the ordering takes .522 seconds. Note that the solve time with nested dissection 1.1116e-01/1.3265e-01 = 84% so if one has many solvers (but only one ordering and factorization) then nested dissection would be faster. Did you run with the default PETSc Cholesky? I'd be interested in seeing how long that takes. Barry > > 2) it is possible the MKL BLAS/LAPACK used by MATLAB use multiple threads to take advantage of multiple cores on the hardware. > > At least in my computer, no. I opened the system monitor to see if MATLAB would use more than one thread and it didn't. > > I think CHOLMOD, used by MATLAB backslash operator, uses AMD (approximate min. degree) ordering scheme by default > > Yes, I searched about that and it really does. > > For PETSc + MATLAB, use the option -pc_factor_mat_ordering_type qmd > > The computation time got bigger! I was thinking that reordering would make it lower. Maybe it doesn't because this matrix comes from a 5-point discretization stencil... I tried also RCM and ND but the best time came from natural. > > Thank you so much for helping, Barry and Abhyankar. > > > On Fri, Nov 21, 2014 at 6:34 PM, Barry Smith wrote: > > > On Nov 21, 2014, at 1:46 PM, Abhyankar, Shrirang G. wrote: > > > > I think this may have to do with the matrix reodering scheme. I think CHOLMOD, used by MATLAB backslash operator, uses AMD (approximate min. degree) ordering scheme by default. PETSc's default reordering scheme is nested dissection. For PETSc + MATLAB, use the option -pc_factor_mat_ordering_type qmd > > This is possible and definitely worth trying with different orders, but I checked the nonzeros in the matrix and the L factor for both approaches and they are very similar > > PETSc call: > Common.fl 1.81877e+10 (flop count from most recent analysis) > Common.lnz 4.46748e+07 (fundamental nz in L) > > MatLab call: > flop count: 1.8188e+10 > nnz(L): 4.4675e+07 > > Unless I am misunderstanding something both factors likely look very similar. > > Barry > > > > > > Shri > > > > On Nov 21, 2014, at 11:23 AM, "Victor Magri" wrote: > > > >> Hi, > >> > >> I'm solving a linear system which comes from the discretization of an elliptic equation. The mesh has 1 million control volumes and there are two more constraints for the system. > >> > >> First, I've solved it in MATLAB using backslash command, which in turn calls CHOLMOD once the matrix is SPD. Using tic and toc funtions, I could see that it takes about 2s for the system to be solved (See attached for more details). > >> > >> I wrote both matrix and rhs in PETSc binary format and solved the linear system using the example problem defined in ksp/examples/tutorials/ex10.c. The command line options were: > >> > >> mpiexec -n -1 ./ex10 -f A1mi.dat -rhs rhs1mi.dat -log_summary -ksp_monitos -ksp_view -ksp_type preonly -pc_type cholesky -pc_factor_mat_solver_package cholmod > >> > >> The configuration parameters for CHOLMOD in PETSc and MATLAB seems the same, so I was expecting a similar computation time, but it took about 13s to solve in PETSc. (See attached for more details). > >> > >> Why does CHOLMOD solution time differs between them? > >> > >> If someone wants to try for yourself, here are the binaries for the matrix and the rhs, respectively. > >> https://www.dropbox.com/s/q9s6mlrmv1qdxon/A1mi.dat?dl=0 > >> https://www.dropbox.com/s/2zadcbjg5d9bycy/rhs1mi.dat?dl=0 > >> > >> > >> > >> > >> > > > From mwelland at anl.gov Sun Nov 23 17:09:18 2014 From: mwelland at anl.gov (Welland, Michael J.) Date: Sun, 23 Nov 2014 23:09:18 +0000 Subject: [petsc-users] PCFieldSplit field ksp diverging - kill outer ksp Message-ID: <3C81A68BC8231541A3C3188449130F9A69A2F29E@BUTKUS.anl.gov> I'm using PCFieldSplit with 2 fields, and sometimes the ksp solver for one of those fields diverges. Is it possible to have it immediately kill the outer ksp solver when this happens? Thanks Mike From jed at jedbrown.org Sun Nov 23 18:36:38 2014 From: jed at jedbrown.org (Jed Brown) Date: Sun, 23 Nov 2014 17:36:38 -0700 Subject: [petsc-users] PCFieldSplit field ksp diverging - kill outer ksp In-Reply-To: <3C81A68BC8231541A3C3188449130F9A69A2F29E@BUTKUS.anl.gov> References: <3C81A68BC8231541A3C3188449130F9A69A2F29E@BUTKUS.anl.gov> Message-ID: <87r3wt8meh.fsf@jedbrown.org> "Welland, Michael J." writes: > I'm using PCFieldSplit with 2 fields, and sometimes the ksp solver for > one of those fields diverges. Is it possible to have it immediately > kill the outer ksp solver when this happens? I can't think of a nice way. You could set a convergence test for the outer that checks the converged reason on the inner solvers (PCFieldSplitGetSubKSP, KSPGetConvergedReason). This is actually a recurring issue across hierarchical preconditioners. In many cases, the hierarchical preconditioner is attempting to add robustness or leverage inexact component solves, in which case inner solves don't "converge". It would not be hard to add a crude check in PCFieldSplit, but fine-grained control over which component solver is expected to converge seems like it would add significant complexity. But I see the value, so we should figure something out. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From knepley at gmail.com Sun Nov 23 18:49:48 2014 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 23 Nov 2014 18:49:48 -0600 Subject: [petsc-users] [petsc-dev] PCFieldSplit field ksp diverging - kill outer ksp In-Reply-To: <87r3wt8meh.fsf@jedbrown.org> References: <3C81A68BC8231541A3C3188449130F9A69A2F29E@BUTKUS.anl.gov> <87r3wt8meh.fsf@jedbrown.org> Message-ID: On Sun, Nov 23, 2014 at 6:36 PM, Jed Brown wrote: > "Welland, Michael J." writes: > > > I'm using PCFieldSplit with 2 fields, and sometimes the ksp solver for > > one of those fields diverges. Is it possible to have it immediately > > kill the outer ksp solver when this happens? > > I can't think of a nice way. You could set a convergence test for the > outer that checks the converged reason on the inner solvers > (PCFieldSplitGetSubKSP, KSPGetConvergedReason). > > This is actually a recurring issue across hierarchical preconditioners. > In many cases, the hierarchical preconditioner is attempting to add > robustness or leverage inexact component solves, in which case inner > solves don't "converge". It would not be hard to add a crude check in > PCFieldSplit, but fine-grained control over which component solver is > expected to converge seems like it would add significant complexity. > But I see the value, so we should figure something out. > This is also a problem when using nonlinear preconditioners, since the SNES rarely converges. We have a nice command line way to tell Newton to ignore KSP failures, so maybe we need something like that for nested KSPs. MAtt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetank at gmail.com Sun Nov 23 19:54:20 2014 From: gaetank at gmail.com (Gaetan Kenway) Date: Sun, 23 Nov 2014 20:54:20 -0500 Subject: [petsc-users] Convergence of transposed linear system. Message-ID: Hi everyone I have a question relating to preconditioning effectiveness on large transposed systems. The linear system I'm trying to solve is jacobian matrix of 3D RANS CFD solver. The bock matrix consists of about 3 million block rows with a block size of 6: 5 for the inviscid part and 1 for the SA turbulence model. The preconditioning matrix is different from the linear system matrix in two ways: It uses a first order discretization (instead of second order) and the viscous fluxes are dropped. The untransposed system converges about 6 orders of magnitude with GRMES(100), ASM (overlap 1) and ILU(1) with RCM reordering. The test is run on 128 processors. There are no convergence difficulties. However, when I try to solve the transpose of the same system, by either calling KSPSolveTranspose() or by assembling the transpose of the linear system and its preconditioner and calling KSPSolve(), GMRES stagnates after a negligible drop in the residual and no further progress is made. I have successfully solved this transpose system by using a different preconditioner that includes the complete linearization of the viscous terms (~4 times as many non-zeros in PC matrix) and a much much stronger preconditioner (ASM(2), ILU(2) with 200 GMRES reset. My question is why does the solution of the transpose system with the same method perform so terribly? Is it normal that vastly stronger preconditioning method is required to solve transpose systems? Any suggestions would be greatly appreciated Gaetan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Nov 24 01:15:49 2014 From: jed at jedbrown.org (Jed Brown) Date: Mon, 24 Nov 2014 00:15:49 -0700 Subject: [petsc-users] AIJ asembly from AIJs In-Reply-To: References: Message-ID: <87zjbh6pcq.fsf@jedbrown.org> Dave May writes: > PETSc team, seems the webpages > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSeqAIJGetArray.html > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSeqAIJRestoreArray.html > both contain a typo. The input matrix should be of type MATSEQAIJ Thanks, Dave. This was fixed when you reported it a couple months ago, but the "petsc-current" website docs only gets updated when a new patch release is made. It didn't make 3.5.2, but will be in 3.5.3. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Mon Nov 24 04:55:50 2014 From: jed at jedbrown.org (Jed Brown) Date: Mon, 24 Nov 2014 03:55:50 -0700 Subject: [petsc-users] Indefinite PC In-Reply-To: References: Message-ID: <87y4r07tqh.fsf@jedbrown.org> Anush Krishnan writes: > And is it possible to generate an indefinite preconditioner from a > matrix that is symmetric positive definite (I remember seeing a thread > about this recently)? See, for example, Kershaw's matrix in src/ksp/pc/examples/tutorials/ex1.c. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From fpacull at hotmail.com Mon Nov 24 05:24:46 2014 From: fpacull at hotmail.com (francois Pacull) Date: Mon, 24 Nov 2014 06:24:46 -0500 Subject: [petsc-users] Convergence of transposed linear system. In-Reply-To: References: Message-ID: Hello, This is just an idea but this might be due to the fact that the structure of the preconditioner is severely unsymmetrical when using a first-order upwind scheme without viscous terms: when building the overlap, the non-zero terms in the row-wise extra-diagonal blocks yield the list of vertices to add to each subdomain. If you use the transpose of the preconditioner, it still uses the row-wise and not the column-wise extra-diagonal blocks. So maybe you should build the ASM(1) preconditioner with the untransposed matrix first, and then transpose the preconditioning matrix? You may also change the side of the preconditioner, for the transposed system. Francois. Date: Sun, 23 Nov 2014 20:54:20 -0500 From: gaetank at gmail.com To: petsc-users at mcs.anl.gov Subject: [petsc-users] Convergence of transposed linear system. Hi everyone I have a question relating to preconditioning effectiveness on large transposed systems. The linear system I'm trying to solve is jacobian matrix of 3D RANS CFD solver. The bock matrix consists of about 3 million block rows with a block size of 6: 5 for the inviscid part and 1 for the SA turbulence model. The preconditioning matrix is different from the linear system matrix in two ways: It uses a first order discretization (instead of second order) and the viscous fluxes are dropped. The untransposed system converges about 6 orders of magnitude with GRMES(100), ASM (overlap 1) and ILU(1) with RCM reordering. The test is run on 128 processors. There are no convergence difficulties. However, when I try to solve the transpose of the same system, by either calling KSPSolveTranspose() or by assembling the transpose of the linear system and its preconditioner and calling KSPSolve(), GMRES stagnates after a negligible drop in the residual and no further progress is made. I have successfully solved this transpose system by using a different preconditioner that includes the complete linearization of the viscous terms (~4 times as many non-zeros in PC matrix) and a much much stronger preconditioner (ASM(2), ILU(2) with 200 GMRES reset. My question is why does the solution of the transpose system with the same method perform so terribly? Is it normal that vastly stronger preconditioning method is required to solve transpose systems? Any suggestions would be greatly appreciated Gaetan -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetank at gmail.com Mon Nov 24 07:03:17 2014 From: gaetank at gmail.com (Gaetan Kenway) Date: Mon, 24 Nov 2014 08:03:17 -0500 Subject: [petsc-users] Convergence of transposed linear system. In-Reply-To: References: Message-ID: That is a good idea to try Francois. Do you know if there is a easy way to try that in PETSc? However, in my case, I'm not using an upwind scheme, but rather a 2nd order JST scheme for the preconditioner. Also, we have observed the same behavior even for Euler systems, although both the direct/adjoint systems in this case are easier to solve and the difference between the systems is less dramatic. I also though about using left preconditioning for the adjoint system instead of right preconditioning, but left preconditioning consistently fails even for the untransposed system. I have no idea why left preconditioning doesn't work. Gaetan On Mon, Nov 24, 2014 at 6:24 AM, francois Pacull wrote: > Hello, > > This is just an idea but this might be due to the fact that the structure > of the preconditioner is severely unsymmetrical when using a first-order > upwind scheme without viscous terms: when building the overlap, the > non-zero terms in the row-wise extra-diagonal blocks yield the list of > vertices to add to each subdomain. If you use the transpose of the > preconditioner, it still uses the row-wise and not the column-wise > extra-diagonal blocks. So maybe you should build the ASM(1) preconditioner > with the untransposed matrix first, and then transpose the preconditioning > matrix? You may also change the side of the preconditioner, for the > transposed system. > > Francois. > > > ------------------------------ > Date: Sun, 23 Nov 2014 20:54:20 -0500 > From: gaetank at gmail.com > To: petsc-users at mcs.anl.gov > Subject: [petsc-users] Convergence of transposed linear system. > > Hi everyone > > I have a question relating to preconditioning effectiveness on large > transposed systems. The linear system I'm trying to solve is jacobian > matrix of 3D RANS CFD solver. The bock matrix consists of about 3 million > block rows with a block size of 6: 5 for the inviscid part and 1 for the SA > turbulence model. > > The preconditioning matrix is different from the linear system matrix in > two ways: It uses a first order discretization (instead of second order) > and the viscous fluxes are dropped. > > The untransposed system converges about 6 orders of magnitude with > GRMES(100), ASM (overlap 1) and ILU(1) with RCM reordering. The test is run > on 128 processors. There are no convergence difficulties. > > However, when I try to solve the transpose of the same system, by either > calling KSPSolveTranspose() or by assembling the transpose of the linear > system and its preconditioner and calling KSPSolve(), GMRES stagnates after > a negligible drop in the residual and no further progress is made. > > I have successfully solved this transpose system by using a different > preconditioner that includes the complete linearization of the viscous > terms (~4 times as many non-zeros in PC matrix) and a much much stronger > preconditioner (ASM(2), ILU(2) with 200 GMRES reset. > > My question is why does the solution of the transpose system with the same > method perform so terribly? Is it normal that vastly stronger > preconditioning method is required to solve transpose systems? > > Any suggestions would be greatly appreciated > > Gaetan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.antonio.magri at gmail.com Mon Nov 24 10:03:59 2014 From: victor.antonio.magri at gmail.com (Victor Magri) Date: Mon, 24 Nov 2014 14:03:59 -0200 Subject: [petsc-users] Why does CHOLMOD solution time differs between PETSc and MATLAB? In-Reply-To: <30CD8E75-9944-435F-B441-BCF753978685@mcs.anl.gov> References: <98D4398D-7858-4CDA-A534-7A38F3E5BB1B@anl.gov> <30CD8E75-9944-435F-B441-BCF753978685@mcs.anl.gov> Message-ID: On Sun, Nov 23, 2014 at 4:12 PM, Barry Smith wrote: > Performance for the various orderings can be slightly "counter" intuitive > if all you count is flops Yes. As you pointed, there are more variables to analyze besides flops. Did you run with the default PETSc Cholesky? I'd be interested in seeing > how long that takes. I've just run with the default cholesky, it took about 30 min for executing (See attached). A brief summary: MatCholFctrSym 1 1.0 1.0375e+03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 65 0 0 0 0 65 0 0 0 0 0 MatCholFctrNum 1 1.0 5.4339e+02 1.0 1.00e+06 1.0 0.0e+00 0.0e+00 0.0e+00 34 0 0 0 0 34 0 0 0 0 0 PCSetUp 1 1.0 1.5810e+03 1.0 1.00e+06 1.0 0.0e+00 0.0e+00 0.0e+00 99 0 0 0 0 99 0 0 0 0 0 The symbolic and numeric Cholesky factorization are taking too long. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_natural.dat Type: application/octet-stream Size: 13730 bytes Desc: not available URL: From haakon at hakostra.net Mon Nov 24 13:10:34 2014 From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=) Date: Mon, 24 Nov 2014 20:10:34 +0100 Subject: [petsc-users] Problem with PETSc + HDF5 VecView Message-ID: <547382AA.10705@hakostra.net> Hi, I have some problems with PETSc and HDF5 VecLoad/VecView. The VecLoad problems can rest for now, but the VecView are more serious. In short: I have a 3D DMDA with and some vectors that I want to save to a HDF5 file. This works perfectly on my workstation, but not on the compute cluster I have access to. I have attached a typical error message. I have also attached an piece of code that can trigger the error. The code is merely a 2D->3D rewrite of DMDA ex 10 (http://www.mcs.anl.gov/petsc/petsc-current/src/dm/examples/tutorials/ex10.c.html), nothing else is done. The program typically works on small number of processes. I have successfully executed the attached program on up to 32 processes. That works. Always. I have never had a single success when trying to run on 64 processes. Always same error. The computer I am struggling with is an SGI machine with SLES 11sp1 and Intel CPUs, hence I have used Intels compilers. I have tried both 2013, 2014 and 2015 versions of the compilers, so that's probably not the cause. I have also tried GCC 4.9.1, just to be safe, same error there. The same compiler is used for both HDF5 and PETSc. The same error message occurs for both debug and release builds. I have tried HDF5 versions 1.8.11 and 1.8.13. I have tried PETSc version 3.4.1 and the latest from Git. The MPI implementation on the machine is SGI's MPT, and i have tried both 2.06 and 2.10. Always same error. Other MPI implementations is unfortunately not available. What really drives me mad is that this works like a charm on my workstation with Linux Mint... I have successfully executed the attached example on 254 processes (my machine breaks down if I try anything more than that). Does any of you have any tips on how to attack this problem and find out what's wrong? Regards, H?kon Strandenes -------------- next part -------------- A non-text attachment was scrubbed... Name: error.log Type: text/x-log Size: 12488 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PETSc-DM-ex10.tar.gz Type: application/gzip Size: 2236 bytes Desc: not available URL: From evanum at gmail.com Mon Nov 24 13:28:41 2014 From: evanum at gmail.com (Evan Um) Date: Mon, 24 Nov 2014 11:28:41 -0800 Subject: [petsc-users] Difference between PETSC 3.5.0.0 and 3.5.1.0 Message-ID: Dear PETSC users, I tried to compile my PETSC application code with PETSC 3.5.1.0 but error messages below. For example, some PETSC types were not identified or defined multiple times. The code has #include and is compiled with 3.5.0 without any error. I wonder if there is a critical change from 3.5.0.0 to 3.5.1.0. I don't see any changes in Documentation: Changes: 3.5 from PETSC webpage. Thanks for your help. Regards, Evan CC -c -O3 esu_main.cpp In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(274): error: identifier "PETSC_CXX_STATIC_INLINE" is undefined PETSC_STATIC_INLINE PetscReal PetscAbsScalar(PetscScalar a) {return a < 0.0 ? -a : a;} ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(274): error: "PetscReal" has already been declared in the current scope PETSC_STATIC_INLINE PetscReal PetscAbsScalar(PetscScalar a) {return a < 0.0 ? -a : a;} ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(274): error: expected a ";" PETSC_STATIC_INLINE PetscReal PetscAbsScalar(PetscScalar a) {return a < 0.0 ? -a : a;} ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(315): warning #12: parsing restarts here after previous syntax error typedef enum { PETSC_SCALAR_DOUBLE,PETSC_SCALAR_SINGLE, PETSC_SCALAR_LONG_DOUBLE } PetscScalarPrecision; ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(518): error: "PetscInt" has already been declared in the current scope PETSC_STATIC_INLINE PetscInt PetscPowInt(PetscInt base,PetscInt power) ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(518): error: expected a ";" PETSC_STATIC_INLINE PetscInt PetscPowInt(PetscInt base,PetscInt power) ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(404): warning #12: parsing restarts here after previous syntax error typedef enum { PETSC_COPY_VALUES, PETSC_OWN_POINTER, PETSC_USE_POINTER} PetscCopyMode; ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1414): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscBitMemcpy(void*,PetscInt,const void*,PetscInt,PetscInt,PetscDataType); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1414): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscBitMemcpy(void*,PetscInt,const void*,PetscInt,PetscInt,PetscDataType); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1414): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscBitMemcpy(void*,PetscInt,const void*,PetscInt,PetscInt,PetscDataType); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1436): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscStrendswithwhich(const char[],const char *const*,PetscInt*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1457): error: expected a ")" PETSC_EXTERN PetscErrorCode PetscEListFind(PetscInt,const char *const*,const char*,PetscInt*,PetscBool*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1476): error #303: explicit type is missing ("int" assumed) PETSC_EXTERN PetscErrorCode PetscMaxSum(MPI_Comm,const PetscInt[],PetscInt*,PetscInt*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1476): error: parameter "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscMaxSum(MPI_Comm,const PetscInt[],PetscInt*,PetscInt*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1476): error: parameter "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscMaxSum(MPI_Comm,const PetscInt[],PetscInt*,PetscInt*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1478): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode MPIULong_Send(void*,PetscInt,MPI_Datatype,PetscMPIInt,PetscMPIInt,MPI_Comm); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1479): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode MPIULong_Recv(void*,PetscInt,MPI_Datatype,PetscMPIInt,PetscMPIInt,MPI_Comm); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(453): error #77: this declaration has no storage class or type specifier PETSC_STATIC_INLINE void* PetscThreadLocalGetValue(PetscThreadKey key) ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(453): error: expected a ";" PETSC_STATIC_INLINE void* PetscThreadLocalGetValue(PetscThreadKey key) ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(485): warning #12: parsing restarts here after previous syntax error } PetscStack; ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(497): error: identifier "PetscStack" is undefined PETSC_EXTERN PetscStack *petscstack; ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(500): error: identifier "PetscStack" is undefined PETSC_EXTERN PetscErrorCode PetscStackCopy(PetscStack*,PetscStack*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(500): error: expected an expression PETSC_EXTERN PetscErrorCode PetscStackCopy(PetscStack*,PetscStack*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(500): error: expected a ")" PETSC_EXTERN PetscErrorCode PetscStackCopy(PetscStack*,PetscStack*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(501): error: expected an expression PETSC_EXTERN PetscErrorCode PetscStackPrint(PetscStack*,FILE* fp); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(501): error: expected a ")" PETSC_EXTERN PetscErrorCode PetscStackPrint(PetscStack*,FILE* fp); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(725): error: variable "PETSC_CXX_STATIC_INLINE" is not a type name PETSC_STATIC_INLINE PetscBool PetscStackActive(void) {return PETSC_FALSE;} ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(725): error: "PetscBool" has already been declared in the current scope PETSC_STATIC_INLINE PetscBool PetscStackActive(void) {return PETSC_FALSE;} ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(725): error: expected a ";" PETSC_STATIC_INLINE PetscBool PetscStackActive(void) {return PETSC_FALSE;} ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(770): warning #12: parsing restarts here after previous syntax error PETSC_EXTERN PetscErrorCode PetscStackCreate(void); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1564): error: expected a ")" PETSC_EXTERN PetscErrorCode PetscInfoAllow(PetscBool ,const char []); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1573): error: expected an expression PETSC_EXTERN PetscErrorCode PetscInitialized(PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1574): error: expected an expression PETSC_EXTERN PetscErrorCode PetscFinalized(PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1609): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscObjectSetTabLevel(PetscObject,PetscInt); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1610): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscObjectGetTabLevel(PetscObject,PetscInt*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1611): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscObjectIncrementTabLevel(PetscObject,PetscObject,PetscInt); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1613): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscObjectGetReference(PetscObject,PetscInt*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1627): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscObjectsGetGlobalNumbering(MPI_Comm,PetscInt,PetscObject*,PetscInt*,PetscInt*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1627): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscObjectsGetGlobalNumbering(MPI_Comm,PetscInt,PetscObject*,PetscInt*,PetscInt*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1627): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscObjectsGetGlobalNumbering(MPI_Comm,PetscInt,PetscObject*,PetscInt*,PetscInt*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(9): error: variable "PetscBool" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsHasName(const char[],const char[],PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(10): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetInt(const char[],const char [],PetscInt *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(10): error: variable "PetscBool" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetInt(const char[],const char [],PetscInt *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(11): error: variable "PetscBool" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetBool(const char[],const char [],PetscBool *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(11): error: variable "PetscBool" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetBool(const char[],const char [],PetscBool *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(12): error: variable "PetscReal" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetReal(const char[],const char[],PetscReal *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(12): error: variable "PetscBool" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetReal(const char[],const char[],PetscReal *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(13): error: variable "PetscBool" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetScalar(const char[],const char[],PetscScalar *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(14): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetIntArray(const char[],const char[],PetscInt[],PetscInt *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(14): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetIntArray(const char[],const char[],PetscInt[],PetscInt *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(15): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetRealArray(const char[],const char[],PetscReal[],PetscInt *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(15): error: variable "PetscBool" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetRealArray(const char[],const char[],PetscReal[],PetscInt *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(16): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetScalarArray(const char[],const char[],PetscScalar[],PetscInt *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(17): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetBoolArray(const char[],const char[],PetscBool [],PetscInt *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(17): error: variable "PetscBool" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetBoolArray(const char[],const char[],PetscBool [],PetscInt *,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(18): error: variable "PetscBool" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetString(const char[],const char[],char[],size_t,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(19): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetStringArray(const char[],const char[],char*[],PetscInt*,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(19): error: variable "PetscBool" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetStringArray(const char[],const char[],char*[],PetscInt*,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(20): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsGetEList(const char[],const char[],const char*const*,PetscInt,PetscInt*,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(50): error: variable "PetscReal" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsStringToReal(const char[],PetscReal*); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(204): error: variable "PetscInt" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsInt(const char[],const char[],const char[],PetscInt,PetscInt*,PetscBool *); ^ ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(205): error: variable "PetscReal" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsReal(const char[],const char[],const char[],PetscReal,PetscReal*,PetscBool *); ^ In file included from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), from /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), from esu_driver_csem.h(19), from esu_main.cpp(19): /opt/cray/petsc/ 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(205): error: variable "PetscBool" is not a type name PETSC_EXTERN PetscErrorCode PetscOptionsReal(const char[],const char[],const char[],PetscReal,PetscReal*,PetscBool *); ^ compilation aborted for esu_main.cpp (code 4) make: *** [esu_main.o] Error 4 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bavier at cray.com Mon Nov 24 13:43:24 2014 From: bavier at cray.com (Eric Bavier) Date: Mon, 24 Nov 2014 13:43:24 -0600 Subject: [petsc-users] petsc-users Digest, Vol 71, Issue 59 In-Reply-To: References: Message-ID: <87ioi4tmeb.fsf@cray.com> petsc-users-request at mcs.anl.gov writes: > Message: 2 > Date: Mon, 24 Nov 2014 11:28:41 -0800 > From: Evan Um > To: petsc-users > Subject: [petsc-users] Difference between PETSC 3.5.0.0 and 3.5.1.0 > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Dear PETSC users, > > I tried to compile my PETSC application code with PETSC 3.5.1.0 but error > messages below. For example, some PETSC types were not identified or > defined multiple times. The code has #include and is compiled > with 3.5.0 without any error. I wonder if there is a critical change from > 3.5.0.0 to 3.5.1.0. I don't see any changes in Documentation: Changes: 3.5 > from PETSC webpage. Thanks for your help. > > Regards, > Evan > > CC -c -O3 esu_main.cpp > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(274): > error: identifier "PETSC_CXX_STATIC_INLINE" is undefined Evan, It looks like you're encountering a known bug in the cray release of petsc 3.5.1.0, which affects use of petsc with C++ compilers. This bug will be fixed in cray-petsc/3.5.2.0. In the meantime, you should be able to safely add the following two lines before #include :: #define PETSC_CXX_STATIC_INLINE static inline #define PETSC_CXX_RESTRICT __restrict__ -- Eric Bavier, Scientific Libraries, Cray Inc. From balay at mcs.anl.gov Mon Nov 24 13:48:30 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 24 Nov 2014 13:48:30 -0600 Subject: [petsc-users] Difference between PETSC 3.5.0.0 and 3.5.1.0 In-Reply-To: References: Message-ID: If using PETSc makefile format - you can try: make executable CXX=CC PCC_LINKER=CC CPPFLAGS='-DPETSC_CXX_STATIC_INLINE="static inline" -DPETSC_CXX_RESTRICT=__restrict__' [or add the equivalent stuff to your makefile] The issue is - the newer versions of PETSc from cray are not configured with the c++ compiler - so some of the flags required to use it from c++ are missing from petscconf.h file. BTW: The 3.5.0.0 version you refere to is perhaps not a cray build.. Satish On Mon, 24 Nov 2014, Evan Um wrote: > Dear PETSC users, > > I tried to compile my PETSC application code with PETSC 3.5.1.0 but error > messages below. For example, some PETSC types were not identified or > defined multiple times. The code has #include and is compiled > with 3.5.0 without any error. I wonder if there is a critical change from > 3.5.0.0 to 3.5.1.0. I don't see any changes in Documentation: Changes: 3.5 > from PETSC webpage. Thanks for your help. > > Regards, > Evan > > CC -c -O3 esu_main.cpp > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(274): > error: identifier "PETSC_CXX_STATIC_INLINE" is undefined > PETSC_STATIC_INLINE PetscReal PetscAbsScalar(PetscScalar a) {return a < > 0.0 ? -a : a;} > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(274): > error: "PetscReal" has already been declared in the current scope > PETSC_STATIC_INLINE PetscReal PetscAbsScalar(PetscScalar a) {return a < > 0.0 ? -a : a;} > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(274): > error: expected a ";" > PETSC_STATIC_INLINE PetscReal PetscAbsScalar(PetscScalar a) {return a < > 0.0 ? -a : a;} > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(315): > warning #12: parsing restarts here after previous syntax error > typedef enum { PETSC_SCALAR_DOUBLE,PETSC_SCALAR_SINGLE, > PETSC_SCALAR_LONG_DOUBLE } PetscScalarPrecision; > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(518): > error: "PetscInt" has already been declared in the current scope > PETSC_STATIC_INLINE PetscInt PetscPowInt(PetscInt base,PetscInt power) > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(518): > error: expected a ";" > PETSC_STATIC_INLINE PetscInt PetscPowInt(PetscInt base,PetscInt power) > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(404): > warning #12: parsing restarts here after previous syntax error > typedef enum { PETSC_COPY_VALUES, PETSC_OWN_POINTER, PETSC_USE_POINTER} > PetscCopyMode; > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1414): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscBitMemcpy(void*,PetscInt,const > void*,PetscInt,PetscInt,PetscDataType); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1414): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscBitMemcpy(void*,PetscInt,const > void*,PetscInt,PetscInt,PetscDataType); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1414): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscBitMemcpy(void*,PetscInt,const > void*,PetscInt,PetscInt,PetscDataType); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1436): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscStrendswithwhich(const char[],const char > *const*,PetscInt*); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1457): > error: expected a ")" > PETSC_EXTERN PetscErrorCode PetscEListFind(PetscInt,const char > *const*,const char*,PetscInt*,PetscBool*); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1476): > error #303: explicit type is missing ("int" assumed) > PETSC_EXTERN PetscErrorCode PetscMaxSum(MPI_Comm,const > PetscInt[],PetscInt*,PetscInt*); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1476): > error: parameter "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscMaxSum(MPI_Comm,const > PetscInt[],PetscInt*,PetscInt*); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1476): > error: parameter "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscMaxSum(MPI_Comm,const > PetscInt[],PetscInt*,PetscInt*); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1478): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode > MPIULong_Send(void*,PetscInt,MPI_Datatype,PetscMPIInt,PetscMPIInt,MPI_Comm); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1479): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode > MPIULong_Recv(void*,PetscInt,MPI_Datatype,PetscMPIInt,PetscMPIInt,MPI_Comm); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(453): > error #77: this declaration has no storage class or type specifier > PETSC_STATIC_INLINE void* PetscThreadLocalGetValue(PetscThreadKey key) > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(453): > error: expected a ";" > PETSC_STATIC_INLINE void* PetscThreadLocalGetValue(PetscThreadKey key) > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(485): > warning #12: parsing restarts here after previous syntax error > } PetscStack; > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(497): > error: identifier "PetscStack" is undefined > PETSC_EXTERN PetscStack *petscstack; > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(500): > error: identifier "PetscStack" is undefined > PETSC_EXTERN PetscErrorCode PetscStackCopy(PetscStack*,PetscStack*); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(500): > error: expected an expression > PETSC_EXTERN PetscErrorCode PetscStackCopy(PetscStack*,PetscStack*); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(500): > error: expected a ")" > PETSC_EXTERN PetscErrorCode PetscStackCopy(PetscStack*,PetscStack*); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(501): > error: expected an expression > PETSC_EXTERN PetscErrorCode PetscStackPrint(PetscStack*,FILE* fp); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(501): > error: expected a ")" > PETSC_EXTERN PetscErrorCode PetscStackPrint(PetscStack*,FILE* fp); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(725): > error: variable "PETSC_CXX_STATIC_INLINE" is not a type name > PETSC_STATIC_INLINE PetscBool PetscStackActive(void) {return PETSC_FALSE;} > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(725): > error: "PetscBool" has already been declared in the current scope > PETSC_STATIC_INLINE PetscBool PetscStackActive(void) {return PETSC_FALSE;} > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(725): > error: expected a ";" > PETSC_STATIC_INLINE PetscBool PetscStackActive(void) {return PETSC_FALSE;} > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(770): > warning #12: parsing restarts here after previous syntax error > PETSC_EXTERN PetscErrorCode PetscStackCreate(void); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1564): > error: expected a ")" > PETSC_EXTERN PetscErrorCode PetscInfoAllow(PetscBool ,const char []); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1573): > error: expected an expression > PETSC_EXTERN PetscErrorCode PetscInitialized(PetscBool *); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1574): > error: expected an expression > PETSC_EXTERN PetscErrorCode PetscFinalized(PetscBool *); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1609): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscObjectSetTabLevel(PetscObject,PetscInt); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1610): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscObjectGetTabLevel(PetscObject,PetscInt*); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1611): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode > PetscObjectIncrementTabLevel(PetscObject,PetscObject,PetscInt); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1613): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode > PetscObjectGetReference(PetscObject,PetscInt*); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1627): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode > PetscObjectsGetGlobalNumbering(MPI_Comm,PetscInt,PetscObject*,PetscInt*,PetscInt*); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1627): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode > PetscObjectsGetGlobalNumbering(MPI_Comm,PetscInt,PetscObject*,PetscInt*,PetscInt*); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1627): > error: variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode > PetscObjectsGetGlobalNumbering(MPI_Comm,PetscInt,PetscObject*,PetscInt*,PetscInt*); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(9): > error: variable "PetscBool" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsHasName(const char[],const > char[],PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(10): error: > variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetInt(const char[],const char > [],PetscInt *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(10): error: > variable "PetscBool" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetInt(const char[],const char > [],PetscInt *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(11): error: > variable "PetscBool" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetBool(const char[],const char > [],PetscBool *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(11): error: > variable "PetscBool" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetBool(const char[],const char > [],PetscBool *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(12): error: > variable "PetscReal" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetReal(const char[],const > char[],PetscReal *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(12): error: > variable "PetscBool" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetReal(const char[],const > char[],PetscReal *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(13): error: > variable "PetscBool" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetScalar(const char[],const > char[],PetscScalar *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(14): error: > variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetIntArray(const char[],const > char[],PetscInt[],PetscInt *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(14): error: > variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetIntArray(const char[],const > char[],PetscInt[],PetscInt *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(15): error: > variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetRealArray(const char[],const > char[],PetscReal[],PetscInt *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(15): error: > variable "PetscBool" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetRealArray(const char[],const > char[],PetscReal[],PetscInt *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(16): error: > variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetScalarArray(const char[],const > char[],PetscScalar[],PetscInt *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(17): error: > variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetBoolArray(const char[],const > char[],PetscBool [],PetscInt *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(17): error: > variable "PetscBool" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetBoolArray(const char[],const > char[],PetscBool [],PetscInt *,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(18): error: > variable "PetscBool" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetString(const char[],const > char[],char[],size_t,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(19): error: > variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetStringArray(const char[],const > char[],char*[],PetscInt*,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(19): error: > variable "PetscBool" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetStringArray(const char[],const > char[],char*[],PetscInt*,PetscBool *); > > ^ > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(20): error: > variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsGetEList(const char[],const > char[],const char*const*,PetscInt,PetscInt*,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(50): error: > variable "PetscReal" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsStringToReal(const > char[],PetscReal*); > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(204): error: > variable "PetscInt" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsInt(const char[],const > char[],const char[],PetscInt,PetscInt*,PetscBool *); > > ^ > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(205): error: > variable "PetscReal" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsReal(const char[],const > char[],const char[],PetscReal,PetscReal*,PetscBool *); > > ^ > > In file included from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > from /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > from esu_driver_csem.h(19), > from esu_main.cpp(19): > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(205): error: > variable "PetscBool" is not a type name > PETSC_EXTERN PetscErrorCode PetscOptionsReal(const char[],const > char[],const char[],PetscReal,PetscReal*,PetscBool *); > > ^ > > compilation aborted for esu_main.cpp (code 4) > make: *** [esu_main.o] Error 4 > From balay at mcs.anl.gov Mon Nov 24 13:52:29 2014 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 24 Nov 2014 13:52:29 -0600 Subject: [petsc-users] petsc-users Digest, Vol 71, Issue 59 In-Reply-To: <87ioi4tmeb.fsf@cray.com> References: <87ioi4tmeb.fsf@cray.com> Message-ID: On Mon, 24 Nov 2014, Eric Bavier wrote: > Evan, > > It looks like you're encountering a known bug in the cray release of > petsc 3.5.1.0, which affects use of petsc with C++ compilers. This bug > will be fixed in cray-petsc/3.5.2.0. In the meantime, you should be > able to safely add the following two lines before #include > :: > > #define PETSC_CXX_STATIC_INLINE static inline > #define PETSC_CXX_RESTRICT __restrict__ Eric, Glad to know the issue will be fixed in cray-petsc/3.5.2.0. Thanks for the update. Satish From evanum at gmail.com Mon Nov 24 15:01:05 2014 From: evanum at gmail.com (Evan Um) Date: Mon, 24 Nov 2014 13:01:05 -0800 Subject: [petsc-users] petsc-users Digest, Vol 71, Issue 59 In-Reply-To: <87ioi4tmeb.fsf@cray.com> References: <87ioi4tmeb.fsf@cray.com> Message-ID: Dear Eric, After adding the two lines before , I was able to compile the code. It runs correctly. Thanks for your help. Regards, Evan On Mon, Nov 24, 2014 at 11:43 AM, Eric Bavier wrote: > > petsc-users-request at mcs.anl.gov writes: > > > Message: 2 > > Date: Mon, 24 Nov 2014 11:28:41 -0800 > > From: Evan Um > > To: petsc-users > > Subject: [petsc-users] Difference between PETSC 3.5.0.0 and 3.5.1.0 > > Message-ID: > > j+JGQ at mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Dear PETSC users, > > > > I tried to compile my PETSC application code with PETSC 3.5.1.0 but error > > messages below. For example, some PETSC types were not identified or > > defined multiple times. The code has #include and is compiled > > with 3.5.0 without any error. I wonder if there is a critical change from > > 3.5.0.0 to 3.5.1.0. I don't see any changes in Documentation: Changes: > 3.5 > > from PETSC webpage. Thanks for your help. > > > > Regards, > > Evan > > > > CC -c -O3 esu_main.cpp > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(274): > > error: identifier "PETSC_CXX_STATIC_INLINE" is undefined > > Evan, > > It looks like you're encountering a known bug in the cray release of > petsc 3.5.1.0, which affects use of petsc with C++ compilers. This bug > will be fixed in cray-petsc/3.5.2.0. In the meantime, you should be > able to safely add the following two lines before #include > :: > > #define PETSC_CXX_STATIC_INLINE static inline > #define PETSC_CXX_RESTRICT __restrict__ > > -- > Eric Bavier, Scientific Libraries, Cray Inc. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From evanum at gmail.com Mon Nov 24 15:03:41 2014 From: evanum at gmail.com (Evan Um) Date: Mon, 24 Nov 2014 13:03:41 -0800 Subject: [petsc-users] Difference between PETSC 3.5.0.0 and 3.5.1.0 In-Reply-To: References: Message-ID: Dear Satish, Thanks for your kind help. After adding the two defines suggested by Eric, I was able to compile and run the code. You are right. The 3.5.0 version was not Cray build. Regards, Evan On Mon, Nov 24, 2014 at 11:48 AM, Satish Balay wrote: > If using PETSc makefile format - you can try: > > make executable CXX=CC PCC_LINKER=CC > CPPFLAGS='-DPETSC_CXX_STATIC_INLINE="static inline" > -DPETSC_CXX_RESTRICT=__restrict__' > > [or add the equivalent stuff to your makefile] > > The issue is - the newer versions of PETSc from cray are not > configured with the c++ compiler - so some of the flags required to > use it from c++ are missing from petscconf.h file. > > BTW: The 3.5.0.0 version you refere to is perhaps not a cray build.. > > Satish > > On Mon, 24 Nov 2014, Evan Um wrote: > > > Dear PETSC users, > > > > I tried to compile my PETSC application code with PETSC 3.5.1.0 but error > > messages below. For example, some PETSC types were not identified or > > defined multiple times. The code has #include and is compiled > > with 3.5.0 without any error. I wonder if there is a critical change from > > 3.5.0.0 to 3.5.1.0. I don't see any changes in Documentation: Changes: > 3.5 > > from PETSC webpage. Thanks for your help. > > > > Regards, > > Evan > > > > CC -c -O3 esu_main.cpp > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(274): > > error: identifier "PETSC_CXX_STATIC_INLINE" is undefined > > PETSC_STATIC_INLINE PetscReal PetscAbsScalar(PetscScalar a) {return a < > > 0.0 ? -a : a;} > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(274): > > error: "PetscReal" has already been declared in the current scope > > PETSC_STATIC_INLINE PetscReal PetscAbsScalar(PetscScalar a) {return a < > > 0.0 ? -a : a;} > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(274): > > error: expected a ";" > > PETSC_STATIC_INLINE PetscReal PetscAbsScalar(PetscScalar a) {return a < > > 0.0 ? -a : a;} > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(315): > > warning #12: parsing restarts here after previous syntax error > > typedef enum { PETSC_SCALAR_DOUBLE,PETSC_SCALAR_SINGLE, > > PETSC_SCALAR_LONG_DOUBLE } PetscScalarPrecision; > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(518): > > error: "PetscInt" has already been declared in the current scope > > PETSC_STATIC_INLINE PetscInt PetscPowInt(PetscInt base,PetscInt power) > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(390), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmath.h(518): > > error: expected a ";" > > PETSC_STATIC_INLINE PetscInt PetscPowInt(PetscInt base,PetscInt power) > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(404): > > warning #12: parsing restarts here after previous syntax error > > typedef enum { PETSC_COPY_VALUES, PETSC_OWN_POINTER, PETSC_USE_POINTER} > > PetscCopyMode; > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1414): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscBitMemcpy(void*,PetscInt,const > > void*,PetscInt,PetscInt,PetscDataType); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1414): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscBitMemcpy(void*,PetscInt,const > > void*,PetscInt,PetscInt,PetscDataType); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1414): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscBitMemcpy(void*,PetscInt,const > > void*,PetscInt,PetscInt,PetscDataType); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1436): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscStrendswithwhich(const char[],const > char > > *const*,PetscInt*); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1457): > > error: expected a ")" > > PETSC_EXTERN PetscErrorCode PetscEListFind(PetscInt,const char > > *const*,const char*,PetscInt*,PetscBool*); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1476): > > error #303: explicit type is missing ("int" assumed) > > PETSC_EXTERN PetscErrorCode PetscMaxSum(MPI_Comm,const > > PetscInt[],PetscInt*,PetscInt*); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1476): > > error: parameter "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscMaxSum(MPI_Comm,const > > PetscInt[],PetscInt*,PetscInt*); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1476): > > error: parameter "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscMaxSum(MPI_Comm,const > > PetscInt[],PetscInt*,PetscInt*); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1478): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode > > > MPIULong_Send(void*,PetscInt,MPI_Datatype,PetscMPIInt,PetscMPIInt,MPI_Comm); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1479): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode > > > MPIULong_Recv(void*,PetscInt,MPI_Datatype,PetscMPIInt,PetscMPIInt,MPI_Comm); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(453): > > error #77: this declaration has no storage class or type specifier > > PETSC_STATIC_INLINE void* PetscThreadLocalGetValue(PetscThreadKey key) > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(453): > > error: expected a ";" > > PETSC_STATIC_INLINE void* PetscThreadLocalGetValue(PetscThreadKey key) > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(485): > > warning #12: parsing restarts here after previous syntax error > > } PetscStack; > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(497): > > error: identifier "PetscStack" is undefined > > PETSC_EXTERN PetscStack *petscstack; > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(500): > > error: identifier "PetscStack" is undefined > > PETSC_EXTERN PetscErrorCode PetscStackCopy(PetscStack*,PetscStack*); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(500): > > error: expected an expression > > PETSC_EXTERN PetscErrorCode PetscStackCopy(PetscStack*,PetscStack*); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(500): > > error: expected a ")" > > PETSC_EXTERN PetscErrorCode PetscStackCopy(PetscStack*,PetscStack*); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(501): > > error: expected an expression > > PETSC_EXTERN PetscErrorCode PetscStackPrint(PetscStack*,FILE* fp); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(501): > > error: expected a ")" > > PETSC_EXTERN PetscErrorCode PetscStackPrint(PetscStack*,FILE* fp); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(725): > > error: variable "PETSC_CXX_STATIC_INLINE" is not a type name > > PETSC_STATIC_INLINE PetscBool PetscStackActive(void) {return > PETSC_FALSE;} > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(725): > > error: "PetscBool" has already been declared in the current scope > > PETSC_STATIC_INLINE PetscBool PetscStackActive(void) {return > PETSC_FALSE;} > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(725): > > error: expected a ";" > > PETSC_STATIC_INLINE PetscBool PetscStackActive(void) {return > PETSC_FALSE;} > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1549), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscerror.h(770): > > warning #12: parsing restarts here after previous syntax error > > PETSC_EXTERN PetscErrorCode PetscStackCreate(void); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1564): > > error: expected a ")" > > PETSC_EXTERN PetscErrorCode PetscInfoAllow(PetscBool ,const char []); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1573): > > error: expected an expression > > PETSC_EXTERN PetscErrorCode PetscInitialized(PetscBool *); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1574): > > error: expected an expression > > PETSC_EXTERN PetscErrorCode PetscFinalized(PetscBool *); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1609): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode > PetscObjectSetTabLevel(PetscObject,PetscInt); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1610): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode > PetscObjectGetTabLevel(PetscObject,PetscInt*); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1611): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode > > PetscObjectIncrementTabLevel(PetscObject,PetscObject,PetscInt); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1613): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode > > PetscObjectGetReference(PetscObject,PetscInt*); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1627): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode > > > PetscObjectsGetGlobalNumbering(MPI_Comm,PetscInt,PetscObject*,PetscInt*,PetscInt*); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1627): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode > > > PetscObjectsGetGlobalNumbering(MPI_Comm,PetscInt,PetscObject*,PetscInt*,PetscInt*); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1627): > > error: variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode > > > PetscObjectsGetGlobalNumbering(MPI_Comm,PetscInt,PetscObject*,PetscInt*,PetscInt*); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(9): > > error: variable "PetscBool" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsHasName(const char[],const > > char[],PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(10): error: > > variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetInt(const char[],const char > > [],PetscInt *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(10): error: > > variable "PetscBool" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetInt(const char[],const char > > [],PetscInt *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(11): error: > > variable "PetscBool" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetBool(const char[],const char > > [],PetscBool *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(11): error: > > variable "PetscBool" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetBool(const char[],const char > > [],PetscBool *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(12): error: > > variable "PetscReal" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetReal(const char[],const > > char[],PetscReal *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(12): error: > > variable "PetscBool" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetReal(const char[],const > > char[],PetscReal *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(13): error: > > variable "PetscBool" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetScalar(const char[],const > > char[],PetscScalar *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(14): error: > > variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetIntArray(const char[],const > > char[],PetscInt[],PetscInt *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(14): error: > > variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetIntArray(const char[],const > > char[],PetscInt[],PetscInt *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(15): error: > > variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetRealArray(const char[],const > > char[],PetscReal[],PetscInt *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(15): error: > > variable "PetscBool" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetRealArray(const char[],const > > char[],PetscReal[],PetscInt *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(16): error: > > variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetScalarArray(const > char[],const > > char[],PetscScalar[],PetscInt *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(17): error: > > variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetBoolArray(const char[],const > > char[],PetscBool [],PetscInt *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(17): error: > > variable "PetscBool" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetBoolArray(const char[],const > > char[],PetscBool [],PetscInt *,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(18): error: > > variable "PetscBool" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetString(const char[],const > > char[],char[],size_t,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(19): error: > > variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetStringArray(const > char[],const > > char[],char*[],PetscInt*,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(19): error: > > variable "PetscBool" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetStringArray(const > char[],const > > char[],char*[],PetscInt*,PetscBool *); > > > > ^ > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(20): error: > > variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsGetEList(const char[],const > > char[],const char*const*,PetscInt,PetscInt*,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(50): error: > > variable "PetscReal" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsStringToReal(const > > char[],PetscReal*); > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(204): error: > > variable "PetscInt" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsInt(const char[],const > > char[],const char[],PetscInt,PetscInt*,PetscBool *); > > > > ^ > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(205): error: > > variable "PetscReal" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsReal(const char[],const > > char[],const char[],PetscReal,PetscReal*,PetscBool *); > > > > ^ > > > > In file included from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscsys.h(1630), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscis.h(7), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscvec.h(9), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscmat.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscpc.h(6), > > from /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscksp.h(6), > > from esu_driver_csem.h(19), > > from esu_main.cpp(19): > > /opt/cray/petsc/ > > 3.5.1.0/real/INTEL/140/sandybridge/include/petscoptions.h(205): error: > > variable "PetscBool" is not a type name > > PETSC_EXTERN PetscErrorCode PetscOptionsReal(const char[],const > > char[],const char[],PetscReal,PetscReal*,PetscBool *); > > > > ^ > > > > compilation aborted for esu_main.cpp (code 4) > > make: *** [esu_main.o] Error 4 > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fpacull at hotmail.com Mon Nov 24 15:50:51 2014 From: fpacull at hotmail.com (francois Pacull) Date: Mon, 24 Nov 2014 16:50:51 -0500 Subject: [petsc-users] Convergence of transposed linear system. In-Reply-To: References: , , Message-ID: Gaetan, Do you observe this behavior on some smaller matrices? I would say that an easy way to test this on a small linear system is to build the PC with the untransposed preconditioning matrix, get the overlapping partition IS with PCASMGetLocalSubdomains, destroy the PC, create a new one with the transposed preconditioning matrix, and set the partition with PCASMSetLocalSubdomains (using the stored IS). And did you ever observe this behavior without any overlap: ASM(overlap 0)? What do you mean when you say that left preconditioning fails? Does it stagnate? Converge slowly? It is rather strange I think. Francois. Date: Mon, 24 Nov 2014 08:03:17 -0500 Subject: Re: [petsc-users] Convergence of transposed linear system. From: gaetank at gmail.com To: fpacull at hotmail.com CC: petsc-users at mcs.anl.gov That is a good idea to try Francois. Do you know if there is a easy way to try that in PETSc? However, in my case, I'm not using an upwind scheme, but rather a 2nd order JST scheme for the preconditioner. Also, we have observed the same behavior even for Euler systems, although both the direct/adjoint systems in this case are easier to solve and the difference between the systems is less dramatic. I also though about using left preconditioning for the adjoint system instead of right preconditioning, but left preconditioning consistently fails even for the untransposed system. I have no idea why left preconditioning doesn't work. Gaetan On Mon, Nov 24, 2014 at 6:24 AM, francois Pacull wrote: Hello, This is just an idea but this might be due to the fact that the structure of the preconditioner is severely unsymmetrical when using a first-order upwind scheme without viscous terms: when building the overlap, the non-zero terms in the row-wise extra-diagonal blocks yield the list of vertices to add to each subdomain. If you use the transpose of the preconditioner, it still uses the row-wise and not the column-wise extra-diagonal blocks. So maybe you should build the ASM(1) preconditioner with the untransposed matrix first, and then transpose the preconditioning matrix? You may also change the side of the preconditioner, for the transposed system. Francois. Date: Sun, 23 Nov 2014 20:54:20 -0500 From: gaetank at gmail.com To: petsc-users at mcs.anl.gov Subject: [petsc-users] Convergence of transposed linear system. Hi everyone I have a question relating to preconditioning effectiveness on large transposed systems. The linear system I'm trying to solve is jacobian matrix of 3D RANS CFD solver. The bock matrix consists of about 3 million block rows with a block size of 6: 5 for the inviscid part and 1 for the SA turbulence model. The preconditioning matrix is different from the linear system matrix in two ways: It uses a first order discretization (instead of second order) and the viscous fluxes are dropped. The untransposed system converges about 6 orders of magnitude with GRMES(100), ASM (overlap 1) and ILU(1) with RCM reordering. The test is run on 128 processors. There are no convergence difficulties. However, when I try to solve the transpose of the same system, by either calling KSPSolveTranspose() or by assembling the transpose of the linear system and its preconditioner and calling KSPSolve(), GMRES stagnates after a negligible drop in the residual and no further progress is made. I have successfully solved this transpose system by using a different preconditioner that includes the complete linearization of the viscous terms (~4 times as many non-zeros in PC matrix) and a much much stronger preconditioner (ASM(2), ILU(2) with 200 GMRES reset. My question is why does the solution of the transpose system with the same method perform so terribly? Is it normal that vastly stronger preconditioning method is required to solve transpose systems? Any suggestions would be greatly appreciated Gaetan -------------- next part -------------- An HTML attachment was scrubbed... URL: From thronesf at gmail.com Mon Nov 24 16:07:53 2014 From: thronesf at gmail.com (Sharp Stone) Date: Mon, 24 Nov 2014 17:07:53 -0500 Subject: [petsc-users] Linear Solver Problem Message-ID: Dear all, I encountered a weird results from petsc linear solver. I have output the same matrix and RHS vector from the linear system, and solve it also with matlab. When dof is 1, both solvers from Matlab and Petsc can give me the same results, However, Matlab can give me the right result but petsc does not when dof=4. I attached the matrix and RHS vectors for dof=4 case. I don't know where I made wrong, or else. Thanks a million in advance! -- Best regards, Feng -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Mat0.dat Type: application/octet-stream Size: 4969024 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: RHSVec0.dat Type: application/octet-stream Size: 164840 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: compare.jpg Type: image/jpeg Size: 181030 bytes Desc: not available URL: From olivier.bonnefon at avignon.inra.fr Tue Nov 25 04:08:46 2014 From: olivier.bonnefon at avignon.inra.fr (Olivier Bonnefon) Date: Tue, 25 Nov 2014 11:08:46 +0100 Subject: [petsc-users] petsc-3.5.2: ex12 with Neumann BC In-Reply-To: References: <524430F2.2020507@avignon.inra.fr> <524D3D5D.5090301@avignon.inra.fr> <525804C2.3000101@avignon.inra.fr> <5453AE09.5090806@avignon.inra.fr> <54574EED.3060309@avignon.inra.fr> Message-ID: <5474552E.2080808@avignon.inra.fr> Hello, Thank you for your answer, I agree with you. I have adapted the ex12 example for the system: -\nabla \nabla u + u + f = 0 in \Omega (1) with f = 4-x^2-y^2 The solution is still x^2-y^2. It consists in adding the following gradient function: void g0_uu(const PetscScalar u[], const PetscScalar u_t[], const PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar a_x[], const PetscReal x[], PetscScalar g0[]) { g0[0] = 1.0; } ... //and set gradient ierr = PetscDSSetJacobian(prob, 0, 0, g0_uu, NULL, NULL, g3_uu);CHKERRQ(ierr); ... Of course, I have modified the f0_u function. With Dirichlet BC, I get the solution. With Neumann BC, the solution is still shifted down (-2/3). The problem (1) with Neumann BC has a unique solution. How remove the condition \int_\Omega u = 0 ? Thanks Olivier Bonnefon On 11/06/2014 07:14 AM, Matthew Knepley wrote: > On Mon, Nov 3, 2014 at 9:46 AM, Olivier Bonnefon > > wrote: > > Hello, > > Thank for your answer, I'll explain my trouble: > > My problem is that the BC Neumann leads to wrong result. > > With Dirichlet BC, I get: > ------------------------------- > > > ./ex12 -run_type full -refinement_limit 0.0 -bc_type dirichlet > -interpolate 0 -petscspace_order 1 -show_initial -show_solution > -dm_plex_print_fem 1 > ... > ... > Solution > Vec Object:potential 1 MPI processes > type: seq > 0.5 > > This result is correct. > > With Neuman BC, I get: > -------------------------------- > > > ./ex12 -run_type full -refinement_limit 0.0 -bc_type neumann > -interpolate 1 -petscspace_order 1 -show_initial > -dm_plex_print_fem 1 -show_solution -bd_petscspace_order 1 > -snes_linesearch_monitor -snes_monitor -ksp_monitor_true_residual > -snes_converged_reason -ksp_converged_reason > .... > .... > > > Solution > Vec Object:potential 1 MPI processes > type: seq > -0.75 > -0.583333 > 0.0833333 > -0.583333 > -0.333333 > 0.416667 > 0.0833333 > 0.416667 > 1.25 > > > That is not the values of the solution x*x+y*y. > > > Sorry, this is poor documentation on my part. I will fix it in the > example. The Naumann solution > explicitly discards the constant part, meaning > > \int_\Omega u = 0 > > If we look at my solution > > \int^1_0 dx \int^1_0 dy x^2 + y^2 = 2/3 > > However, this is for the continuum result, whereas we are enforcing > this in the discrete case. > Thus, what we really get is 0.75, since we are integrating linear > patches instead of quadratic > patches. After shifting by that, you still do not get exactly x^2 + > y^2 since there is some > discretization error. If you run with P2 you should get the exact > answer, shifted down to eliminate > the DC component. > > Thanks, > > Matt > > > I tried many ksp options. > Moreover, the neumann BC with "-run_type full" is not cover in the > list > https://bitbucket.org/petsc/petsc/src/fced3c3f9e703542693913793d15321603e40fe6/config/builder.py?at=master#cl-257 > > Do you know what is wrong ? > > Thanks, > > Olivier Bonnefon > > On 10/31/2014 06:50 PM, Matthew Knepley wrote: >> On Fri, Oct 31, 2014 at 10:43 AM, Olivier Bonnefon >> > > wrote: >> >> Hello, >> >> I'm working on the snes/examples/tutorial/ex12 version 3.5.2. >> >> I didn't succed to run the simplest case with Neumann BC: >> >> ./ex12 -run_type full -refinement_limit 0.0 -bc_type neumann >> -interpolate 1 -petscspace_order 1 -show_initial >> -dm_plex_print_fem 1 -show_solution -bd_petscspace_order 1 >> -snes_linesearch_monitor -snes_monitor >> -ksp_monitor_true_residual -snes_converged_reason >> -ksp_converged_reason >> >> This leads to dofs negatives values. >> >> >> I do not understand what you mean. Please always mail the full >> error message. >> >> Do you know the options to get a correct result with Neumann BC ? >> >> >> There are some tests here: >> >> https://bitbucket.org/petsc/petsc/src/fced3c3f9e703542693913793d15321603e40fe6/config/builder.py?at=master#cl-257 >> >> Matt >> >> Regards, >> Olivier Bonnefon >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > -- > Olivier Bonnefon > INRA PACA-Avignon, Unit? BioSP > Tel:+33 (0)4 32 72 21 58 > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -- Olivier Bonnefon INRA PACA-Avignon, Unit? BioSP Tel: +33 (0)4 32 72 21 58 -------------- next part -------------- An HTML attachment was scrubbed... URL: From davydden at gmail.com Tue Nov 25 04:48:01 2014 From: davydden at gmail.com (Denis Davydov) Date: Tue, 25 Nov 2014 11:48:01 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: <3BE6DE10-465D-4D53-8D81-207C426F5502@dsic.upv.es> References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> <98D02121-2E62-47F8-8841-F9D89D1B87DE@gmail.com> <396D16B5-D8C1-4180-A8B5-BC1F65A9C9D5@dsic.upv.es> <386CA68B-2577-462C-A16D-F0A1F4F445B0@gmail.com> <3BE6DE10-465D-4D53-8D81-207C426F5502@dsic.upv.es> Message-ID: <51A03B32-245C-48FC-AE93-DC2A1F4427FA@gmail.com> Hi Jose, A follow up question on KrylovSchur solver: >> One last thing, if I force EPSSetTrueResidual(eps, PETSC_TRUE) >> will that guarantee that EPSComputeRelativeError() >> will give the norm consistent to that, used internally by all SLEPc solvers? > > I would not recommend that, since it is not implemented in all solvers. > > I tried with your matrices. In your case, it is much better to compute eigenvalues close to the origin with shift-and-invert, rather than computing smallest_real eigenvalues. That is, replace > > -eps_smallest_real > > with > > -eps_target 0 -st_type sinvert 1) I tried forcing -eps_true_residual (without sinvert) but have only 2 eigenpairs converged. Same happens with sinvert around 0.0 you recommended above. I would expect more iterations to be required for convergence rather than the solver diverging completely... 2) Another peculiarity is that `-eps_smallest_real` and `-eps_target 0 -st_type sinvert` return different sets of eigenvalues. In the latter case there are degenerate eigenvalues. Those are consistent with the results given by ARPACK with shift and invert around 0. The matrices I sent you originally correspond to the eigenvalues of the 2D Laplace operator on a uniform mesh with 256 cells. If more refined mesh is used (1024 cells instead of 256), same set with degenerate eigenvalues is returned in both cases. This is not directly related to SLEPc and is a question out of curiosity. 3) It is not stated in the documentation explicitly, but I suppose the residual discussed in ?SolverControl? section always corresponds to the direct problem (even in case when, say, shift-and-invert is applied) and so EPSComputeRelativeError and EPSComputeResidualNorm? Sincerely, Denis From kvoronin at labchem.sscc.ru Tue Nov 25 05:14:33 2014 From: kvoronin at labchem.sscc.ru (Kirill Voronin) Date: Tue, 25 Nov 2014 18:14:33 +0700 Subject: [petsc-users] Using PETSC iterative solvers for a multiple righthand side problem In-Reply-To: <63dd84852409b753b072170cbd892396.squirrel@mx.sscc.ru> Message-ID: <325d812eefd8d1072b2f8a0ee80a4731.squirrel@mx.sscc.ru> Hello everyone! Roughly speaking, the question is - what are the possibilities to solve the linear system Ax = b with multiple righthand sides? I am solving using PETSC a linear system Ax = b. Solution method is BiCGStab with my user-defined preconditioner. Can I somehow exploit the fact that I will need to solve a lot of systems with the same matrix but different righthand sides? I mean, can I simultaneously calculate all the solution vectors for all righthand sides like it is done, e.g., in PARDISO? As far as I understand, the only way is to call KSP_solve in a loop over all righthand sides, each time finding only one solution for the current righthand side (changing inside the loop). Am I right? I would greatly appreciate any help! Thanks in advance! Best regards, Kirill Voronin From bsmith at mcs.anl.gov Tue Nov 25 07:01:08 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 25 Nov 2014 07:01:08 -0600 Subject: [petsc-users] Using PETSC iterative solvers for a multiple righthand side problem In-Reply-To: <325d812eefd8d1072b2f8a0ee80a4731.squirrel@mx.sscc.ru> References: <325d812eefd8d1072b2f8a0ee80a4731.squirrel@mx.sscc.ru> Message-ID: We do not support a "generic" multiple right hand sides; solving one at a time is reasonably efficient (that is for iterative solvers you wouldn't see more than roughly a speedup of 50% no matter how many right hand sides you have.) However with direct solvers you can see much more dramatic improvement. How large is your linear system and do you want to do the solves in parallel? You can use MatMatSolve() for multiple right hand sides with direct solvers. Barry > On Nov 25, 2014, at 5:14 AM, Kirill Voronin wrote: > > > Hello everyone! > > Roughly speaking, the question is - what are the possibilities to solve > the linear system Ax = b with multiple righthand sides? > > I am solving using PETSC a linear system Ax = b. Solution method is > BiCGStab with my user-defined preconditioner. > > Can I somehow exploit the fact that I will need to solve a lot of systems > with the same matrix but different righthand sides? > I mean, can I simultaneously calculate all the solution vectors for all > righthand sides like it is done, e.g., in PARDISO? > > As far as I understand, the only way is to call KSP_solve in a loop over > all righthand sides, each time finding only one solution for the current > righthand side (changing inside the loop). Am I right? > > I would greatly appreciate any help! Thanks in advance! > > > Best regards, > > Kirill Voronin > From knepley at gmail.com Tue Nov 25 08:01:41 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Nov 2014 08:01:41 -0600 Subject: [petsc-users] petsc-3.5.2: ex12 with Neumann BC In-Reply-To: <5474552E.2080808@avignon.inra.fr> References: <524430F2.2020507@avignon.inra.fr> <524D3D5D.5090301@avignon.inra.fr> <525804C2.3000101@avignon.inra.fr> <5453AE09.5090806@avignon.inra.fr> <54574EED.3060309@avignon.inra.fr> <5474552E.2080808@avignon.inra.fr> Message-ID: On Tue, Nov 25, 2014 at 4:08 AM, Olivier Bonnefon < olivier.bonnefon at avignon.inra.fr> wrote: > Hello, > > Thank you for your answer, I agree with you. > > I have adapted the ex12 example for the system: > > -\nabla \nabla u + u + f = 0 in \Omega (1) > > with f = 4-x^2-y^2 > The solution is still x^2-y^2. > > It consists in adding the following gradient function: > > void g0_uu(const PetscScalar u[], const PetscScalar u_t[], const > PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const > PetscScalar a_x[], const PetscReal x[], PetscScalar g0[]) > { > g0[0] = 1.0; > } > ... > //and set gradient > ierr = PetscDSSetJacobian(prob, 0, 0, g0_uu, NULL, NULL, > g3_uu);CHKERRQ(ierr); > ... > Of course, I have modified the f0_u function. > > > With Dirichlet BC, I get the solution. > > With Neumann BC, the solution is still shifted down (-2/3). The problem > (1) with Neumann BC has a unique solution. > How remove the condition \int_\Omega u = 0 ? > That condition is one way of imposing boundary conditions for pressure. When I call MatSetNullSpace(), I am implying this condition. You can take out that call, but the system will be singular. You would have to add a condition on the pressure somehow, like fixing it at a point. This tends to make the system more ill-conditioned, but if that is appropriate for you, then you make a similar call to ierr = DMPlexAddBoundary(cdm, user->bcType == DIRICHLET, "wall", user->bcType == NEUMANN ? "boundary" : "marker", 0, user->exactFuncs[0], 1, &id, user);CHKERRQ(ierr); but for field 1. To test , you can jsut change 0 to 1 in this call. However, this sets the entire boundary, and you really only need to set the pressure at one point, so you could make another label that only has a single point and use that instead. Thanks, Matt > Thanks > Olivier Bonnefon > > > > On 11/06/2014 07:14 AM, Matthew Knepley wrote: > > On Mon, Nov 3, 2014 at 9:46 AM, Olivier Bonnefon < > olivier.bonnefon at avignon.inra.fr> wrote: > >> Hello, >> >> Thank for your answer, I'll explain my trouble: >> >> My problem is that the BC Neumann leads to wrong result. >> >> With Dirichlet BC, I get: >> ------------------------------- >> >> > ./ex12 -run_type full -refinement_limit 0.0 -bc_type dirichlet >> -interpolate 0 -petscspace_order 1 -show_initial -show_solution >> -dm_plex_print_fem 1 >> ... >> ... >> Solution >> Vec Object:potential 1 MPI processes >> type: seq >> 0.5 >> >> This result is correct. >> >> With Neuman BC, I get: >> -------------------------------- >> >> > ./ex12 -run_type full -refinement_limit 0.0 -bc_type neumann >> -interpolate 1 -petscspace_order 1 -show_initial -dm_plex_print_fem 1 >> -show_solution -bd_petscspace_order 1 -snes_linesearch_monitor >> -snes_monitor -ksp_monitor_true_residual -snes_converged_reason >> -ksp_converged_reason >> .... >> .... >> >> >> Solution >> Vec Object:potential 1 MPI processes >> type: seq >> -0.75 >> -0.583333 >> 0.0833333 >> -0.583333 >> -0.333333 >> 0.416667 >> 0.0833333 >> 0.416667 >> 1.25 >> >> >> That is not the values of the solution x*x+y*y. >> > > Sorry, this is poor documentation on my part. I will fix it in the > example. The Naumann solution > explicitly discards the constant part, meaning > > \int_\Omega u = 0 > > If we look at my solution > > \int^1_0 dx \int^1_0 dy x^2 + y^2 = 2/3 > > However, this is for the continuum result, whereas we are enforcing this > in the discrete case. > Thus, what we really get is 0.75, since we are integrating linear patches > instead of quadratic > patches. After shifting by that, you still do not get exactly x^2 + y^2 > since there is some > discretization error. If you run with P2 you should get the exact answer, > shifted down to eliminate > the DC component. > > Thanks, > > Matt > > >> >> I tried many ksp options. >> Moreover, the neumann BC with "-run_type full" is not cover in the list >> https://bitbucket.org/petsc/petsc/src/fced3c3f9e703542693913793d15321603e40fe6/config/builder.py?at=master#cl-257 >> >> Do you know what is wrong ? >> >> Thanks, >> >> Olivier Bonnefon >> >> On 10/31/2014 06:50 PM, Matthew Knepley wrote: >> >> On Fri, Oct 31, 2014 at 10:43 AM, Olivier Bonnefon < >> olivier.bonnefon at avignon.inra.fr> wrote: >> >>> Hello, >>> >>> I'm working on the snes/examples/tutorial/ex12 version 3.5.2. >>> >>> I didn't succed to run the simplest case with Neumann BC: >>> >>> ./ex12 -run_type full -refinement_limit 0.0 -bc_type neumann >>> -interpolate 1 -petscspace_order 1 -show_initial -dm_plex_print_fem 1 >>> -show_solution -bd_petscspace_order 1 -snes_linesearch_monitor >>> -snes_monitor -ksp_monitor_true_residual -snes_converged_reason >>> -ksp_converged_reason >>> >>> This leads to dofs negatives values. >>> >> >> I do not understand what you mean. Please always mail the full error >> message. >> >> >>> Do you know the options to get a correct result with Neumann BC ? >>> >> >> There are some tests here: >> >> >> https://bitbucket.org/petsc/petsc/src/fced3c3f9e703542693913793d15321603e40fe6/config/builder.py?at=master#cl-257 >> >> Matt >> >> >>> Regards, >>> Olivier Bonnefon >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> >> -- >> Olivier Bonnefon >> INRA PACA-Avignon, Unit? BioSP >> Tel: +33 (0)4 32 72 21 58 >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Olivier Bonnefon > INRA PACA-Avignon, Unit? BioSP > Tel: +33 (0)4 32 72 21 58 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From kvoronin at labchem.sscc.ru Tue Nov 25 09:07:34 2014 From: kvoronin at labchem.sscc.ru (Kirill Voronin) Date: Tue, 25 Nov 2014 22:07:34 +0700 Subject: [petsc-users] Creating SBAIJ matrix using 1-based csr arrays ia, ja and acsr In-Reply-To: <325d812eefd8d1072b2f8a0ee80a4731.squirrel@mx.sscc.ru> Message-ID: Hello! I am trying to create a matrix for solving Ax = b with PETSC with given 1-based(!) indexed ia, ja and acsr arrays for a symmetric (only upper part is stored) complex matrix. Without memory preallocation it takes too long to fill in all the values of acsr into my Mat A object. But I failed to call ierr = MatSeqSBAIJSetPreallocationCSR(Afromcsr, dim, iaHSS, jaHSS, acHSS); because of the 1 based indexing. And I didn't find an option how to handle this issue. >From the manual: "By default the internal data representation for the AIJ formats employs zero-based indexing. For compatibility with standard Fortran storage, thus enabling use of external Fortran software packages such as SPARSKIT, the option -mat_aij_oneindex enables one-based indexing, where the stored row and column indices begin at one, not zero. All user calls to PETSc routines, regardless of this option, use zero-based indexing." So, what can you recommend to do? I obviously don't want to double the memory for storing ia and ja arrays in 0-based indexing. I don't want to change the indexing everywhere to 0-based since I use a piece of external code which works only for 1-based indexing as a preconditioner in PETSC KSP solver. Thanks in advance! With best regards, Kirill Voronin From jroman at dsic.upv.es Tue Nov 25 10:32:52 2014 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 25 Nov 2014 17:32:52 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: <51A03B32-245C-48FC-AE93-DC2A1F4427FA@gmail.com> References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> <98D02121-2E62-47F8-8841-F9D89D1B87DE@gmail.com> <396D16B5-D8C1-4180-A8B5-BC1F65A9C9D5@dsic.upv.es> <386CA68B-2577-462C-A16D-F0A1F4F445B0@gmail.com> <3BE6DE10-465D-4D53-8D81-207C426F5502@dsic.upv.es> <51A03B32-245C-48FC-AE93-DC2A1F4427FA@gmail.com> Message-ID: <5F8A7816-6425-4273-ABF2-5F793C38A797@dsic.upv.es> El 25/11/2014, a las 11:48, Denis Davydov escribi?: > Hi Jose, > > A follow up question on KrylovSchur solver: > >>> One last thing, if I force EPSSetTrueResidual(eps, PETSC_TRUE) >>> will that guarantee that EPSComputeRelativeError() >>> will give the norm consistent to that, used internally by all SLEPc solvers? >> >> I would not recommend that, since it is not implemented in all solvers. >> >> I tried with your matrices. In your case, it is much better to compute eigenvalues close to the origin with shift-and-invert, rather than computing smallest_real eigenvalues. That is, replace >> >> -eps_smallest_real >> >> with >> >> -eps_target 0 -st_type sinvert > > 1) I tried forcing -eps_true_residual (without sinvert) but have only 2 eigenpairs converged. > Same happens with sinvert around 0.0 you recommended above. > I would expect more iterations to be required for convergence rather than the solver diverging completely... sinvert should give you more that 2 eigenvalues if you set eps_nev > 2. You may need to increase the number of iterations with eps_max_it > > 2) Another peculiarity is that `-eps_smallest_real` and `-eps_target 0 -st_type sinvert` return different sets of eigenvalues. The matrix from the 2D Laplace operator is positive definite, so both should give the same eigenvalues. > In the latter case there are degenerate eigenvalues. > Those are consistent with the results given by ARPACK with shift and invert around 0. > The matrices I sent you originally correspond to the eigenvalues of the 2D Laplace operator on a uniform mesh with 256 cells. > If more refined mesh is used (1024 cells instead of 256), same set with degenerate eigenvalues is returned in both cases. > This is not directly related to SLEPc and is a question out of curiosity. Some eigenvalues of the 2D Laplacian have multiplicity 2. The default SLEPc solver may not return the two copies of the eigenvalue, because the second copy appears much later and the iteration stops as soon as nev eigenvalues have been computed. > > 3) It is not stated in the documentation explicitly, but I suppose the residual discussed in ?SolverControl? section > always corresponds to the direct problem (even in case when, say, shift-and-invert is applied) and so > EPSComputeRelativeError and EPSComputeResidualNorm? The convergence criterion is applied to the transformed problem (shift-and-invert). Jose > > Sincerely, > Denis > From knepley at gmail.com Tue Nov 25 10:35:20 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Nov 2014 10:35:20 -0600 Subject: [petsc-users] Creating SBAIJ matrix using 1-based csr arrays ia, ja and acsr In-Reply-To: References: <325d812eefd8d1072b2f8a0ee80a4731.squirrel@mx.sscc.ru> Message-ID: On Tue, Nov 25, 2014 at 9:07 AM, Kirill Voronin wrote: > > Hello! > > I am trying to create a matrix for solving Ax = b with PETSC with given > 1-based(!) indexed ia, ja and acsr arrays for a symmetric (only upper part > is stored) complex matrix. > > Without memory preallocation it takes too long to fill in all the values > of acsr into my Mat A object. > > But I failed to call > ierr = MatSeqSBAIJSetPreallocationCSR(Afromcsr, dim, iaHSS, jaHSS, acHSS); > because of the 1 based indexing. And I didn't find an option how to handle > this issue. > > From the manual: > "By default the internal data representation for the AIJ formats employs > zero-based indexing. For compatibility > with standard Fortran storage, thus enabling use of external Fortran > software packages such as > SPARSKIT, the option -mat_aij_oneindex enables one-based indexing, where > the stored row and > column indices begin at one, not zero. All user calls to PETSc routines, > regardless of this option, use > zero-based indexing." > > So, what can you recommend to do? I obviously don't want to double the > memory for storing ia and ja arrays in 0-based indexing. I don't want to > change the indexing everywhere to 0-based since I use a piece of external > code which works only for 1-based indexing as a preconditioner in PETSC > KSP solver. > Just decrement ja, make the call, and increment ja. Thanks, Matt > Thanks in advance! > > With best regards, > > Kirill Voronin > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bisheshkh at gmail.com Tue Nov 25 11:36:22 2014 From: bisheshkh at gmail.com (Bishesh Khanal) Date: Tue, 25 Nov 2014 18:36:22 +0100 Subject: [petsc-users] interpolate staggered grid values in parallel Message-ID: Dear all, I'm solving a system using petsc and I get a global Vec, say X . It uses DMDA with 4 dofs (3 velocity + 1 pressure). X contains velocity at the cell faces since I solve using staggered grid. Now I'd like to create one array for velocity with values at cell centers and another array for pressure (not the Vec so that I can send the pointer to the array to other part of the code that does not use Petsc). Currently what I do is : ------ Vec X, X_local; ------ PetscScalar *X_array; // Scatter X to X_local and then use: ------- VecGetArray(X_local, &X_array) And then have a function, say getVelocityAt(x, y, z, component) { // interpolate velocity at (x,y,z) cell center using X_array } The function getVelocityAt() gets called from outside petsc in a loop over all (x,y,z) positions. This is not done in parallel. Now, how do I use Petsc to instead interpolate the cell center velocities in parallel and store it in an array say PetscScalar *X_array_cellCenter; ? This would need to have size one less along each axis compared to the orginal DMDA size. This way I intend to return X_array_cellCenter to the code outside Petsc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 25 11:40:03 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Nov 2014 11:40:03 -0600 Subject: [petsc-users] interpolate staggered grid values in parallel In-Reply-To: References: Message-ID: On Tue, Nov 25, 2014 at 11:36 AM, Bishesh Khanal wrote: > Dear all, > I'm solving a system using petsc and I get a global Vec, say X . It uses > DMDA with 4 dofs (3 velocity + 1 pressure). X contains velocity at the cell > faces since I solve using staggered grid. > Now I'd like to create one array for velocity with values at cell centers > and another array for pressure (not the Vec so that I can send the pointer > to the array to other part of the code that does not use Petsc). > > Currently what I do is : > ------ Vec X, X_local; > ------ PetscScalar *X_array; > > // Scatter X to X_local and then use: > > ------- VecGetArray(X_local, &X_array) > > And then have a function, say > getVelocityAt(x, y, z, component) { > // interpolate velocity at (x,y,z) cell center using X_array > } > > The function getVelocityAt() gets called from outside petsc in a loop over > all (x,y,z) positions. > This is not done in parallel. > > Now, how do I use Petsc to instead interpolate the cell center velocities > in parallel and store it > in an array say > PetscScalar *X_array_cellCenter; > ? > This would need to have size one less along each axis compared to the > orginal DMDA size. > This way I intend to return X_array_cellCenter to the code outside Petsc. > SNES ex30 is an example of a staggered grid code using DMDA. It does this kind of interpolation, and puts the result in a Vec. Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 25 11:40:58 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Nov 2014 11:40:58 -0600 Subject: [petsc-users] Linear Solver Problem In-Reply-To: References: Message-ID: On Mon, Nov 24, 2014 at 4:07 PM, Sharp Stone wrote: > Dear all, > > I encountered a weird results from petsc linear solver. I have output the > same matrix and RHS vector from the linear system, and solve it also with > matlab. When dof is 1, both solvers from Matlab and Petsc can give me the > same results, However, Matlab can give me the right result but petsc does > not when dof=4. > > I attached the matrix and RHS vectors for dof=4 case. I don't know where I > made wrong, or else. Thanks a million in advance! > Solve it with LU to check first, -pc_type lu. Matt > -- > Best regards, > > Feng > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 25 11:47:43 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Nov 2014 11:47:43 -0600 Subject: [petsc-users] Problem with PETSc + HDF5 VecView In-Reply-To: <547382AA.10705@hakostra.net> References: <547382AA.10705@hakostra.net> Message-ID: On Mon, Nov 24, 2014 at 1:10 PM, H?kon Strandenes wrote: > Hi, > > I have some problems with PETSc and HDF5 VecLoad/VecView. The VecLoad > problems can rest for now, but the VecView are more serious. > > In short: I have a 3D DMDA with and some vectors that I want to save to a > HDF5 file. This works perfectly on my workstation, but not on the compute > cluster I have access to. I have attached a typical error message. > > I have also attached an piece of code that can trigger the error. The code > is merely a 2D->3D rewrite of DMDA ex 10 (http://www.mcs.anl.gov/petsc/ > petsc-current/src/dm/examples/tutorials/ex10.c.html), nothing else is > done. > > The program typically works on small number of processes. I have > successfully executed the attached program on up to 32 processes. That > works. Always. I have never had a single success when trying to run on 64 > processes. Always same error. > > The computer I am struggling with is an SGI machine with SLES 11sp1 and > Intel CPUs, hence I have used Intels compilers. I have tried both 2013, > 2014 and 2015 versions of the compilers, so that's probably not the cause. > I have also tried GCC 4.9.1, just to be safe, same error there. The same > compiler is used for both HDF5 and PETSc. The same error message occurs for > both debug and release builds. I have tried HDF5 versions 1.8.11 and > 1.8.13. I have tried PETSc version 3.4.1 and the latest from Git. The MPI > implementation on the machine is SGI's MPT, and i have tried both 2.06 and > 2.10. Always same error. Other MPI implementations is unfortunately not > available. > > What really drives me mad is that this works like a charm on my > workstation with Linux Mint... I have successfully executed the attached > example on 254 processes (my machine breaks down if I try anything more > than that). > > Does any of you have any tips on how to attack this problem and find out > what's wrong? > This does sound like a pain to track down. It seems to be complaining about an MPI datatype: #005: H5Dmpio.c line 998 in H5D__link_chunk_collective_io(): MPI_Type_struct failed major: Internal error (too specific to document in detail) minor: Some MPI function failed #006: H5Dmpio.c line 998 in H5D__link_chunk_collective_io(): Invalid datatype argument major: Internal error (too specific to document in detail) minor: MPI Error String In this call, we pass in 'scalartype', which is H5T_NATIVE_DOUBLE (unless you configured for single precision). This was used successfully to create the dataspace, so it is unlikely to be the problem. I am guessing that HDF5 creates internal MPI datatypes to use in the MPI/IO routines (maybe using MPI_Type_struct). I believe we have seen type creation routines fail in some MPI implementations if you try to create too many of them. Right now, this looks a lot like a bug in MPT, although it might be an HDF5 bug with forgetting to release MPI types that they do not need. Thanks, Matt > Regards, > H?kon Strandenes > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From kvoronin at labchem.sscc.ru Tue Nov 25 12:54:36 2014 From: kvoronin at labchem.sscc.ru (Kirill Voronin) Date: Wed, 26 Nov 2014 01:54:36 +0700 Subject: [petsc-users] Creating SBAIJ matrix using 1-based csr arrays ia, ja and acsr In-Reply-To: Message-ID: <2ae5140c56d0fa7cc4a36e221df9fc0d.squirrel@mx.sscc.ru> Thank you for the prompt answer! I shifted the indices but when calling MatSeqSBAIJSetPreallocationCSR got the error - nnz cannot be greater than block row length: local row 0 value 8 rowlength 1. Can anyone please make a bit more clear some details of block AIJ format in PETSC? My code looks like: ----------------------------------------------------- ierr = MatCreate(comm,&Afromcsr);CHKERRQ(ierr); ierr = MatSetType(Afromcsr,"sbaij"); CHKERRQ(ierr); ierr = MatSetSizes(Afromcsr,dim,dim,dim,dim);CHKERRQ(ierr); ierr = MatSetFromOptions(Afromcsr);CHKERRQ(ierr); printf("fill in for values of acsr \n"); for (i = 0; i <= dim; i++) ia[i] -= 1; for (i = 0; i < ia[dim]; i++) ja[i] -= 1; ierr = MatSeqSBAIJSetPreallocationCSR(Afromcsr, dim, ia, ja, acsr); CHKERRQ(ierr); ----------------------------------------------------------- The questions: 1) Why no MatMPISBAIJSetPreaalocationCSR? Or should MatCreateMPISBAIJWithArrays be used instead? 2) What is block size and why it is assumed to be 1 (according to the error I got). 3) What is a local row when I call MatSeqSBAIJSetPreallocationCSR? 4) Can I consider my matrix as a single block when calling MatSeqSBAIJSetPreallocationCSR ? 5) Can I put my acsr array (with values of nonzero elements of the matrix) as the argument v[] in the MatSeqSBAIJSetPreallocationCSR? The manual pages told me that v[] should be of the form v[nnz][bs][bs] and I don't get why not only v[nnz]? Sorry for that set of questions but I missed an example of usage for a symmetric CSR matrix. Maybe a few lines of code for getting a symmetric (with upper part only stored) matrix from ia, ja and a arrays (usual CSR format) would help me to get the whole idea of PETSC matrix creating. Thanks in advance! Best regards, Kirill Voronin > On Tue, Nov 25, 2014 at 9:07 AM, Kirill Voronin > > wrote: > > >> >> Hello! >> >> >> I am trying to create a matrix for solving Ax = b with PETSC with given >> 1-based(!) indexed ia, ja and acsr arrays for a symmetric (only upper >> part is stored) complex matrix. >> >> Without memory preallocation it takes too long to fill in all the >> values of acsr into my Mat A object. >> >> But I failed to call >> ierr = MatSeqSBAIJSetPreallocationCSR(Afromcsr, dim, iaHSS, jaHSS, >> acHSS); because of the 1 based indexing. And I didn't find an option how >> to handle this issue. >> >> From the manual: >> "By default the internal data representation for the AIJ formats employs >> zero-based indexing. For compatibility with standard Fortran storage, >> thus enabling use of external Fortran software packages such as SPARSKIT, >> the option -mat_aij_oneindex enables one-based indexing, where the >> stored row and column indices begin at one, not zero. All user calls to >> PETSc routines, >> regardless of this option, use zero-based indexing." >> >> So, what can you recommend to do? I obviously don't want to double the >> memory for storing ia and ja arrays in 0-based indexing. I don't want to >> change the indexing everywhere to 0-based since I use a piece of >> external code which works only for 1-based indexing as a preconditioner >> in PETSC KSP solver. >> >> > > Just decrement ja, make the call, and increment ja. > > > Thanks, > > > Matt > > > >> Thanks in advance! >> >> >> With best regards, >> >> >> Kirill Voronin >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. -- Norbert Wiener > > -- From knepley at gmail.com Tue Nov 25 14:16:58 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Nov 2014 14:16:58 -0600 Subject: [petsc-users] Creating SBAIJ matrix using 1-based csr arrays ia, ja and acsr In-Reply-To: <2ae5140c56d0fa7cc4a36e221df9fc0d.squirrel@mx.sscc.ru> References: <2ae5140c56d0fa7cc4a36e221df9fc0d.squirrel@mx.sscc.ru> Message-ID: On Tue, Nov 25, 2014 at 12:54 PM, Kirill Voronin wrote: > > Thank you for the prompt answer! > > I shifted the indices but when calling MatSeqSBAIJSetPreallocationCSR got > the error - nnz cannot be greater than block row length: local row 0 value > 8 rowlength 1. > It sounds like the input you are giving is for actual rows, but this call expects the CSR to reference blocks. Does that make sense? > Can anyone please make a bit more clear some details of block AIJ format > in PETSC? > > My code looks like: > ----------------------------------------------------- > ierr = MatCreate(comm,&Afromcsr);CHKERRQ(ierr); > ierr = MatSetType(Afromcsr,"sbaij"); CHKERRQ(ierr); > ierr = MatSetSizes(Afromcsr,dim,dim,dim,dim);CHKERRQ(ierr); > ierr = MatSetFromOptions(Afromcsr);CHKERRQ(ierr); > > printf("fill in for values of acsr \n"); > > for (i = 0; i <= dim; i++) > ia[i] -= 1; > for (i = 0; i < ia[dim]; i++) > ja[i] -= 1; > ierr = MatSeqSBAIJSetPreallocationCSR(Afromcsr, dim, ia, ja, acsr); > CHKERRQ(ierr); > ----------------------------------------------------------- > The questions: > > 1) Why no MatMPISBAIJSetPreaalocationCSR? Or should > MatCreateMPISBAIJWithArrays be used instead? > Not written yet. > 2) What is block size and why it is assumed to be 1 (according to the > error I got). > Block size is the size of the dense blocks out of which the matrix is constructed. This can achieve better performance since indexing is only done on blocks, not individual elements, and dense algebra can be done on block-block computation. > 3) What is a local row when I call MatSeqSBAIJSetPreallocationCSR? > Its a row of blocks. > 4) Can I consider my matrix as a single block when calling > MatSeqSBAIJSetPreallocationCSR ? > The maximum block size is 13 I think. > 5) Can I put my acsr array (with values of nonzero elements of the matrix) > as the argument v[] in the MatSeqSBAIJSetPreallocationCSR? The manual > pages told me that v[] should be of the form v[nnz][bs][bs] and I don't > get why not only v[nnz]? > There is no v[] argument: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSeqSBAIJSetPreallocation.html Thanks, Matt > Sorry for that set of questions but I missed an example of usage for a > symmetric CSR matrix. > > Maybe a few lines of code for getting a symmetric (with upper part only > stored) matrix from ia, ja and a arrays (usual CSR format) would help me > to get the whole idea of PETSC matrix creating. > > Thanks in advance! > > > Best regards, > > Kirill Voronin > > > > On Tue, Nov 25, 2014 at 9:07 AM, Kirill Voronin > > > > wrote: > > > > > >> > >> Hello! > >> > >> > >> I am trying to create a matrix for solving Ax = b with PETSC with given > >> 1-based(!) indexed ia, ja and acsr arrays for a symmetric (only upper > >> part is stored) complex matrix. > >> > >> Without memory preallocation it takes too long to fill in all the > >> values of acsr into my Mat A object. > >> > >> But I failed to call > >> ierr = MatSeqSBAIJSetPreallocationCSR(Afromcsr, dim, iaHSS, jaHSS, > >> acHSS); because of the 1 based indexing. And I didn't find an option how > >> to handle this issue. > >> > >> From the manual: > >> "By default the internal data representation for the AIJ formats employs > >> zero-based indexing. For compatibility with standard Fortran storage, > >> thus enabling use of external Fortran software packages such as > SPARSKIT, > >> the option -mat_aij_oneindex enables one-based indexing, where the > >> stored row and column indices begin at one, not zero. All user calls to > >> PETSc routines, > >> regardless of this option, use zero-based indexing." > >> > >> So, what can you recommend to do? I obviously don't want to double the > >> memory for storing ia and ja arrays in 0-based indexing. I don't want to > >> change the indexing everywhere to 0-based since I use a piece of > >> external code which works only for 1-based indexing as a preconditioner > >> in PETSC KSP solver. > >> > >> > > > > Just decrement ja, make the call, and increment ja. > > > > > > Thanks, > > > > > > Matt > > > > > > > >> Thanks in advance! > >> > >> > >> With best regards, > >> > >> > >> Kirill Voronin > >> > >> > >> > >> > >> > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > their > > experiments lead. -- Norbert Wiener > > > > > > > -- > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From haakon at hakostra.net Tue Nov 25 14:34:00 2014 From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=) Date: Tue, 25 Nov 2014 21:34:00 +0100 Subject: [petsc-users] Problem with PETSc + HDF5 VecView In-Reply-To: References: <547382AA.10705@hakostra.net> Message-ID: <5474E7B8.1090108@hakostra.net> I have done a whole lot of debugging on this issue. In my first post I wrote that "The VecLoad problems can rest for now, but the VecView are more serious.". I now believe that these problems in essence are the same. I have however not yet found out where the problem is, in PETSc, HDF5, MPT or (perhaps unlikely) the Lustre file system. The VecLoad bug is as follows: I have a properly formatted HDF5-file with data corresponding to a vector on a DMDA grid. If I try to load this data from the HDF5-file and into a Vec, it works with no error messages. The data is however severely corrupted. This is shown in the attached figure fig-COLELCTIVE.png, where the left column of figures is the three components of a vector that actually is in the HDF5 file, while the right column is how PETSc's VecLoad() reads the data. This bug does not cause any error messages what so ever, and is again dependent on the decomposition pattern. Another strange phenomena related to VecLoad() is that the occurrence of this bug seems to be dependent on the chunking in the dataset being loaded. Different chinking patterns seems to produce different results. This might be an important lead. My workaround is to replace the two occurrences of H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_COLLECTIVE) with H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_INDEPENDENT) in file src/dm/impls/da/gr2.c These two occurrences are found in the functions VecView_MPI_HDF5_DA() and VecLoad_HDF5_DA(). This solves all problems (at least so far). Both VecLoad() and VecView() works as expected. See attached figure fig-INDEPENDENT.png. The left column is again input and the right column is output. I have actually made a small example program that performs two tasks: loads a Vec from a HDF5-file and writes the same Vec back again to a new file. When comparing the data inside these files, it is exactly identical for all combinations of decomposition, number of processes and chunking patterns in the input file I have tried. This leads me to an important question I cannot answer because I don't have enough knowledge and insight in HDF5: Is PETSc using the collective/independent flags correctly? Can you use collective IO in all cases for all DMDA's and all variants of decomposition? The following document describe some features of parallel HDF5: http://www.hdfgroup.org/HDF5/PHDF5/parallelhdf5hints.pdf Does PETSc currently satisfy the conditions for using collective IO and chunking as described on page 5-6? Perhaps the reason why this is not occurring on my workstation is that HDF5 recognise that my file system is serial and falls back to some simple serial IO routines? My cluster is equipped with a parallel Lustre file system, and as far as I know HDF5 handles these differently. The same document also mentions that HDF5 in deed does create some internal datatypes to accomplish collective IO on chunked storage (top of page 6), as Matthew suggested. Any comments? Tanks for your time. Best regards, H?kon Strandenes On 25. nov. 2014 18:47, Matthew Knepley wrote: > On Mon, Nov 24, 2014 at 1:10 PM, H?kon Strandenes > wrote: > > Hi, > > I have some problems with PETSc and HDF5 VecLoad/VecView. The > VecLoad problems can rest for now, but the VecView are more serious. > > In short: I have a 3D DMDA with and some vectors that I want to save > to a HDF5 file. This works perfectly on my workstation, but not on > the compute cluster I have access to. I have attached a typical > error message. > > I have also attached an piece of code that can trigger the error. > The code is merely a 2D->3D rewrite of DMDA ex 10 > (http://www.mcs.anl.gov/petsc/__petsc-current/src/dm/examples/__tutorials/ex10.c.html > ), > nothing else is done. > > The program typically works on small number of processes. I have > successfully executed the attached program on up to 32 processes. > That works. Always. I have never had a single success when trying to > run on 64 processes. Always same error. > > The computer I am struggling with is an SGI machine with SLES 11sp1 > and Intel CPUs, hence I have used Intels compilers. I have tried > both 2013, 2014 and 2015 versions of the compilers, so that's > probably not the cause. I have also tried GCC 4.9.1, just to be > safe, same error there. The same compiler is used for both HDF5 and > PETSc. The same error message occurs for both debug and release > builds. I have tried HDF5 versions 1.8.11 and 1.8.13. I have tried > PETSc version 3.4.1 and the latest from Git. The MPI implementation > on the machine is SGI's MPT, and i have tried both 2.06 and 2.10. > Always same error. Other MPI implementations is unfortunately not > available. > > What really drives me mad is that this works like a charm on my > workstation with Linux Mint... I have successfully executed the > attached example on 254 processes (my machine breaks down if I try > anything more than that). > > Does any of you have any tips on how to attack this problem and find > out what's wrong? > > > This does sound like a pain to track down. It seems to be complaining > about an MPI datatype: > > #005: H5Dmpio.c line 998 in H5D__link_chunk_collective_io(): > MPI_Type_struct failed > major: Internal error (too specific to document in detail) > minor: Some MPI function failed > #006: H5Dmpio.c line 998 in H5D__link_chunk_collective_io(): Invalid > datatype argument > major: Internal error (too specific to document in detail) > minor: MPI Error String > > In this call, we pass in 'scalartype', which is H5T_NATIVE_DOUBLE > (unless you configured for > single precision). This was used successfully to create the dataspace, > so it is unlikely to be > the problem. I am guessing that HDF5 creates internal MPI datatypes to > use in the MPI/IO > routines (maybe using MPI_Type_struct). > > I believe we have seen type creation routines fail in some MPI > implementations if you try to > create too many of them. Right now, this looks a lot like a bug in MPT, > although it might be > an HDF5 bug with forgetting to release MPI types that they do not need. > > Thanks, > > Matt > > Regards, > H?kon Strandenes > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- A non-text attachment was scrubbed... Name: fig-COLLECTIVE.png Type: image/png Size: 454244 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fig-INDEPENDENT.png Type: image/png Size: 330622 bytes Desc: not available URL: From orxan.shibli at gmail.com Tue Nov 25 15:25:41 2014 From: orxan.shibli at gmail.com (Orxan Shibliyev) Date: Tue, 25 Nov 2014 14:25:41 -0700 Subject: [petsc-users] Viewing solution of unstructured grid Message-ID: Hi I have an unstructured grid formed with DMPLEX. There are several DOFs associated with the grid such as density, pressure and so on. I would like to view the densities (preferably with Tecplot) on each node. How can I do that? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Nov 25 15:35:22 2014 From: jed at jedbrown.org (Jed Brown) Date: Tue, 25 Nov 2014 15:35:22 -0600 Subject: [petsc-users] Viewing solution of unstructured grid In-Reply-To: References: Message-ID: <87mw7f55gl.fsf@jedbrown.org> Orxan Shibliyev writes: > Hi > > I have an unstructured grid formed with DMPLEX. There are several DOFs > associated with the grid such as density, pressure and so on. I would like > to view the densities (preferably with Tecplot) on each node. How can I do > that? View with PETSCVIEWERVTK as a *.vtu file, then load into Tecplot, VisIt, Paraview, etc. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From knepley at gmail.com Tue Nov 25 15:40:53 2014 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Nov 2014 15:40:53 -0600 Subject: [petsc-users] Problem with PETSc + HDF5 VecView In-Reply-To: <5474E7B8.1090108@hakostra.net> References: <547382AA.10705@hakostra.net> <5474E7B8.1090108@hakostra.net> Message-ID: On Tue, Nov 25, 2014 at 2:34 PM, H?kon Strandenes wrote: > I have done a whole lot of debugging on this issue. In my first post I > wrote that "The VecLoad problems can rest for now, but the VecView are more > serious.". I now believe that these problems in essence are the same. I > have however not yet found out where the problem is, in PETSc, HDF5, MPT or > (perhaps unlikely) the Lustre file system. > > The VecLoad bug is as follows: I have a properly formatted HDF5-file with > data corresponding to a vector on a DMDA grid. If I try to load this data > from the HDF5-file and into a Vec, it works with no error messages. The > data is however severely corrupted. This is shown in the attached figure > fig-COLELCTIVE.png, where the left column of figures is the three > components of a vector that actually is in the HDF5 file, while the right > column is how PETSc's VecLoad() reads the data. This bug does not cause any > error messages what so ever, and is again dependent on the decomposition > pattern. > > Another strange phenomena related to VecLoad() is that the occurrence of > this bug seems to be dependent on the chunking in the dataset being loaded. > Different chinking patterns seems to produce different results. This might > be an important lead. > > My workaround is to replace the two occurrences of > H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_COLLECTIVE) > with > H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_INDEPENDENT) > in file src/dm/impls/da/gr2.c These two occurrences are found in the > functions VecView_MPI_HDF5_DA() and VecLoad_HDF5_DA(). > > This solves all problems (at least so far). Both VecLoad() and VecView() > works as expected. See attached figure fig-INDEPENDENT.png. The left column > is again input and the right column is output. I have actually made a small > example program that performs two tasks: loads a Vec from a HDF5-file and > writes the same Vec back again to a new file. When comparing the data > inside these files, it is exactly identical for all combinations of > decomposition, number of processes and chunking patterns in the input file > I have tried. > > This leads me to an important question I cannot answer because I don't > have enough knowledge and insight in HDF5: Is PETSc using the > collective/independent flags correctly? Can you use collective IO in all > cases for all DMDA's and all variants of decomposition? > > The following document describe some features of parallel HDF5: > http://www.hdfgroup.org/HDF5/PHDF5/parallelhdf5hints.pdf > > Does PETSc currently satisfy the conditions for using collective IO and > chunking as described on page 5-6? Perhaps the reason why this is not > occurring on my workstation is that HDF5 recognise that my file system is > serial and falls back to some simple serial IO routines? My cluster is > equipped with a parallel Lustre file system, and as far as I know HDF5 > handles these differently. The same document also mentions that HDF5 in > deed does create some internal datatypes to accomplish collective IO on > chunked storage (top of page 6), as Matthew suggested. > > Any comments? > First, this is great debugging. Second, my reading of the HDF5 document you linked to says that either selection should be valid: "For non-regular hyperslab selection, parallel HDF5 uses independent IO internally for this option." so it ought to fall back to the INDEPENDENT model if it can't do collective calls correctly. However, it appears that the collective call has bugs. My conclusion: Since you have determined that changing the setting to INDEPENDENT produces correct input/output in all the test cases, and since my understanding of the HDF5 documentation is that we should always be able to use COLLECTIVE as an option, this is an HDF5 or MPT bug. Does anyone else see the HDF5 differently? Also, it really looks to me like HDF5 messed up the MPI data type in the COLLECTIVE picture below, since it appears to be sliced incorrectly. Possible Remedies: 1) We can allow you to turn off H5Pset_dxpl_mpio() 2) Send this test case to the MPI/IO people at ANL If you think 1) is what you want, we can do it. If you can package this work for 2), it would be really valuable. Thanks, Matt Tanks for your time. > > Best regards, > H?kon Strandenes > > > On 25. nov. 2014 18:47, Matthew Knepley wrote: > >> On Mon, Nov 24, 2014 at 1:10 PM, H?kon Strandenes > > wrote: >> >> Hi, >> >> I have some problems with PETSc and HDF5 VecLoad/VecView. The >> VecLoad problems can rest for now, but the VecView are more serious. >> >> In short: I have a 3D DMDA with and some vectors that I want to save >> to a HDF5 file. This works perfectly on my workstation, but not on >> the compute cluster I have access to. I have attached a typical >> error message. >> >> I have also attached an piece of code that can trigger the error. >> The code is merely a 2D->3D rewrite of DMDA ex 10 >> (http://www.mcs.anl.gov/petsc/__petsc-current/src/dm/ >> examples/__tutorials/ex10.c.html >> > tutorials/ex10.c.html>), >> >> nothing else is done. >> >> The program typically works on small number of processes. I have >> successfully executed the attached program on up to 32 processes. >> That works. Always. I have never had a single success when trying to >> run on 64 processes. Always same error. >> >> The computer I am struggling with is an SGI machine with SLES 11sp1 >> and Intel CPUs, hence I have used Intels compilers. I have tried >> both 2013, 2014 and 2015 versions of the compilers, so that's >> probably not the cause. I have also tried GCC 4.9.1, just to be >> safe, same error there. The same compiler is used for both HDF5 and >> PETSc. The same error message occurs for both debug and release >> builds. I have tried HDF5 versions 1.8.11 and 1.8.13. I have tried >> PETSc version 3.4.1 and the latest from Git. The MPI implementation >> on the machine is SGI's MPT, and i have tried both 2.06 and 2.10. >> Always same error. Other MPI implementations is unfortunately not >> available. >> >> What really drives me mad is that this works like a charm on my >> workstation with Linux Mint... I have successfully executed the >> attached example on 254 processes (my machine breaks down if I try >> anything more than that). >> >> Does any of you have any tips on how to attack this problem and find >> out what's wrong? >> >> >> This does sound like a pain to track down. It seems to be complaining >> about an MPI datatype: >> >> #005: H5Dmpio.c line 998 in H5D__link_chunk_collective_io(): >> MPI_Type_struct failed >> major: Internal error (too specific to document in detail) >> minor: Some MPI function failed >> #006: H5Dmpio.c line 998 in H5D__link_chunk_collective_io(): Invalid >> datatype argument >> major: Internal error (too specific to document in detail) >> minor: MPI Error String >> >> In this call, we pass in 'scalartype', which is H5T_NATIVE_DOUBLE >> (unless you configured for >> single precision). This was used successfully to create the dataspace, >> so it is unlikely to be >> the problem. I am guessing that HDF5 creates internal MPI datatypes to >> use in the MPI/IO >> routines (maybe using MPI_Type_struct). >> >> I believe we have seen type creation routines fail in some MPI >> implementations if you try to >> create too many of them. Right now, this looks a lot like a bug in MPT, >> although it might be >> an HDF5 bug with forgetting to release MPI types that they do not need. >> >> Thanks, >> >> Matt >> >> Regards, >> H?kon Strandenes >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From davydden at gmail.com Tue Nov 25 16:00:25 2014 From: davydden at gmail.com (Denis Davydov) Date: Tue, 25 Nov 2014 23:00:25 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: <5F8A7816-6425-4273-ABF2-5F793C38A797@dsic.upv.es> References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> <98D02121-2E62-47F8-8841-F9D89D1B87DE@gmail.com> <396D16B5-D8C1-4180-A8B5-BC1F65A9C9D5@dsic.upv.es> <386CA68B-2577-462C-A16D-F0A1F4F445B0@gmail.com> <3BE6DE10-465D-4D53-8D81-207C426F5502@dsic.upv.es> <51A03B32-245C-48FC-AE93-DC2A1F4427FA@gmail.com> <5F8A7816-6425-4273-ABF2-5F793C38A797@dsic.upv.es> Message-ID: > > sinvert should give you more that 2 eigenvalues if you set eps_nev > 2. You may need to increase the number of iterations with eps_max_it i would expect that it should, but it is not the case. I tried setting number of iterations 10 times the number of DoFs, and no change. It does not seem to be related. Here is an output without shift-and-invert EPS Object: 1 MPI processes type: krylovschur Krylov-Schur: 50% of basis vectors kept after restart problem type: generalized symmetric eigenvalue problem selected portion of the spectrum: smallest real parts computing true residuals explicitly number of eigenvalues (nev): 5 number of column vectors (ncv): 20 maximum dimension of projected problem (mpd): 20 maximum number of iterations: 2890 tolerance: 1e-07 convergence test: absolute BV Object: 1 MPI processes type: svec 21 columns of global length 289 orthogonalization method: classical Gram-Schmidt orthogonalization refinement: if needed (eta: 0.7071) non-standard inner product Mat Object: 1 MPI processes type: seqaij rows=289, cols=289 total: nonzeros=1913, allocated nonzeros=5491 total number of mallocs used during MatSetValues calls =0 not using I-node routines DS Object: 1 MPI processes type: hep solving the problem with: Implicit QR method (_steqr) ST Object: 1 MPI processes type: shift shift: 0 number of matrices: 2 all matrices have different nonzero pattern KSP Object: (st_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (st_) 1 MPI processes type: redundant Redundant preconditioner: First (color=0) of 1 PCs follows KSP Object: (st_redundant_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (st_redundant_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5, needed 2.99686 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=289, cols=289 package used to perform factorization: petsc total: nonzeros=5733, allocated nonzeros=5733 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=289, cols=289 total: nonzeros=1913, allocated nonzeros=5491 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=289, cols=289 total: nonzeros=1913, allocated nonzeros=5491 total number of mallocs used during MatSetValues calls =0 not using I-node routines > >> >> 2) Another peculiarity is that `-eps_smallest_real` and `-eps_target 0 -st_type sinvert` return different sets of eigenvalues. > > The matrix from the 2D Laplace operator is positive definite, so both should give the same eigenvalues. > >> In the latter case there are degenerate eigenvalues. >> Those are consistent with the results given by ARPACK with shift and invert around 0. >> The matrices I sent you originally correspond to the eigenvalues of the 2D Laplace operator on a uniform mesh with 256 cells. >> If more refined mesh is used (1024 cells instead of 256), same set with degenerate eigenvalues is returned in both cases. >> This is not directly related to SLEPc and is a question out of curiosity. > > Some eigenvalues of the 2D Laplacian have multiplicity 2. The default SLEPc solver may not return the two copies of the eigenvalue, because the second copy appears much later and the iteration stops as soon as nev eigenvalues have been computed. i see? so technically, one could try asking for more eigenvectors to resolve those degenerate eigenvalues? > >> >> 3) It is not stated in the documentation explicitly, but I suppose the residual discussed in ?SolverControl? section >> always corresponds to the direct problem (even in case when, say, shift-and-invert is applied) and so >> EPSComputeRelativeError and EPSComputeResidualNorm? > > The convergence criterion is applied to the transformed problem (shift-and-invert). understood, thanks. Regards, Denis. -------------- next part -------------- An HTML attachment was scrubbed... URL: From haakon at hakostra.net Wed Nov 26 01:01:04 2014 From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=) Date: Wed, 26 Nov 2014 08:01:04 +0100 Subject: [petsc-users] Problem with PETSc + HDF5 VecView In-Reply-To: References: <547382AA.10705@hakostra.net> <5474E7B8.1090108@hakostra.net> Message-ID: <54757AB0.6070100@hakostra.net> On 25. nov. 2014 22:40, Matthew Knepley wrote: > On Tue, Nov 25, 2014 at 2:34 PM, H?kon Strandenes > wrote: > > (...) > > First, this is great debugging. Thanks. > > Second, my reading of the HDF5 document you linked to says that either > selection should be valid: > > "For non-regular hyperslab selection, parallel HDF5 uses independent > IO internally for this option." > > so it ought to fall back to the INDEPENDENT model if it can't do > collective calls correctly. However, > it appears that the collective call has bugs. > > My conclusion: Since you have determined that changing the setting to > INDEPENDENT produces > correct input/output in all the test cases, and since my understanding > of the HDF5 documentation is > that we should always be able to use COLLECTIVE as an option, this is an > HDF5 or MPT bug. I have conducted yet another test: My example (ex10) that I previously posted to the mailing list was set up with 250 grid points along each axis. When the topic on chunking was brought to the table, I realized that 250 is not evenly dividable on four. The example failed on 64 processes, that is four processes along each direction (the division is 62 + 62 + 63 + 63 = 250). So I have recompiled "my ex10" with 256 gridpoints in each direction. It turns out that this does in deed run successfully on 64 nodes. Great! It also runs on 128 processes, that is a 8x4x4 decomposition. However it does not run on 125 processes, that is a 5x5x5 decomposition. The same pattern is clear if I run my example with 250^3 grid points. It does not run on numbers like 64 and 128, but does run successfully on 125 processes, again only when the sub-domains are of exactly equal size (in this case the domain is divided as 5x5x5). However, I still believe that there is bugs. I did my "roundtrip" by loading a dataset and immediately writing the same dataset to a different file, this time a 250^3 dataset on 125 processes. It did not "pass" this test, i.e. the written dataset was just garbage. I have not yet identified if the garbling is introduced in the reading or writing of the dataset. > > Does anyone else see the HDF5 differently? Also, it really looks to me > like HDF5 messed up the MPI > data type in the COLLECTIVE picture below, since it appears to be sliced > incorrectly. > > Possible Remedies: > > 1) We can allow you to turn off H5Pset_dxpl_mpio() > > 2) Send this test case to the MPI/IO people at ANL > > If you think 1) is what you want, we can do it. If you can package this > work for 2), it would be really valuable. I will be fine editing gr2.c manually each time this file is changed (I use the sources from Git). But *if* this not a bug in MPT, but a bug in PETSc or HDF5 it should be fixed... Because it is that kind of bug that is extremely annoying and a read pain to track down. Perhaps the HDF5 mailing list could contribute in this issue? > > Thanks, > > Matt > > Tanks for your time. > > Best regards, > H?kon Strandenes > > Again thanks for your time. Regards, H?kon > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From jroman at dsic.upv.es Wed Nov 26 05:45:26 2014 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 26 Nov 2014 12:45:26 +0100 Subject: [petsc-users] KrylovSchur solver diverges when setting EPS_GHEP In-Reply-To: References: <56283D60-A368-442F-A877-5680B57D2C96@gmail.com> <68A8CBE9-10FA-42F3-913D-4EF3847285CB@dsic.upv.es> <98D02121-2E62-47F8-8841-F9D89D1B87DE@gmail.com> <396D16B5-D8C1-4180-A8B5-BC1F65A9C9D5@dsic.upv.es> <386CA68B-2577-462C-A16D-F0A1F4F445B0@gmail.com> <3BE6DE10-465D-4D53-8D81-207C426F5502@dsic.upv.es> <51A03B32-245C-48FC-AE93-DC2A1F4427FA@gmail.com> <5F8A7816-6425-4273-ABF2-5F793C38A797@dsic.upv.es> Message-ID: <076C0201-DFF5-4783-8C1B-B6AC8D303EF6@dsic.upv.es> El 25/11/2014, a las 23:00, Denis Davydov escribi?: > >> >> sinvert should give you more that 2 eigenvalues if you set eps_nev > 2. You may need to increase the number of iterations with eps_max_it > > i would expect that it should, but it is not the case. I tried setting number of iterations 10 times the number of DoFs, and no change. > It does not seem to be related. Here is an output without shift-and-invert > > EPS Object: 1 MPI processes > type: krylovschur > Krylov-Schur: 50% of basis vectors kept after restart > problem type: generalized symmetric eigenvalue problem > selected portion of the spectrum: smallest real parts > computing true residuals explicitly > number of eigenvalues (nev): 5 > number of column vectors (ncv): 20 > maximum dimension of projected problem (mpd): 20 > maximum number of iterations: 2890 > tolerance: 1e-07 > convergence test: absolute > BV Object: 1 MPI processes > type: svec > 21 columns of global length 289 > orthogonalization method: classical Gram-Schmidt > orthogonalization refinement: if needed (eta: 0.7071) > non-standard inner product > Mat Object: 1 MPI processes > type: seqaij > rows=289, cols=289 > total: nonzeros=1913, allocated nonzeros=5491 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > DS Object: 1 MPI processes > type: hep > solving the problem with: Implicit QR method (_steqr) > ST Object: 1 MPI processes > type: shift > shift: 0 > number of matrices: 2 > all matrices have different nonzero pattern > KSP Object: (st_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-08, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (st_) 1 MPI processes > type: redundant > Redundant preconditioner: First (color=0) of 1 PCs follows > KSP Object: (st_redundant_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (st_redundant_) 1 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5, needed 2.99686 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=289, cols=289 > package used to perform factorization: petsc > total: nonzeros=5733, allocated nonzeros=5733 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=289, cols=289 > total: nonzeros=1913, allocated nonzeros=5491 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=289, cols=289 > total: nonzeros=1913, allocated nonzeros=5491 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines Once again, do not use the -eps_true_residual option. > >> >>> >>> 2) Another peculiarity is that `-eps_smallest_real` and `-eps_target 0 -st_type sinvert` return different sets of eigenvalues. >> >> The matrix from the 2D Laplace operator is positive definite, so both should give the same eigenvalues. >> >>> In the latter case there are degenerate eigenvalues. >>> Those are consistent with the results given by ARPACK with shift and invert around 0. >>> The matrices I sent you originally correspond to the eigenvalues of the 2D Laplace operator on a uniform mesh with 256 cells. >>> If more refined mesh is used (1024 cells instead of 256), same set with degenerate eigenvalues is returned in both cases. >>> This is not directly related to SLEPc and is a question out of curiosity. >> >> Some eigenvalues of the 2D Laplacian have multiplicity 2. The default SLEPc solver may not return the two copies of the eigenvalue, because the second copy appears much later and the iteration stops as soon as nev eigenvalues have been computed. > > i see? > > so technically, one could try asking for more eigenvectors to resolve those degenerate eigenvalues? Yes, or force more restarts by setting a small value of ncv or mpd. Jose > >> >>> >>> 3) It is not stated in the documentation explicitly, but I suppose the residual discussed in ?SolverControl? section >>> always corresponds to the direct problem (even in case when, say, shift-and-invert is applied) and so >>> EPSComputeRelativeError and EPSComputeResidualNorm? >> >> The convergence criterion is applied to the transformed problem (shift-and-invert). > > understood, thanks. > > > Regards, > Denis. From haakon at hakostra.net Wed Nov 26 06:26:16 2014 From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=) Date: Wed, 26 Nov 2014 13:26:16 +0100 Subject: [petsc-users] Problem with PETSc + HDF5 VecView In-Reply-To: <54757AB0.6070100@hakostra.net> References: <547382AA.10705@hakostra.net> <5474E7B8.1090108@hakostra.net> <54757AB0.6070100@hakostra.net> Message-ID: <5475C6E8.3090609@hakostra.net> My local HPC group have found a solution to this problem: On MPT it is possible to set an environment variable MPI_TYPE_DEPTH with default value 8. The MPI_TYPE_DEPTH variable limits the maximum depth of derived datatypes that an application can create. I have found that setting this to at least 32 will make my examples run perfectly on up to 256 processes. No error messages what so ever, and in my simple load and write dataset roundtrip h5diff compares the two datasets and finds then identical. I also notice that Leibniz Rechenzentrum recommend to set this variable to 100 (or some other suitably large value) when using NetCDF together with MPT (https://www.lrz.de/services/software/io/netcdf/). This bug have been a pain in the (***)... Perhaps it is worthy a FAQ entry? Thanks for your time and effort. Regards, H?kon Strandenes On 26. nov. 2014 08:01, H?kon Strandenes wrote: > > > On 25. nov. 2014 22:40, Matthew Knepley wrote: >> On Tue, Nov 25, 2014 at 2:34 PM, H?kon Strandenes > > wrote: >> >> (...) >> >> First, this is great debugging. > > Thanks. > >> >> Second, my reading of the HDF5 document you linked to says that either >> selection should be valid: >> >> "For non-regular hyperslab selection, parallel HDF5 uses independent >> IO internally for this option." >> >> so it ought to fall back to the INDEPENDENT model if it can't do >> collective calls correctly. However, >> it appears that the collective call has bugs. >> >> My conclusion: Since you have determined that changing the setting to >> INDEPENDENT produces >> correct input/output in all the test cases, and since my understanding >> of the HDF5 documentation is >> that we should always be able to use COLLECTIVE as an option, this is an >> HDF5 or MPT bug. > > I have conducted yet another test: > My example (ex10) that I previously posted to the mailing list was set > up with 250 grid points along each axis. When the topic on chunking was > brought to the table, I realized that 250 is not evenly dividable on > four. The example failed on 64 processes, that is four processes along > each direction (the division is 62 + 62 + 63 + 63 = 250). > > So I have recompiled "my ex10" with 256 gridpoints in each direction. It > turns out that this does in deed run successfully on 64 nodes. Great! It > also runs on 128 processes, that is a 8x4x4 decomposition. However it > does not run on 125 processes, that is a 5x5x5 decomposition. > > The same pattern is clear if I run my example with 250^3 grid points. It > does not run on numbers like 64 and 128, but does run successfully on > 125 processes, again only when the sub-domains are of exactly equal size > (in this case the domain is divided as 5x5x5). > > However, I still believe that there is bugs. I did my "roundtrip" by > loading a dataset and immediately writing the same dataset to a > different file, this time a 250^3 dataset on 125 processes. It did not > "pass" this test, i.e. the written dataset was just garbage. I have not > yet identified if the garbling is introduced in the reading or writing > of the dataset. > >> >> Does anyone else see the HDF5 differently? Also, it really looks to me >> like HDF5 messed up the MPI >> data type in the COLLECTIVE picture below, since it appears to be sliced >> incorrectly. >> >> Possible Remedies: >> >> 1) We can allow you to turn off H5Pset_dxpl_mpio() >> >> 2) Send this test case to the MPI/IO people at ANL >> >> If you think 1) is what you want, we can do it. If you can package this >> work for 2), it would be really valuable. > > I will be fine editing gr2.c manually each time this file is changed (I > use the sources from Git). But *if* this not a bug in MPT, but a bug in > PETSc or HDF5 it should be fixed... Because it is that kind of bug that > is extremely annoying and a read pain to track down. > > Perhaps the HDF5 mailing list could contribute in this issue? > >> >> Thanks, >> >> Matt >> >> Tanks for your time. >> >> Best regards, >> H?kon Strandenes >> >> > > Again thanks for your time. > > Regards, > H?kon > >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener From jed at jedbrown.org Wed Nov 26 07:23:43 2014 From: jed at jedbrown.org (Jed Brown) Date: Wed, 26 Nov 2014 07:23:43 -0600 Subject: [petsc-users] Problem with PETSc + HDF5 VecView In-Reply-To: <5475C6E8.3090609@hakostra.net> References: <547382AA.10705@hakostra.net> <5474E7B8.1090108@hakostra.net> <54757AB0.6070100@hakostra.net> <5475C6E8.3090609@hakostra.net> Message-ID: <87ioi22izk.fsf@jedbrown.org> H?kon Strandenes writes: > My local HPC group have found a solution to this problem: > On MPT it is possible to set an environment variable MPI_TYPE_DEPTH with > default value 8. The MPI_TYPE_DEPTH variable limits the maximum depth of > derived datatypes that an application can create. What a horrible thing. I wonder who made the decision to add this artificial limit despite upstream MPICH not having any such limit. Sounds like a regression to me and I recommend reporting it as such. > I have found that setting this to at least 32 will make my examples run > perfectly on up to 256 processes. No error messages what so ever, and in > my simple load and write dataset roundtrip h5diff compares the two > datasets and finds then identical. I also notice that Leibniz > Rechenzentrum recommend to set this variable to 100 (or some other > suitably large value) when using NetCDF together with MPT > (https://www.lrz.de/services/software/io/netcdf/). > > This bug have been a pain in the (***)... Perhaps it is worthy a FAQ entry? Yes, and tell procurement folks that you consider it to be a bug. What if your web browser could only display pages with less than 8 images? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Wed Nov 26 10:10:51 2014 From: jed at jedbrown.org (Jed Brown) Date: Wed, 26 Nov 2014 10:10:51 -0600 Subject: [petsc-users] Convergence of transposed linear system. In-Reply-To: References: Message-ID: <874mtm2b90.fsf@jedbrown.org> Gaetan Kenway writes: > The untransposed system converges about 6 orders of magnitude with > GRMES(100), ASM (overlap 1) and ILU(1) with RCM reordering. The test is run > on 128 processors. There are no convergence difficulties. > > However, when I try to solve the transpose of the same system, by either > calling KSPSolveTranspose() or by assembling the transpose of the linear > system and its preconditioner and calling KSPSolve(), GMRES stagnates after > a negligible drop in the residual and no further progress is made. Just a guess here, but the ASM default is "restricted ASM". Can you compare the nontransposed and transposed convergence with each of -pc_type I.e., 6 runs in total; how does each converge or not? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From gaetank at gmail.com Wed Nov 26 10:40:59 2014 From: gaetank at gmail.com (Gaetan Kenway) Date: Wed, 26 Nov 2014 11:40:59 -0500 Subject: [petsc-users] Convergence of transposed linear system. In-Reply-To: <874mtm2b90.fsf@jedbrown.org> References: <874mtm2b90.fsf@jedbrown.org> Message-ID: That's not a bad idea. I'll try that on the large RANS adjoints. However, I have been trying to replicate the same sort of behavior on smaller Euler meshes (~500k DOF) and I have been somewhat successful is replicating the same sort of issue, although the effect isn't as severe. For these cases, even with a single processor with an ILU(1) preconditioner, I'm seeing different convergence rates. Obviously in this case the ASM isn't the culprit. I still need to do a bit more digging to try to get to the bottom of this. When I get something, I'll post it. Thanks, Gaetan On Wed, Nov 26, 2014 at 11:10 AM, Jed Brown wrote: > Gaetan Kenway writes: > > The untransposed system converges about 6 orders of magnitude with > > GRMES(100), ASM (overlap 1) and ILU(1) with RCM reordering. The test is > run > > on 128 processors. There are no convergence difficulties. > > > > However, when I try to solve the transpose of the same system, by either > > calling KSPSolveTranspose() or by assembling the transpose of the linear > > system and its preconditioner and calling KSPSolve(), GMRES stagnates > after > > a negligible drop in the residual and no further progress is made. > > Just a guess here, but the ASM default is "restricted ASM". Can you > compare the nontransposed and transposed convergence with each of > > -pc_type > > I.e., 6 runs in total; how does each converge or not? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Nov 26 11:23:35 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 26 Nov 2014 11:23:35 -0600 Subject: [petsc-users] Problem with PETSc + HDF5 VecView In-Reply-To: <5475C6E8.3090609@hakostra.net> References: <547382AA.10705@hakostra.net> <5474E7B8.1090108@hakostra.net> <54757AB0.6070100@hakostra.net> <5475C6E8.3090609@hakostra.net> Message-ID: > On Nov 26, 2014, at 6:26 AM, H?kon Strandenes wrote: > > My local HPC group have found a solution to this problem: > On MPT it is possible to set an environment variable MPI_TYPE_DEPTH with default value 8. The MPI_TYPE_DEPTH variable limits the maximum depth of derived datatypes that an application can create. Is the variable MPI_TYPE_DEPTH actually set in the environment to 8 (by default) or does it use the value of 8 if the variable is not found? We can use getenv("MPI_TYPE_DEPTH"); in PETSc when the HDF5 viewer is created to make sure the value is sane and otherwise produce a useful error message telling the user exactly what to do. BUT we need to somehow limit this test to machines where it matters. So for example PETSC_EXTERN PetscErrorCode PetscViewerCreate_HDF5(PetscViewer v) { PetscViewer_HDF5 *hdf5; PetscErrorCode ierr; const char *typedepth; int itypedepth; PetscFunctionBegin; #if defined(PETSC_HAVE_HDF5_REQUIRE_LARGE_MPI_TYPE_DEPTH) typedepth = getenv("MPI_TYPE_DEPTH") sscanf(typedepth,"%d",&itypedepth); if (itypedepth < 100) SETERRQ(...,"This system requires you do \"export MPI_TYPE_DEPTH=100\" before submitting jobs when using HDF5"); #endif but we need a configure test that determines if this is such a system. Can you tell us a "system command" we could run in our configure to detect these SGI MPT system? Thanks Barry A big FAT error message is always better than a FAQ when possible. > > I have found that setting this to at least 32 will make my examples run perfectly on up to 256 processes. No error messages what so ever, and in my simple load and write dataset roundtrip h5diff compares the two datasets and finds then identical. I also notice that Leibniz Rechenzentrum recommend to set this variable to 100 (or some other suitably large value) when using NetCDF together with MPT (https://www.lrz.de/services/software/io/netcdf/). > > This bug have been a pain in the (***)... Perhaps it is worthy a FAQ entry? > > Thanks for your time and effort. > > Regards, > H?kon Strandenes > > > On 26. nov. 2014 08:01, H?kon Strandenes wrote: >> >> >> On 25. nov. 2014 22:40, Matthew Knepley wrote: >>> On Tue, Nov 25, 2014 at 2:34 PM, H?kon Strandenes >> > wrote: >>> >>> (...) >>> >>> First, this is great debugging. >> >> Thanks. >> >>> >>> Second, my reading of the HDF5 document you linked to says that either >>> selection should be valid: >>> >>> "For non-regular hyperslab selection, parallel HDF5 uses independent >>> IO internally for this option." >>> >>> so it ought to fall back to the INDEPENDENT model if it can't do >>> collective calls correctly. However, >>> it appears that the collective call has bugs. >>> >>> My conclusion: Since you have determined that changing the setting to >>> INDEPENDENT produces >>> correct input/output in all the test cases, and since my understanding >>> of the HDF5 documentation is >>> that we should always be able to use COLLECTIVE as an option, this is an >>> HDF5 or MPT bug. >> >> I have conducted yet another test: >> My example (ex10) that I previously posted to the mailing list was set >> up with 250 grid points along each axis. When the topic on chunking was >> brought to the table, I realized that 250 is not evenly dividable on >> four. The example failed on 64 processes, that is four processes along >> each direction (the division is 62 + 62 + 63 + 63 = 250). >> >> So I have recompiled "my ex10" with 256 gridpoints in each direction. It >> turns out that this does in deed run successfully on 64 nodes. Great! It >> also runs on 128 processes, that is a 8x4x4 decomposition. However it >> does not run on 125 processes, that is a 5x5x5 decomposition. >> >> The same pattern is clear if I run my example with 250^3 grid points. It >> does not run on numbers like 64 and 128, but does run successfully on >> 125 processes, again only when the sub-domains are of exactly equal size >> (in this case the domain is divided as 5x5x5). >> >> However, I still believe that there is bugs. I did my "roundtrip" by >> loading a dataset and immediately writing the same dataset to a >> different file, this time a 250^3 dataset on 125 processes. It did not >> "pass" this test, i.e. the written dataset was just garbage. I have not >> yet identified if the garbling is introduced in the reading or writing >> of the dataset. >> >>> >>> Does anyone else see the HDF5 differently? Also, it really looks to me >>> like HDF5 messed up the MPI >>> data type in the COLLECTIVE picture below, since it appears to be sliced >>> incorrectly. >>> >>> Possible Remedies: >>> >>> 1) We can allow you to turn off H5Pset_dxpl_mpio() >>> >>> 2) Send this test case to the MPI/IO people at ANL >>> >>> If you think 1) is what you want, we can do it. If you can package this >>> work for 2), it would be really valuable. >> >> I will be fine editing gr2.c manually each time this file is changed (I >> use the sources from Git). But *if* this not a bug in MPT, but a bug in >> PETSc or HDF5 it should be fixed... Because it is that kind of bug that >> is extremely annoying and a read pain to track down. >> >> Perhaps the HDF5 mailing list could contribute in this issue? >> >>> >>> Thanks, >>> >>> Matt >>> >>> Tanks for your time. >>> >>> Best regards, >>> H?kon Strandenes >>> >>> >> >> Again thanks for your time. >> >> Regards, >> H?kon >> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their experiments lead. >>> -- Norbert Wiener From bisheshkh at gmail.com Wed Nov 26 15:13:29 2014 From: bisheshkh at gmail.com (Bishesh Khanal) Date: Wed, 26 Nov 2014 22:13:29 +0100 Subject: [petsc-users] interpolate staggered grid values in parallel In-Reply-To: References: Message-ID: On Tue, Nov 25, 2014 at 6:40 PM, Matthew Knepley wrote: > On Tue, Nov 25, 2014 at 11:36 AM, Bishesh Khanal > wrote: > >> Dear all, >> I'm solving a system using petsc and I get a global Vec, say X . It uses >> DMDA with 4 dofs (3 velocity + 1 pressure). X contains velocity at the cell >> faces since I solve using staggered grid. >> Now I'd like to create one array for velocity with values at cell centers >> and another array for pressure (not the Vec so that I can send the pointer >> to the array to other part of the code that does not use Petsc). >> >> Currently what I do is : >> ------ Vec X, X_local; >> ------ PetscScalar *X_array; >> >> // Scatter X to X_local and then use: >> >> ------- VecGetArray(X_local, &X_array) >> >> And then have a function, say >> getVelocityAt(x, y, z, component) { >> // interpolate velocity at (x,y,z) cell center using X_array >> } >> >> The function getVelocityAt() gets called from outside petsc in a loop >> over all (x,y,z) positions. >> This is not done in parallel. >> >> Now, how do I use Petsc to instead interpolate the cell center velocities >> in parallel and store it >> in an array say >> PetscScalar *X_array_cellCenter; >> ? >> This would need to have size one less along each axis compared to the >> orginal DMDA size. >> This way I intend to return X_array_cellCenter to the code outside Petsc. >> > > SNES ex30 is an example of a staggered grid code using DMDA. It does this > kind of interpolation, > and puts the result in a Vec. > After looking at the example, I tried the following but I have a problem: The idea I take from the e.g is to use DMDALocalInfo *info to get local grid coordinates and use for loop to iterate through local grid values that are available in the Field **x array to do the interpolation. Here is what I tried in my case where I get the solution from a ksp attached to a dmda: // Get the solution to Vec x from ksp. // ksp is attached to DMDA da which has 4 dofs: vx,vy,vz,p; KSPGetSoution(ksp, &x); //x has velocity solution in faces of the cell. //Get the array from x: Field ***xArray; //Field is a struct with 4 members: vx,vy,vz and p. DMDAGetVectorArray(da,xVec,&xArray); //Now I want to have a new array to store the cell center velocity values //by interpolating from xArray. DMCreateGlobalVector(daV, &xC); //daV has same size as da but with dof=3 FieldVelocity ***xCArray; //FieldVelocity is a struct with 3 members: vx,vy and vz. DMDAVecGetArray(daV,xC,&xCArray); //Do the interpolation DMDAGetLocalInfo(da,&info); for (PetscInt k=info.zs; k > Thanks, > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From haakon at hakostra.net Wed Nov 26 15:35:55 2014 From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=) Date: Wed, 26 Nov 2014 22:35:55 +0100 Subject: [petsc-users] Problem with PETSc + HDF5 VecView In-Reply-To: References: <547382AA.10705@hakostra.net> <5474E7B8.1090108@hakostra.net> <54757AB0.6070100@hakostra.net> <5475C6E8.3090609@hakostra.net> Message-ID: <547647BB.1080103@hakostra.net> On 26. nov. 2014 18:23, Barry Smith wrote: > >> On Nov 26, 2014, at 6:26 AM, H?kon Strandenes wrote: >> >> My local HPC group have found a solution to this problem: >> On MPT it is possible to set an environment variable MPI_TYPE_DEPTH with default value 8. The MPI_TYPE_DEPTH variable limits the maximum depth of derived datatypes that an application can create. > > Is the variable MPI_TYPE_DEPTH actually set in the environment to 8 (by default) or does it use the value of 8 if the variable is not found? The variable does not exist by default. The variable is described here: http://techpubs.sgi.com/library/dynaweb_docs/0620/SGI_Developer/books/MPT_MPI_PM/sgi_html/ch06.html If not set, default to 8. > > We can use getenv("MPI_TYPE_DEPTH"); in PETSc when the HDF5 viewer is created to make sure the value is sane and otherwise produce a useful error message telling the user exactly what to do. BUT we need to somehow limit this test to machines where it matters. So for example > > PETSC_EXTERN PetscErrorCode PetscViewerCreate_HDF5(PetscViewer v) > { > PetscViewer_HDF5 *hdf5; > PetscErrorCode ierr; > const char *typedepth; > int itypedepth; > > PetscFunctionBegin; > #if defined(PETSC_HAVE_HDF5_REQUIRE_LARGE_MPI_TYPE_DEPTH) > typedepth = getenv("MPI_TYPE_DEPTH") > sscanf(typedepth,"%d",&itypedepth); > if (itypedepth < 100) SETERRQ(...,"This system requires you do \"export MPI_TYPE_DEPTH=100\" before submitting jobs when using HDF5"); > #endif > > but we need a configure test that determines if this is such a system. Can you tell us a "system command" we could run in our configure to detect these SGI MPT system? > Is it really PETSc's taks to warn about this? PETSc should trust HDF5 to "just work" and HDF5 should actually print sensible warnings/error messages. Shouldn't it? I'll think about that system command until tomorrow... > Thanks > > Barry > > A big FAT error message is always better than a FAQ when possible. Of course. H?kon > > >> >> I have found that setting this to at least 32 will make my examples run perfectly on up to 256 processes. No error messages what so ever, and in my simple load and write dataset roundtrip h5diff compares the two datasets and finds then identical. I also notice that Leibniz Rechenzentrum recommend to set this variable to 100 (or some other suitably large value) when using NetCDF together with MPT (https://www.lrz.de/services/software/io/netcdf/). >> >> This bug have been a pain in the (***)... Perhaps it is worthy a FAQ entry? >> >> Thanks for your time and effort. >> >> Regards, >> H?kon Strandenes >> >> >> On 26. nov. 2014 08:01, H?kon Strandenes wrote: >>> >>> >>> On 25. nov. 2014 22:40, Matthew Knepley wrote: >>>> On Tue, Nov 25, 2014 at 2:34 PM, H?kon Strandenes >>> > wrote: >>>> >>>> (...) >>>> >>>> First, this is great debugging. >>> >>> Thanks. >>> >>>> >>>> Second, my reading of the HDF5 document you linked to says that either >>>> selection should be valid: >>>> >>>> "For non-regular hyperslab selection, parallel HDF5 uses independent >>>> IO internally for this option." >>>> >>>> so it ought to fall back to the INDEPENDENT model if it can't do >>>> collective calls correctly. However, >>>> it appears that the collective call has bugs. >>>> >>>> My conclusion: Since you have determined that changing the setting to >>>> INDEPENDENT produces >>>> correct input/output in all the test cases, and since my understanding >>>> of the HDF5 documentation is >>>> that we should always be able to use COLLECTIVE as an option, this is an >>>> HDF5 or MPT bug. >>> >>> I have conducted yet another test: >>> My example (ex10) that I previously posted to the mailing list was set >>> up with 250 grid points along each axis. When the topic on chunking was >>> brought to the table, I realized that 250 is not evenly dividable on >>> four. The example failed on 64 processes, that is four processes along >>> each direction (the division is 62 + 62 + 63 + 63 = 250). >>> >>> So I have recompiled "my ex10" with 256 gridpoints in each direction. It >>> turns out that this does in deed run successfully on 64 nodes. Great! It >>> also runs on 128 processes, that is a 8x4x4 decomposition. However it >>> does not run on 125 processes, that is a 5x5x5 decomposition. >>> >>> The same pattern is clear if I run my example with 250^3 grid points. It >>> does not run on numbers like 64 and 128, but does run successfully on >>> 125 processes, again only when the sub-domains are of exactly equal size >>> (in this case the domain is divided as 5x5x5). >>> >>> However, I still believe that there is bugs. I did my "roundtrip" by >>> loading a dataset and immediately writing the same dataset to a >>> different file, this time a 250^3 dataset on 125 processes. It did not >>> "pass" this test, i.e. the written dataset was just garbage. I have not >>> yet identified if the garbling is introduced in the reading or writing >>> of the dataset. >>> >>>> >>>> Does anyone else see the HDF5 differently? Also, it really looks to me >>>> like HDF5 messed up the MPI >>>> data type in the COLLECTIVE picture below, since it appears to be sliced >>>> incorrectly. >>>> >>>> Possible Remedies: >>>> >>>> 1) We can allow you to turn off H5Pset_dxpl_mpio() >>>> >>>> 2) Send this test case to the MPI/IO people at ANL >>>> >>>> If you think 1) is what you want, we can do it. If you can package this >>>> work for 2), it would be really valuable. >>> >>> I will be fine editing gr2.c manually each time this file is changed (I >>> use the sources from Git). But *if* this not a bug in MPT, but a bug in >>> PETSc or HDF5 it should be fixed... Because it is that kind of bug that >>> is extremely annoying and a read pain to track down. >>> >>> Perhaps the HDF5 mailing list could contribute in this issue? >>> >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> Tanks for your time. >>>> >>>> Best regards, >>>> H?kon Strandenes >>>> >>>> >>> >>> Again thanks for your time. >>> >>> Regards, >>> H?kon >>> >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their experiments lead. >>>> -- Norbert Wiener > > From bsmith at mcs.anl.gov Wed Nov 26 16:37:02 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 26 Nov 2014 16:37:02 -0600 Subject: [petsc-users] Problem with PETSc + HDF5 VecView In-Reply-To: <547647BB.1080103@hakostra.net> References: <547382AA.10705@hakostra.net> <5474E7B8.1090108@hakostra.net> <54757AB0.6070100@hakostra.net> <5475C6E8.3090609@hakostra.net> <547647BB.1080103@hakostra.net> Message-ID: > On Nov 26, 2014, at 3:35 PM, H?kon Strandenes wrote: > >> > Is it really PETSc's taks to warn about this? PETSc should trust HDF5 to "just work" and HDF5 should actually print sensible warnings/error messages. Shouldn't it? Yes, but if we produce a nice error message it makes everyone's lives easier, including ours because we don't have to constantly answer emails about the same problem discovered by a new person over and over again. Hence we do this kind of thing a lot. Barry > > I'll think about that system command until tomorrow... > >> Thanks >> >> Barry >> >> A big FAT error message is always better than a FAQ when possible. > Of course. > > H?kon > >> >> >>> >>> I have found that setting this to at least 32 will make my examples run perfectly on up to 256 processes. No error messages what so ever, and in my simple load and write dataset roundtrip h5diff compares the two datasets and finds then identical. I also notice that Leibniz Rechenzentrum recommend to set this variable to 100 (or some other suitably large value) when using NetCDF together with MPT (https://www.lrz.de/services/software/io/netcdf/). >>> >>> This bug have been a pain in the (***)... Perhaps it is worthy a FAQ entry? >>> >>> Thanks for your time and effort. >>> >>> Regards, >>> H?kon Strandenes >>> >>> >>> On 26. nov. 2014 08:01, H?kon Strandenes wrote: >>>> >>>> >>>> On 25. nov. 2014 22:40, Matthew Knepley wrote: >>>>> On Tue, Nov 25, 2014 at 2:34 PM, H?kon Strandenes >>>> > wrote: >>>>> >>>>> (...) >>>>> >>>>> First, this is great debugging. >>>> >>>> Thanks. >>>> >>>>> >>>>> Second, my reading of the HDF5 document you linked to says that either >>>>> selection should be valid: >>>>> >>>>> "For non-regular hyperslab selection, parallel HDF5 uses independent >>>>> IO internally for this option." >>>>> >>>>> so it ought to fall back to the INDEPENDENT model if it can't do >>>>> collective calls correctly. However, >>>>> it appears that the collective call has bugs. >>>>> >>>>> My conclusion: Since you have determined that changing the setting to >>>>> INDEPENDENT produces >>>>> correct input/output in all the test cases, and since my understanding >>>>> of the HDF5 documentation is >>>>> that we should always be able to use COLLECTIVE as an option, this is an >>>>> HDF5 or MPT bug. >>>> >>>> I have conducted yet another test: >>>> My example (ex10) that I previously posted to the mailing list was set >>>> up with 250 grid points along each axis. When the topic on chunking was >>>> brought to the table, I realized that 250 is not evenly dividable on >>>> four. The example failed on 64 processes, that is four processes along >>>> each direction (the division is 62 + 62 + 63 + 63 = 250). >>>> >>>> So I have recompiled "my ex10" with 256 gridpoints in each direction. It >>>> turns out that this does in deed run successfully on 64 nodes. Great! It >>>> also runs on 128 processes, that is a 8x4x4 decomposition. However it >>>> does not run on 125 processes, that is a 5x5x5 decomposition. >>>> >>>> The same pattern is clear if I run my example with 250^3 grid points. It >>>> does not run on numbers like 64 and 128, but does run successfully on >>>> 125 processes, again only when the sub-domains are of exactly equal size >>>> (in this case the domain is divided as 5x5x5). >>>> >>>> However, I still believe that there is bugs. I did my "roundtrip" by >>>> loading a dataset and immediately writing the same dataset to a >>>> different file, this time a 250^3 dataset on 125 processes. It did not >>>> "pass" this test, i.e. the written dataset was just garbage. I have not >>>> yet identified if the garbling is introduced in the reading or writing >>>> of the dataset. >>>> >>>>> >>>>> Does anyone else see the HDF5 differently? Also, it really looks to me >>>>> like HDF5 messed up the MPI >>>>> data type in the COLLECTIVE picture below, since it appears to be sliced >>>>> incorrectly. >>>>> >>>>> Possible Remedies: >>>>> >>>>> 1) We can allow you to turn off H5Pset_dxpl_mpio() >>>>> >>>>> 2) Send this test case to the MPI/IO people at ANL >>>>> >>>>> If you think 1) is what you want, we can do it. If you can package this >>>>> work for 2), it would be really valuable. >>>> >>>> I will be fine editing gr2.c manually each time this file is changed (I >>>> use the sources from Git). But *if* this not a bug in MPT, but a bug in >>>> PETSc or HDF5 it should be fixed... Because it is that kind of bug that >>>> is extremely annoying and a read pain to track down. >>>> >>>> Perhaps the HDF5 mailing list could contribute in this issue? >>>> >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> Tanks for your time. >>>>> >>>>> Best regards, >>>>> H?kon Strandenes >>>>> >>>>> >>>> >>>> Again thanks for your time. >>>> >>>> Regards, >>>> H?kon >>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which >>>>> their experiments lead. >>>>> -- Norbert Wiener >> >> From jed at jedbrown.org Wed Nov 26 23:53:14 2014 From: jed at jedbrown.org (Jed Brown) Date: Wed, 26 Nov 2014 23:53:14 -0600 Subject: [petsc-users] Problem with PETSc + HDF5 VecView In-Reply-To: References: <547382AA.10705@hakostra.net> <5474E7B8.1090108@hakostra.net> <54757AB0.6070100@hakostra.net> <5475C6E8.3090609@hakostra.net> <547647BB.1080103@hakostra.net> Message-ID: <87y4qxyyt1.fsf@jedbrown.org> Barry Smith writes: >> On Nov 26, 2014, at 3:35 PM, H?kon Strandenes wrote: >> >>> >> Is it really PETSc's taks to warn about this? PETSc should trust HDF5 to "just work" and HDF5 should actually print sensible warnings/error messages. Shouldn't it? > > Yes, but if we produce a nice error message it makes everyone's > lives easier, including ours because we don't have to constantly > answer emails about the same problem discovered by a new person > over and over again. Hence we do this kind of thing a lot. The only concern is that the user might not exceed that depth, so why should they have to set an arbitrary and excessively big value? MPI_REQUEST_MAX and MPI_COMM_MAX also look like candidates for needing increases. Whoever is responsible for this perversion should be fired. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From kvoronin at labchem.sscc.ru Thu Nov 27 02:54:05 2014 From: kvoronin at labchem.sscc.ru (Kirill Voronin) Date: Thu, 27 Nov 2014 15:54:05 +0700 Subject: [petsc-users] User-defined residual norm calculation in PETSC solvers Message-ID: <6198d91b2e8dd3211ce6fac15873a4ca.squirrel@mx.sscc.ru> Hello! Thanks Matthew Knepley and Barry Smith for the answers to my previous questions! I am solving Ax = b with PETSC BiCGStab with user-defined preconditioner and want to compare it with other iteration process. The question is 1) Is there a possibility to monitor a user-defined function of residual norm(e.g., useer-defined computation of NORM_1) at each iteration of PETSC solver? (I cannot rewrite the user-defined routine for residual norm using PETSC interface since it is just fixed and given to me)? ------ It seems now that the residual norm (L1 = NORM_1) is calculated somewhat different through PETSC and user-defined routine. The difference is rather small but still for comparison with other solver I would prefer to have the same numbers. As a result, by now I have different values of residual norms measured for the same data by PETSC and external user-defined routine. (I checked it just be getting arrays from PETSC vectors and then giving them to the user-defined routine of residual norm). I found KSPSetConvergenceTest but the convergence is measured always in terms of NORM_2 of the error as far as I understand. Thanks in advance! -- Best regards, Kirill Voronin From jed at jedbrown.org Thu Nov 27 04:19:00 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 27 Nov 2014 04:19:00 -0600 Subject: [petsc-users] User-defined residual norm calculation in PETSC solvers In-Reply-To: <6198d91b2e8dd3211ce6fac15873a4ca.squirrel@mx.sscc.ru> References: <6198d91b2e8dd3211ce6fac15873a4ca.squirrel@mx.sscc.ru> Message-ID: <87oartymi3.fsf@jedbrown.org> Kirill Voronin writes: > Hello! > > Thanks Matthew Knepley and Barry Smith for the answers to my previous > questions! > > I am solving Ax = b with PETSC BiCGStab with user-defined preconditioner > and want to compare it with other iteration process. > > The question is > 1) > Is there a possibility to monitor a user-defined function of residual > norm(e.g., useer-defined computation of NORM_1) at each iteration of PETSC > solver? KSPMonitorSet your own implementation, written similarly to this: http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/interface/iterativ.c.html#KSPMonitorTrueResidualMaxNorm > (I cannot rewrite the user-defined routine for residual norm using > PETSC interface since it is just fixed and given to me)? > > ------ > It seems now that the residual norm (L1 = NORM_1) is calculated somewhat > different through PETSC and user-defined routine. You're comparing VecNorm(X,NORM_1,&nrm1) to a user-defined function that purports to compute the same thing? What is your implementation? > The difference is rather > small but still for comparison with other solver I would prefer to have > the same numbers. > As a result, by now I have different values of residual norms measured for > the same data by PETSC and external user-defined routine. (I checked it > just be getting arrays from PETSC vectors and then giving them to the > user-defined routine of residual norm). > > I found KSPSetConvergenceTest but the convergence is measured always in > terms of NORM_2 of the error as far as I understand. Your convergence test can use any norm you want, but it's expensive to build a residual for some Krylov methods (e.g., GMRES), so you might not be happy with the performance. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From asmund.ervik at ntnu.no Thu Nov 27 05:45:30 2014 From: asmund.ervik at ntnu.no (=?iso-8859-1?Q?=C5smund_Ervik?=) Date: Thu, 27 Nov 2014 11:45:30 +0000 Subject: [petsc-users] Problem with PETSc + HDF5 VecView Message-ID: <0E576811AB298343AC632BBCAAEFC37948417EBC@WAREHOUSE08.win.ntnu.no> Hello H?kon, I have to express my sincere thanks for digging in and solving this one. We have a code that uses HDF5 and 3D DMDAs, that has been running on smaller clusters and will be running next year on VIlje. (I assume you are using Vilje?) So we would have hit this problem, and you've saved us quite a lot of debugging. I'm in London for the next few months, but I'll make sure to buy you a coffee some time when I'm back in Moustache City. Best regards, ?smund >Date: Wed, 26 Nov 2014 22:35:55 +0100 >From: H?kon Strandenes >To: Barry Smith >Cc: petsc-users at mcs.anl.gov >Subject: Re: [petsc-users] Problem with PETSc + HDF5 VecView >Message-ID: <547647BB.1080103 at hakostra.net> >Content-Type: text/plain; charset=utf-8; format=flowed > > > >On 26. nov. 2014 18:23, Barry Smith wrote: > >> On Nov 26, 2014, at 6:26 AM, H?kon Strandenes wrote: >> >> My local HPC group have found a solution to this problem: >> On MPT it is possible to set an environment variable MPI_TYPE_DEPTH with default value 8. The MPI_TYPE_DEPTH variable limits the maximum depth of derived datatypes that an application can create. > > Is the variable MPI_TYPE_DEPTH actually set in the environment to 8 (by default) or does it use the value of 8 if the variable is not found? The variable does not exist by default. The variable is described here: http://techpubs.sgi.com/library/dynaweb_docs/0620/SGI_Developer/books/MPT_MPI_PM/sgi_html/ch06.html If not set, default to 8. > > We can use getenv("MPI_TYPE_DEPTH"); in PETSc when the HDF5 viewer is created to make sure the value is sane and otherwise produce a useful error message telling the user exactly what to do. BUT we need to somehow limit this test to machines where it matters. So for example > > PETSC_EXTERN PetscErrorCode PetscViewerCreate_HDF5(PetscViewer v) > { > PetscViewer_HDF5 *hdf5; > PetscErrorCode ierr; > const char *typedepth; > int itypedepth; > > PetscFunctionBegin; > #if defined(PETSC_HAVE_HDF5_REQUIRE_LARGE_MPI_TYPE_DEPTH) > typedepth = getenv("MPI_TYPE_DEPTH") > sscanf(typedepth,"%d",&itypedepth); > if (itypedepth < 100) SETERRQ(...,"This system requires you do \"export MPI_TYPE_DEPTH=100\" before submitting jobs when using HDF5"); > #endif > > but we need a configure test that determines if this is such a system. Can you tell us a "system command" we could run in our configure to detect these SGI MPT system? > Is it really PETSc's taks to warn about this? PETSc should trust HDF5 to "just work" and HDF5 should actually print sensible warnings/error messages. Shouldn't it? I'll think about that system command until tomorrow... > Thanks > > Barry > > A big FAT error message is always better than a FAQ when possible. Of course. H?kon > > >> >> I have found that setting this to at least 32 will make my examples run perfectly on up to 256 processes. No error messages what so ever, and in my simple load and write dataset roundtrip h5diff compares the two datasets and finds then identical. I also notice that Leibniz Rechenzentrum recommend to set this variable to 100 (or some other suitably large value) when using NetCDF together with MPT (https://www.lrz.de/services/software/io/netcdf/). >> >> This bug have been a pain in the (***)... Perhaps it is worthy a FAQ entry? >> >> Thanks for your time and effort. >> >> Regards, >> H?kon Strandenes >> >> >> On 26. nov. 2014 08:01, H?kon Strandenes wrote: >>> >>> >>> On 25. nov. 2014 22:40, Matthew Knepley wrote: >>>> On Tue, Nov 25, 2014 at 2:34 PM, H?kon Strandenes >>> > wrote: >>>> >>>> (...) >>>> >>>> First, this is great debugging. >>> >>> Thanks. >>> >>>> >>>> Second, my reading of the HDF5 document you linked to says that either >>>> selection should be valid: >>>> >>>> "For non-regular hyperslab selection, parallel HDF5 uses independent >>>> IO internally for this option." >>>> >>>> so it ought to fall back to the INDEPENDENT model if it can't do >>>> collective calls correctly. However, >>>> it appears that the collective call has bugs. >>>> >>>> My conclusion: Since you have determined that changing the setting to >>>> INDEPENDENT produces >>>> correct input/output in all the test cases, and since my understanding >>>> of the HDF5 documentation is >>>> that we should always be able to use COLLECTIVE as an option, this is an >>>> HDF5 or MPT bug. >>> >>> I have conducted yet another test: >>> My example (ex10) that I previously posted to the mailing list was set >>> up with 250 grid points along each axis. When the topic on chunking was >>> brought to the table, I realized that 250 is not evenly dividable on >>> four. The example failed on 64 processes, that is four processes along >>> each direction (the division is 62 + 62 + 63 + 63 = 250). >>> >>> So I have recompiled "my ex10" with 256 gridpoints in each direction. It >>> turns out that this does in deed run successfully on 64 nodes. Great! It >>> also runs on 128 processes, that is a 8x4x4 decomposition. However it >>> does not run on 125 processes, that is a 5x5x5 decomposition. >>> >>> The same pattern is clear if I run my example with 250^3 grid points. It >>> does not run on numbers like 64 and 128, but does run successfully on >>> 125 processes, again only when the sub-domains are of exactly equal size >>> (in this case the domain is divided as 5x5x5). >>> >>> However, I still believe that there is bugs. I did my "roundtrip" by >>> loading a dataset and immediately writing the same dataset to a >>> different file, this time a 250^3 dataset on 125 processes. It did not >>> "pass" this test, i.e. the written dataset was just garbage. I have not >>> yet identified if the garbling is introduced in the reading or writing >>> of the dataset. >>> >>>> >>>> Does anyone else see the HDF5 differently? Also, it really looks to me >>>> like HDF5 messed up the MPI >>>> data type in the COLLECTIVE picture below, since it appears to be sliced >>>> incorrectly. >>>> >>>> Possible Remedies: >>>> >>>> 1) We can allow you to turn off H5Pset_dxpl_mpio() >>>> >>>> 2) Send this test case to the MPI/IO people at ANL >>>> >>>> If you think 1) is what you want, we can do it. If you can package this >>>> work for 2), it would be really valuable. >>> >>> I will be fine editing gr2.c manually each time this file is changed (I >>> use the sources from Git). But *if* this not a bug in MPT, but a bug in >>> PETSc or HDF5 it should be fixed... Because it is that kind of bug that >>> is extremely annoying and a read pain to track down. >>> >>> Perhaps the HDF5 mailing list could contribute in this issue? >>> >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> Tanks for your time. >>>> >>>> Best regards, >>>> H?kon Strandenes >>>> >>>> >>> >>> Again thanks for your time. >>> >>> Regards, >>> H?kon >>> >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their experiments lead. >>>> -- Norbert Wiener > > From olivier.bonnefon at avignon.inra.fr Thu Nov 27 07:06:58 2014 From: olivier.bonnefon at avignon.inra.fr (Olivier Bonnefon) Date: Thu, 27 Nov 2014 14:06:58 +0100 Subject: [petsc-users] petsc-3.5.2: ex12 with Neumann BC In-Reply-To: References: <524430F2.2020507@avignon.inra.fr> <524D3D5D.5090301@avignon.inra.fr> <525804C2.3000101@avignon.inra.fr> <5453AE09.5090806@avignon.inra.fr> <54574EED.3060309@avignon.inra.fr> <5474552E.2080808@avignon.inra.fr> Message-ID: <547721F2.1010005@avignon.inra.fr> Hello, Thanks you, it works. If I remove the NullSpaceCondition I get the correct solution also with Neumann BC. From my point of view, the linear system I get is not singular because there is a unique solution. Do you agree ? Regards, Olivier Bonnefon On 11/25/2014 03:01 PM, Matthew Knepley wrote: > On Tue, Nov 25, 2014 at 4:08 AM, Olivier Bonnefon > > wrote: > > Hello, > > Thank you for your answer, I agree with you. > > I have adapted the ex12 example for the system: > > -\nabla \nabla u + u + f = 0 in \Omega (1) > > with f = 4-x^2-y^2 > The solution is still x^2-y^2. > > It consists in adding the following gradient function: > > void g0_uu(const PetscScalar u[], const PetscScalar u_t[], const > PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], > const PetscScalar a_x[], const PetscReal x[], PetscScalar g0[]) > { > g0[0] = 1.0; > } > ... > //and set gradient > ierr = PetscDSSetJacobian(prob, 0, 0, g0_uu, NULL, NULL, > g3_uu);CHKERRQ(ierr); > ... > Of course, I have modified the f0_u function. > > > With Dirichlet BC, I get the solution. > > With Neumann BC, the solution is still shifted down (-2/3). The > problem (1) with Neumann BC has a unique solution. > How remove the condition \int_\Omega u = 0 ? > > > That condition is one way of imposing boundary conditions for > pressure. When I call MatSetNullSpace(), I am implying > this condition. You can take out that call, but the system will be > singular. You would have to add a condition on the > pressure somehow, like fixing it at a point. This tends to make the > system more ill-conditioned, but if that is appropriate > for you, then you make a similar call to > > ierr = DMPlexAddBoundary(cdm, user->bcType == DIRICHLET, "wall", > user->bcType == NEUMANN ? "boundary" : "marker", 0, > user->exactFuncs[0], 1, &id, user);CHKERRQ(ierr); > > but for field 1. To test , you can jsut change 0 to 1 in this call. > However, this sets the entire boundary, and you really only > need to set the pressure at one point, so you could make another label > that only has a single point and use that instead. > > Thanks, > > Matt > > Thanks > Olivier Bonnefon > > > > On 11/06/2014 07:14 AM, Matthew Knepley wrote: >> On Mon, Nov 3, 2014 at 9:46 AM, Olivier Bonnefon >> > > wrote: >> >> Hello, >> >> Thank for your answer, I'll explain my trouble: >> >> My problem is that the BC Neumann leads to wrong result. >> >> With Dirichlet BC, I get: >> ------------------------------- >> >> > ./ex12 -run_type full -refinement_limit 0.0 -bc_type >> dirichlet -interpolate 0 -petscspace_order 1 -show_initial >> -show_solution -dm_plex_print_fem 1 >> ... >> ... >> Solution >> Vec Object:potential 1 MPI processes >> type: seq >> 0.5 >> >> This result is correct. >> >> With Neuman BC, I get: >> -------------------------------- >> >> > ./ex12 -run_type full -refinement_limit 0.0 -bc_type >> neumann -interpolate 1 -petscspace_order 1 -show_initial >> -dm_plex_print_fem 1 -show_solution -bd_petscspace_order 1 >> -snes_linesearch_monitor -snes_monitor >> -ksp_monitor_true_residual -snes_converged_reason >> -ksp_converged_reason >> .... >> .... >> >> >> Solution >> Vec Object:potential 1 MPI processes >> type: seq >> -0.75 >> -0.583333 >> 0.0833333 >> -0.583333 >> -0.333333 >> 0.416667 >> 0.0833333 >> 0.416667 >> 1.25 >> >> >> That is not the values of the solution x*x+y*y. >> >> >> Sorry, this is poor documentation on my part. I will fix it in >> the example. The Naumann solution >> explicitly discards the constant part, meaning >> >> \int_\Omega u = 0 >> >> If we look at my solution >> >> \int^1_0 dx \int^1_0 dy x^2 + y^2 = 2/3 >> >> However, this is for the continuum result, whereas we are >> enforcing this in the discrete case. >> Thus, what we really get is 0.75, since we are integrating linear >> patches instead of quadratic >> patches. After shifting by that, you still do not get exactly x^2 >> + y^2 since there is some >> discretization error. If you run with P2 you should get the exact >> answer, shifted down to eliminate >> the DC component. >> >> Thanks, >> >> Matt >> >> >> I tried many ksp options. >> Moreover, the neumann BC with "-run_type full" is not cover >> in the list >> https://bitbucket.org/petsc/petsc/src/fced3c3f9e703542693913793d15321603e40fe6/config/builder.py?at=master#cl-257 >> >> Do you know what is wrong ? >> >> Thanks, >> >> Olivier Bonnefon >> >> On 10/31/2014 06:50 PM, Matthew Knepley wrote: >>> On Fri, Oct 31, 2014 at 10:43 AM, Olivier Bonnefon >>> >> > wrote: >>> >>> Hello, >>> >>> I'm working on the snes/examples/tutorial/ex12 version >>> 3.5.2. >>> >>> I didn't succed to run the simplest case with Neumann BC: >>> >>> ./ex12 -run_type full -refinement_limit 0.0 -bc_type >>> neumann -interpolate 1 -petscspace_order 1 -show_initial >>> -dm_plex_print_fem 1 -show_solution -bd_petscspace_order >>> 1 -snes_linesearch_monitor -snes_monitor >>> -ksp_monitor_true_residual -snes_converged_reason >>> -ksp_converged_reason >>> >>> This leads to dofs negatives values. >>> >>> >>> I do not understand what you mean. Please always mail the >>> full error message. >>> >>> Do you know the options to get a correct result with >>> Neumann BC ? >>> >>> >>> There are some tests here: >>> >>> https://bitbucket.org/petsc/petsc/src/fced3c3f9e703542693913793d15321603e40fe6/config/builder.py?at=master#cl-257 >>> >>> Matt >>> >>> Regards, >>> Olivier Bonnefon >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >> >> >> -- >> Olivier Bonnefon >> INRA PACA-Avignon, Unit? BioSP >> Tel:+33 (0)4 32 72 21 58 >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > -- > Olivier Bonnefon > INRA PACA-Avignon, Unit? BioSP > Tel:+33 (0)4 32 72 21 58 > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -- Olivier Bonnefon INRA PACA-Avignon, Unit? BioSP Tel: +33 (0)4 32 72 21 58 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Nov 27 07:44:29 2014 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 27 Nov 2014 07:44:29 -0600 Subject: [petsc-users] petsc-3.5.2: ex12 with Neumann BC In-Reply-To: <547721F2.1010005@avignon.inra.fr> References: <524430F2.2020507@avignon.inra.fr> <524D3D5D.5090301@avignon.inra.fr> <525804C2.3000101@avignon.inra.fr> <5453AE09.5090806@avignon.inra.fr> <54574EED.3060309@avignon.inra.fr> <5474552E.2080808@avignon.inra.fr> <547721F2.1010005@avignon.inra.fr> Message-ID: On Thu, Nov 27, 2014 at 7:06 AM, Olivier Bonnefon < olivier.bonnefon at avignon.inra.fr> wrote: > Hello, > > Thanks you, it works. If I remove the NullSpaceCondition I get the correct > solution also with Neumann BC. > From my point of view, the linear system I get is not singular because > there is a unique solution. Do you agree ? > Yes, the non-uniqueness is for the Poisson prblem only. Thanks, Matt > Regards, > Olivier Bonnefon > > > > On 11/25/2014 03:01 PM, Matthew Knepley wrote: > > On Tue, Nov 25, 2014 at 4:08 AM, Olivier Bonnefon < > olivier.bonnefon at avignon.inra.fr> wrote: > >> Hello, >> >> Thank you for your answer, I agree with you. >> >> I have adapted the ex12 example for the system: >> >> -\nabla \nabla u + u + f = 0 in \Omega (1) >> >> with f = 4-x^2-y^2 >> The solution is still x^2-y^2. >> >> It consists in adding the following gradient function: >> >> void g0_uu(const PetscScalar u[], const PetscScalar u_t[], const >> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >> PetscScalar a_x[], const PetscReal x[], PetscScalar g0[]) >> { >> g0[0] = 1.0; >> } >> ... >> //and set gradient >> ierr = PetscDSSetJacobian(prob, 0, 0, g0_uu, NULL, NULL, >> g3_uu);CHKERRQ(ierr); >> ... >> Of course, I have modified the f0_u function. >> >> >> With Dirichlet BC, I get the solution. >> >> With Neumann BC, the solution is still shifted down (-2/3). The problem >> (1) with Neumann BC has a unique solution. >> How remove the condition \int_\Omega u = 0 ? >> > > That condition is one way of imposing boundary conditions for pressure. > When I call MatSetNullSpace(), I am implying > this condition. You can take out that call, but the system will be > singular. You would have to add a condition on the > pressure somehow, like fixing it at a point. This tends to make the system > more ill-conditioned, but if that is appropriate > for you, then you make a similar call to > > ierr = DMPlexAddBoundary(cdm, user->bcType == DIRICHLET, "wall", > user->bcType == NEUMANN ? "boundary" : "marker", 0, user->exactFuncs[0], 1, > &id, user);CHKERRQ(ierr); > > but for field 1. To test , you can jsut change 0 to 1 in this call. > However, this sets the entire boundary, and you really only > need to set the pressure at one point, so you could make another label > that only has a single point and use that instead. > > Thanks, > > Matt > > >> Thanks >> Olivier Bonnefon >> >> >> >> On 11/06/2014 07:14 AM, Matthew Knepley wrote: >> >> On Mon, Nov 3, 2014 at 9:46 AM, Olivier Bonnefon < >> olivier.bonnefon at avignon.inra.fr> wrote: >> >>> Hello, >>> >>> Thank for your answer, I'll explain my trouble: >>> >>> My problem is that the BC Neumann leads to wrong result. >>> >>> With Dirichlet BC, I get: >>> ------------------------------- >>> >>> > ./ex12 -run_type full -refinement_limit 0.0 -bc_type dirichlet >>> -interpolate 0 -petscspace_order 1 -show_initial -show_solution >>> -dm_plex_print_fem 1 >>> ... >>> ... >>> Solution >>> Vec Object:potential 1 MPI processes >>> type: seq >>> 0.5 >>> >>> This result is correct. >>> >>> With Neuman BC, I get: >>> -------------------------------- >>> >>> > ./ex12 -run_type full -refinement_limit 0.0 -bc_type neumann >>> -interpolate 1 -petscspace_order 1 -show_initial -dm_plex_print_fem 1 >>> -show_solution -bd_petscspace_order 1 -snes_linesearch_monitor >>> -snes_monitor -ksp_monitor_true_residual -snes_converged_reason >>> -ksp_converged_reason >>> .... >>> .... >>> >>> >>> Solution >>> Vec Object:potential 1 MPI processes >>> type: seq >>> -0.75 >>> -0.583333 >>> 0.0833333 >>> -0.583333 >>> -0.333333 >>> 0.416667 >>> 0.0833333 >>> 0.416667 >>> 1.25 >>> >>> >>> That is not the values of the solution x*x+y*y. >>> >> >> Sorry, this is poor documentation on my part. I will fix it in the >> example. The Naumann solution >> explicitly discards the constant part, meaning >> >> \int_\Omega u = 0 >> >> If we look at my solution >> >> \int^1_0 dx \int^1_0 dy x^2 + y^2 = 2/3 >> >> However, this is for the continuum result, whereas we are enforcing >> this in the discrete case. >> Thus, what we really get is 0.75, since we are integrating linear patches >> instead of quadratic >> patches. After shifting by that, you still do not get exactly x^2 + y^2 >> since there is some >> discretization error. If you run with P2 you should get the exact answer, >> shifted down to eliminate >> the DC component. >> >> Thanks, >> >> Matt >> >> >>> >>> I tried many ksp options. >>> Moreover, the neumann BC with "-run_type full" is not cover in the list >>> https://bitbucket.org/petsc/petsc/src/fced3c3f9e703542693913793d15321603e40fe6/config/builder.py?at=master#cl-257 >>> >>> Do you know what is wrong ? >>> >>> Thanks, >>> >>> Olivier Bonnefon >>> >>> On 10/31/2014 06:50 PM, Matthew Knepley wrote: >>> >>> On Fri, Oct 31, 2014 at 10:43 AM, Olivier Bonnefon < >>> olivier.bonnefon at avignon.inra.fr> wrote: >>> >>>> Hello, >>>> >>>> I'm working on the snes/examples/tutorial/ex12 version 3.5.2. >>>> >>>> I didn't succed to run the simplest case with Neumann BC: >>>> >>>> ./ex12 -run_type full -refinement_limit 0.0 -bc_type neumann >>>> -interpolate 1 -petscspace_order 1 -show_initial -dm_plex_print_fem 1 >>>> -show_solution -bd_petscspace_order 1 -snes_linesearch_monitor >>>> -snes_monitor -ksp_monitor_true_residual -snes_converged_reason >>>> -ksp_converged_reason >>>> >>>> This leads to dofs negatives values. >>>> >>> >>> I do not understand what you mean. Please always mail the full error >>> message. >>> >>> >>>> Do you know the options to get a correct result with Neumann BC ? >>>> >>> >>> There are some tests here: >>> >>> >>> https://bitbucket.org/petsc/petsc/src/fced3c3f9e703542693913793d15321603e40fe6/config/builder.py?at=master#cl-257 >>> >>> Matt >>> >>> >>>> Regards, >>>> Olivier Bonnefon >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> >>> -- >>> Olivier Bonnefon >>> INRA PACA-Avignon, Unit? BioSP >>> Tel: +33 (0)4 32 72 21 58 >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> >> -- >> Olivier Bonnefon >> INRA PACA-Avignon, Unit? BioSP >> Tel: +33 (0)4 32 72 21 58 >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Olivier Bonnefon > INRA PACA-Avignon, Unit? BioSP > Tel: +33 (0)4 32 72 21 58 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Nov 27 11:30:31 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 27 Nov 2014 11:30:31 -0600 Subject: [petsc-users] Problem with PETSc + HDF5 VecView In-Reply-To: <87y4qxyyt1.fsf@jedbrown.org> References: <547382AA.10705@hakostra.net> <5474E7B8.1090108@hakostra.net> <54757AB0.6070100@hakostra.net> <5475C6E8.3090609@hakostra.net> <547647BB.1080103@hakostra.net> <87y4qxyyt1.fsf@jedbrown.org> Message-ID: > On Nov 26, 2014, at 11:53 PM, Jed Brown wrote: > > Barry Smith writes: > >>> On Nov 26, 2014, at 3:35 PM, H?kon Strandenes wrote: >>> >>>> >>> Is it really PETSc's taks to warn about this? PETSc should trust HDF5 to "just work" and HDF5 should actually print sensible warnings/error messages. Shouldn't it? >> >> Yes, but if we produce a nice error message it makes everyone's >> lives easier, including ours because we don't have to constantly >> answer emails about the same problem discovered by a new person >> over and over again. Hence we do this kind of thing a lot. > > The only concern is that the user might not exceed that depth, so why > should they have to set an arbitrary and excessively big value? We could check the value of the environmental variable and print the error message when the error occurs and is returned up the HDF5 stack only. > > MPI_REQUEST_MAX and MPI_COMM_MAX also look like candidates for needing > increases. > > Whoever is responsible for this perversion should be fired. From jed at jedbrown.org Thu Nov 27 11:36:25 2014 From: jed at jedbrown.org (Jed Brown) Date: Thu, 27 Nov 2014 10:36:25 -0700 Subject: [petsc-users] Problem with PETSc + HDF5 VecView In-Reply-To: References: <547382AA.10705@hakostra.net> <5474E7B8.1090108@hakostra.net> <54757AB0.6070100@hakostra.net> <5475C6E8.3090609@hakostra.net> <547647BB.1080103@hakostra.net> <87y4qxyyt1.fsf@jedbrown.org> Message-ID: <87ioi0zgti.fsf@jedbrown.org> Barry Smith writes: >> The only concern is that the user might not exceed that depth, so why >> should they have to set an arbitrary and excessively big value? > > We could check the value of the environmental variable and print the error message when the error occurs and is returned up the HDF5 stack only. Perfect. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From Eric.Chamberland at giref.ulaval.ca Thu Nov 27 12:52:36 2014 From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland) Date: Thu, 27 Nov 2014 13:52:36 -0500 Subject: [petsc-users] Cholesky: matrix ordering with aij? Bug with debugging=1 for ordering on sbaij? Message-ID: <547772F4.3080808@giref.ulaval.ca> Hi, Using petsc 3.5.2. ------------------------ Question #1: ------------------------ I configured a cholesky like this: KSP Object:(o_slin) 1 MPI processes type: preonly maximum iterations=4000, initial guess is zero tolerances: relative=1e-12, absolute=1e-20, divergence=1e+30 left preconditioning using NONE norm type for convergence test PC Object:(o_slin) 1 MPI processes type: cholesky Cholesky: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5, needed 2.59172 Factored matrix follows: Mat Object: 1 MPI processes type: seqsbaij rows=693, cols=693 package used to perform factorization: petsc total: nonzeros=10328, allocated nonzeros=10328 total number of mallocs used during MatSetValues calls =0 block size is 1 linear system matrix = precond matrix: Mat Object: (o_slin) 1 MPI processes type: seqaij rows=693, cols=693 total: nonzeros=7277, allocated nonzeros=7345 total number of mallocs used during MatSetValues calls =0 not using I-node routines with a "mat_type aij" for the assembled matrix and "pc_factor_mat_ordering_type nd". The PC creates an "sbaij". When/where is the "matrix ordering: nd" used since it is documented to not works on sbaij matrix? Compare the number of non-zeros for the factored matrix obtained with a "natural ordering": KSP Object:(o_slin) 1 MPI processes type: preonly maximum iterations=4000, initial guess is zero tolerances: relative=1e-12, absolute=1e-20, divergence=1e+30 left preconditioning using NONE norm type for convergence test PC Object:(o_slin) 1 MPI processes type: cholesky Cholesky: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 5, needed 44.1099 Factored matrix follows: Mat Object: 1 MPI processes type: seqsbaij rows=693, cols=693 package used to perform factorization: petsc total: nonzeros=175778, allocated nonzeros=175778 total number of mallocs used during MatSetValues calls =0 block size is 1 linear system matrix = precond matrix: Mat Object: (o_slin) 1 MPI processes type: seqaij rows=693, cols=693 total: nonzeros=7277, allocated nonzeros=7345 total number of mallocs used during MatSetValues calls =0 not using I-node routines there is really something happening with "nd"... ------------------------ Question #2: ------------------------ Using petsc 3.5.2 compiled with "--with-debugging=no", if a try to use "mat_type sbaij", I get this friendly error: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Matrix reordering is not supported for sbaij matrix. Use aij format [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 [0]PETSC ERROR: /home/mefpp_ericc/depots_prepush/GIREF/bin/probDiffusion.dev on a arch-linux2-c-opt named melkor by eric Thu Nov 27 13:35:01 2014 [0]PETSC ERROR: Configure options --with-mpi-compilers=1 --with-make-np=12 --prefix=/opt/petsc-3.5.2_opt_openmpi-1.6.5_mkl_mt --with-mpi-dir=/opt/openmpi-1.6.5 --with-debugging=no --download-ml=yes --with-mkl_pardiso=1 --with-mkl_pardiso-dir=/opt/intel/composerxe/mkl --download-mumps=yes --download-superlu=yes --download-superlu_dist=yes --download-parmetis=yes --download-metis=yes --download-hypre=yes --with-shared-libraries=1 --with-scalapack=1 --with-scalapack-include=/opt/intel/composerxe/mkl/include --with-scalapack-lib="-L/opt/intel/composerxe/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-blas-lapack-dir=/opt/intel/composerxe/mkl/lib/intel64 [0]PETSC ERROR: #1 MatCholeskyFactorSymbolic_SeqSBAIJ() line 245 in /home/mefpp_ericc/petsc-3.5.2/src/mat/impls/sbaij/seq/sbaijfact.c [0]PETSC ERROR: #2 MatCholeskyFactorSymbolic() line 3012 in /home/mefpp_ericc/petsc-3.5.2/src/mat/interface/matrix.c [0]PETSC ERROR: #3 PCSetUp_Cholesky() line 124 in /home/mefpp_ericc/petsc-3.5.2/src/ksp/pc/impls/factor/cholesky/cholesky.c [0]PETSC ERROR: #4 PCSetUp() line 902 in /home/mefpp_ericc/petsc-3.5.2/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #5 KSPSetUp() line 305 in /home/mefpp_ericc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c But if i use petsc compiled with "--with-debugging=yes", I got this one: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Index set is not a permutation [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 [0]PETSC ERROR: probDiffusion.dev on a arch-linux2-c-debug named melkor by eric Thu Nov 27 13:45:04 2014 [0]PETSC ERROR: Configure options --with-mpi-compilers=1 --with-make-np=12 --prefix=/opt/petsc-3.5.2_debug_openmpi-1.6.5_mkl_mt --with-mpi-dir=/opt/openmpi-1.6.5 --with-debugging=yes --download-ml=yes --with-mkl_pardiso=1 --with-mkl_pardiso-dir=/opt/intel/composerxe/mkl --download-mumps=yes --download-superlu=yes --download-superlu_dist=yes --download-parmetis=yes --download-metis=yes --download-hypre=yes --with-shared-libraries=1 --with-scalapack=1 --with-scalapack-include=/opt/intel/composerxe/mkl/include --with-scalapack-lib="-L/opt/intel/composerxe/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-blas-lapack-dir=/opt/intel/composerxe/mkl/lib/intel64 [0]PETSC ERROR: #1 ISSetPermutation() line 183 in /home/mefpp_ericc/petsc-3.5.2/src/vec/is/is/interface/index.c [0]PETSC ERROR: #2 MatGetOrdering() line 261 in /home/mefpp_ericc/petsc-3.5.2/src/mat/order/sorder.c [0]PETSC ERROR: #3 PCSetUp_Cholesky() line 107 in /home/mefpp_ericc/petsc-3.5.2/src/ksp/pc/impls/factor/cholesky/cholesky.c [0]PETSC ERROR: #4 PCSetUp() line 902 in /home/mefpp_ericc/petsc-3.5.2/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #5 KSPSetUp() line 305 in /home/mefpp_ericc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c Is this the expected behavior? Thanks, Eric From bsmith at mcs.anl.gov Thu Nov 27 13:22:10 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 27 Nov 2014 13:22:10 -0600 Subject: [petsc-users] Cholesky: matrix ordering with aij? Bug with debugging=1 for ordering on sbaij? In-Reply-To: <547772F4.3080808@giref.ulaval.ca> References: <547772F4.3080808@giref.ulaval.ca> Message-ID: > On Nov 27, 2014, at 12:52 PM, Eric Chamberland wrote: > > Hi, > > Using petsc 3.5.2. > > ------------------------ > Question #1: > ------------------------ > > I configured a cholesky like this: > > KSP Object:(o_slin) 1 MPI processes > type: preonly > maximum iterations=4000, initial guess is zero > tolerances: relative=1e-12, absolute=1e-20, divergence=1e+30 > left preconditioning > using NONE norm type for convergence test > PC Object:(o_slin) 1 MPI processes > type: cholesky > Cholesky: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5, needed 2.59172 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqsbaij > rows=693, cols=693 > package used to perform factorization: petsc > total: nonzeros=10328, allocated nonzeros=10328 > total number of mallocs used during MatSetValues calls =0 > block size is 1 > linear system matrix = precond matrix: > Mat Object: (o_slin) 1 MPI processes > type: seqaij > rows=693, cols=693 > total: nonzeros=7277, allocated nonzeros=7345 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > with a "mat_type aij" for the assembled matrix and "pc_factor_mat_ordering_type nd". The PC creates an "sbaij". > > When/where is the "matrix ordering: nd" used since it is documented to not works on sbaij matrix? The factorization is done on the aij matrix and the result put into a sbaij matrix. Since with aij it is easy to reorder rows/columns it can do the factorization in any order it wants. sbaij only stores the upper half of the matrix so factorizing in any ordering besides "natural" is nasty and we don't do it. > > Compare the number of non-zeros for the factored matrix obtained with a "natural ordering": > > KSP Object:(o_slin) 1 MPI processes > type: preonly > maximum iterations=4000, initial guess is zero > tolerances: relative=1e-12, absolute=1e-20, divergence=1e+30 > left preconditioning > using NONE norm type for convergence test > PC Object:(o_slin) 1 MPI processes > type: cholesky > Cholesky: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 5, needed 44.1099 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqsbaij > rows=693, cols=693 > package used to perform factorization: petsc > total: nonzeros=175778, allocated nonzeros=175778 > total number of mallocs used during MatSetValues calls =0 > block size is 1 > linear system matrix = precond matrix: > Mat Object: (o_slin) 1 MPI processes > type: seqaij > rows=693, cols=693 > total: nonzeros=7277, allocated nonzeros=7345 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > there is really something happening with "nd"... > > ------------------------ > Question #2: > ------------------------ > Using petsc 3.5.2 compiled with "--with-debugging=no", if a try to use "mat_type sbaij", I get this friendly error: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Matrix reordering is not supported for sbaij matrix. Use aij format > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > [0]PETSC ERROR: /home/mefpp_ericc/depots_prepush/GIREF/bin/probDiffusion.dev on a arch-linux2-c-opt named melkor by eric Thu Nov 27 13:35:01 2014 > [0]PETSC ERROR: Configure options --with-mpi-compilers=1 --with-make-np=12 --prefix=/opt/petsc-3.5.2_opt_openmpi-1.6.5_mkl_mt --with-mpi-dir=/opt/openmpi-1.6.5 --with-debugging=no --download-ml=yes --with-mkl_pardiso=1 --with-mkl_pardiso-dir=/opt/intel/composerxe/mkl --download-mumps=yes --download-superlu=yes --download-superlu_dist=yes --download-parmetis=yes --download-metis=yes --download-hypre=yes --with-shared-libraries=1 --with-scalapack=1 --with-scalapack-include=/opt/intel/composerxe/mkl/include --with-scalapack-lib="-L/opt/intel/composerxe/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-blas-lapack-dir=/opt/intel/composerxe/mkl/lib/intel64 > [0]PETSC ERROR: #1 MatCholeskyFactorSymbolic_SeqSBAIJ() line 245 in /home/mefpp_ericc/petsc-3.5.2/src/mat/impls/sbaij/seq/sbaijfact.c > [0]PETSC ERROR: #2 MatCholeskyFactorSymbolic() line 3012 in /home/mefpp_ericc/petsc-3.5.2/src/mat/interface/matrix.c > [0]PETSC ERROR: #3 PCSetUp_Cholesky() line 124 in /home/mefpp_ericc/petsc-3.5.2/src/ksp/pc/impls/factor/cholesky/cholesky.c > [0]PETSC ERROR: #4 PCSetUp() line 902 in /home/mefpp_ericc/petsc-3.5.2/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSPSetUp() line 305 in /home/mefpp_ericc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c > > But if i use petsc compiled with "--with-debugging=yes", I got this one: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Index set is not a permutation > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > [0]PETSC ERROR: probDiffusion.dev on a arch-linux2-c-debug named melkor by eric Thu Nov 27 13:45:04 2014 > [0]PETSC ERROR: Configure options --with-mpi-compilers=1 --with-make-np=12 --prefix=/opt/petsc-3.5.2_debug_openmpi-1.6.5_mkl_mt --with-mpi-dir=/opt/openmpi-1.6.5 --with-debugging=yes --download-ml=yes --with-mkl_pardiso=1 --with-mkl_pardiso-dir=/opt/intel/composerxe/mkl --download-mumps=yes --download-superlu=yes --download-superlu_dist=yes --download-parmetis=yes --download-metis=yes --download-hypre=yes --with-shared-libraries=1 --with-scalapack=1 --with-scalapack-include=/opt/intel/composerxe/mkl/include --with-scalapack-lib="-L/opt/intel/composerxe/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-blas-lapack-dir=/opt/intel/composerxe/mkl/lib/intel64 > [0]PETSC ERROR: #1 ISSetPermutation() line 183 in /home/mefpp_ericc/petsc-3.5.2/src/vec/is/is/interface/index.c > [0]PETSC ERROR: #2 MatGetOrdering() line 261 in /home/mefpp_ericc/petsc-3.5.2/src/mat/order/sorder.c > [0]PETSC ERROR: #3 PCSetUp_Cholesky() line 107 in /home/mefpp_ericc/petsc-3.5.2/src/ksp/pc/impls/factor/cholesky/cholesky.c > [0]PETSC ERROR: #4 PCSetUp() line 902 in /home/mefpp_ericc/petsc-3.5.2/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSPSetUp() line 305 in /home/mefpp_ericc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c > > Is this the expected behavior? Well kind of. When debugging is turned off the the error checking that an ordering is actually a permutation is turned off hence the code gets further (until it cannot do the factorization). Hence two different error messages. Now I suspect that it is generating a bad ordering with the sbaij matrix so probably it would be best if we errored out in the MatGetOrdering() always. But even in the current situation nothing is really going wrong. Barry > > Thanks, > > Eric From Eric.Chamberland at giref.ulaval.ca Thu Nov 27 13:59:50 2014 From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland) Date: Thu, 27 Nov 2014 14:59:50 -0500 Subject: [petsc-users] Cholesky: matrix ordering with aij? Bug with debugging=1 for ordering on sbaij? In-Reply-To: References: <547772F4.3080808@giref.ulaval.ca> Message-ID: <547782B6.5010104@giref.ulaval.ca> On 11/27/2014 02:22 PM, Barry Smith wrote: >> When/where is the "matrix ordering: nd" used since it is documented to not works on sbaij matrix? > > The factorization is done on the aij matrix and the result put into a sbaij matrix. Since with aij it is easy to reorder rows/columns it can do the factorization in any order it wants. sbaij only stores the upper half of the matrix so factorizing in any ordering besides "natural" is nasty and we don't do it. ok! So the assembled matrix is permuted then returned to it's original ordering or a temporary copy is made? (answer can point into the code) >> >> Is this the expected behavior? > > Well kind of. When debugging is turned off the the error checking that an ordering is actually a permutation is turned off hence the code gets further (until it cannot do the factorization). Hence two different error messages. > > Now I suspect that it is generating a bad ordering with the sbaij matrix so probably it would be best if we errored out in the MatGetOrdering() always. But even in the current situation nothing is really going wrong. > > Barry understood. Thanks again! Eric From bsmith at mcs.anl.gov Thu Nov 27 17:28:00 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 27 Nov 2014 17:28:00 -0600 Subject: [petsc-users] Cholesky: matrix ordering with aij? Bug with debugging=1 for ordering on sbaij? In-Reply-To: <547782B6.5010104@giref.ulaval.ca> References: <547772F4.3080808@giref.ulaval.ca> <547782B6.5010104@giref.ulaval.ca> Message-ID: <3FFDF5A4-20E3-43CB-9274-2C99997E9F76@mcs.anl.gov> > On Nov 27, 2014, at 1:59 PM, Eric Chamberland wrote: > > On 11/27/2014 02:22 PM, Barry Smith wrote: >>> When/where is the "matrix ordering: nd" used since it is documented to not works on sbaij matrix? >> >> The factorization is done on the aij matrix and the result put into a sbaij matrix. Since with aij it is easy to reorder rows/columns it can do the factorization in any order it wants. sbaij only stores the upper half of the matrix so factorizing in any ordering besides "natural" is nasty and we don't do it. > > ok! So the assembled matrix is permuted then returned to it's original ordering or a temporary copy is made? (answer can point into the code) factored sparse matrices are never actually permuted around to a different ordering and then factored. Instead the factorization process uses the original data and ordering as it does the factorization. For example if the 5th row is the 1st row in the new ordering then the 5th row is used to produce the 1st row of the factored matrix. > >>> >>> Is this the expected behavior? >> >> Well kind of. When debugging is turned off the the error checking that an ordering is actually a permutation is turned off hence the code gets further (until it cannot do the factorization). Hence two different error messages. >> >> Now I suspect that it is generating a bad ordering with the sbaij matrix so probably it would be best if we errored out in the MatGetOrdering() always. But even in the current situation nothing is really going wrong. >> >> Barry > > understood. > > Thanks again! > > Eric > > From friedmud at gmail.com Sat Nov 29 20:37:26 2014 From: friedmud at gmail.com (Derek Gaston) Date: Sat, 29 Nov 2014 21:37:26 -0500 Subject: [petsc-users] wrong no of nonzeros Message-ID: I am changing the stencil of my preconditioning matrix in the middle of a solve a solve... and I'm running into this: [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Cannot reuse matrix. wrong no of nonzeros [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014 [0]PETSC ERROR: ../bison-opt on a arch-darwin-c-opt named dereksmacpro.home by gastdr Sat Nov 29 21:15:28 2014 [0]PETSC ERROR: Configure options --prefix=/opt/moose/petsc/openmpi_petsc-3.5.1/clang-opt-superlu --with-hypre-dir=/opt/moose/hypre/openmpi_hypre-2.8.0b/clang-opt --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 F90=mpif90 CFLAGS="-fPIC -fopenmp" CXXFLAGS="-fPIC -fopenmp" FFLAGS="-fPIC -fopenmp" FCFLAGS="-fPIC -fopenmp" F90FLAGS="-fPIC -fopenmp" F77FLAGS="-fPIC -fopenmp" PETSC_DIR=/opt/moose/stack_src/petsc-3.5.1 [0]PETSC ERROR: #1 MatGetSubMatrices_MPIAIJ_Local() line 1312 in /opt/moose/stack_src/petsc-3.5.1/src/mat/impls/aij/mpi/mpiov.c [0]PETSC ERROR: #2 MatGetSubMatrices_MPIAIJ() line 790 in /opt/moose/stack_src/petsc-3.5.1/src/mat/impls/aij/mpi/mpiov.c [0]PETSC ERROR: #3 MatGetSubMatrices() line 6352 in /opt/moose/stack_src/petsc-3.5.1/src/mat/interface/matrix.c [0]PETSC ERROR: #4 PCSetUp_ASM() line 369 in /opt/moose/stack_src/petsc-3.5.1/src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: #5 PCSetUp() line 902 in /opt/moose/stack_src/petsc-3.5.1/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #6 KSPSetUp() line 305 in /opt/moose/stack_src/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #7 KSPSolve() line 417 in /opt/moose/stack_src/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 232 in /opt/moose/stack_src/petsc-3.5.1/src/snes/impls/ls/ls.c [0]PETSC ERROR: #9 SNESSolve() line 3743 in /opt/moose/stack_src/petsc-3.5.1/src/snes/interface/snes.c [0]PETSC ERROR: #10 solve() line 559 in src/solvers/petsc_nonlinear_solver.C I know that in the past we used to have to specify SAME_NONZERO_PATTERN or DIFFERENT_NONZERO_PATTERN... but I thought those times were all past us ;-) I'm using PETSc 3.5.1 on OSX ... underneath libMesh and MOOSE of course... Any ideas where to start digging? Thanks! Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sat Nov 29 20:47:09 2014 From: jed at jedbrown.org (Jed Brown) Date: Sat, 29 Nov 2014 19:47:09 -0700 Subject: [petsc-users] wrong no of nonzeros In-Reply-To: References: Message-ID: <87zjb9h0b6.fsf@jedbrown.org> Derek Gaston writes: > I am changing the stencil of my preconditioning matrix in the middle of a > solve a solve... and I'm running into this: Can you PCReset() after changing the stencil? The implementation of MatGetSubMatrices_MPIAIJ caches some intermediate structures to speed up subsequent calls. I'll grant that Mat should do a better job of remembering what has changed so that it can automatically invalidate caches in this case. There are only two things hard in computer science... -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sat Nov 29 20:53:17 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 29 Nov 2014 20:53:17 -0600 Subject: [petsc-users] wrong no of nonzeros In-Reply-To: References: Message-ID: Can you please rerun with the maint branch of PETSc (this is the release version with all fixes since the release). The line numbers you report do not match up with maint. From the code: (slightly different line numbers) } else { /* Destroy the blocks from the previous iteration */ if (pc->flag == DIFFERENT_NONZERO_PATTERN) { ierr = MatDestroyMatrices(osm->n_local_true,&osm->pmat);CHKERRQ(ierr); scall = MAT_INITIAL_MATRIX; } } /* Extract out the submatrices */ ierr = MatGetSubMatrices(pc->pmat,osm->n_local_true,osm->is,osm->is,scall,&osm->pmat);CHKERRQ(ierr); What is suppose to have happened is that the pc->flag would have been set to DIFFERENT_NONZERO_PATTERN and hence scall reset to to MAT_INITIAIL_MATRIX (hence you should never get to the state you are in). Now it is possible we introduced a bug but we need to debug it off of maint; not some outdated 3.5.1 Thanks Barry BTW: Ignore Jed's gibberish about PCReset(), you've got no business messing with that. > On Nov 29, 2014, at 8:37 PM, Derek Gaston wrote: > > I am changing the stencil of my preconditioning matrix in the middle of a solve a solve... and I'm running into this: > > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Cannot reuse matrix. wrong no of nonzeros > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.1, Jul, 24, 2014 > [0]PETSC ERROR: ../bison-opt on a arch-darwin-c-opt named dereksmacpro.home by gastdr Sat Nov 29 21:15:28 2014 > [0]PETSC ERROR: Configure options --prefix=/opt/moose/petsc/openmpi_petsc-3.5.1/clang-opt-superlu --with-hypre-dir=/opt/moose/hypre/openmpi_hypre-2.8.0b/clang-opt --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 F90=mpif90 CFLAGS="-fPIC -fopenmp" CXXFLAGS="-fPIC -fopenmp" FFLAGS="-fPIC -fopenmp" FCFLAGS="-fPIC -fopenmp" F90FLAGS="-fPIC -fopenmp" F77FLAGS="-fPIC -fopenmp" PETSC_DIR=/opt/moose/stack_src/petsc-3.5.1 > [0]PETSC ERROR: #1 MatGetSubMatrices_MPIAIJ_Local() line 1312 in /opt/moose/stack_src/petsc-3.5.1/src/mat/impls/aij/mpi/mpiov.c > [0]PETSC ERROR: #2 MatGetSubMatrices_MPIAIJ() line 790 in /opt/moose/stack_src/petsc-3.5.1/src/mat/impls/aij/mpi/mpiov.c > [0]PETSC ERROR: #3 MatGetSubMatrices() line 6352 in /opt/moose/stack_src/petsc-3.5.1/src/mat/interface/matrix.c > [0]PETSC ERROR: #4 PCSetUp_ASM() line 369 in /opt/moose/stack_src/petsc-3.5.1/src/ksp/pc/impls/asm/asm.c > [0]PETSC ERROR: #5 PCSetUp() line 902 in /opt/moose/stack_src/petsc-3.5.1/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #6 KSPSetUp() line 305 in /opt/moose/stack_src/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #7 KSPSolve() line 417 in /opt/moose/stack_src/petsc-3.5.1/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 232 in /opt/moose/stack_src/petsc-3.5.1/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #9 SNESSolve() line 3743 in /opt/moose/stack_src/petsc-3.5.1/src/snes/interface/snes.c > [0]PETSC ERROR: #10 solve() line 559 in src/solvers/petsc_nonlinear_solver.C > > > I know that in the past we used to have to specify SAME_NONZERO_PATTERN or DIFFERENT_NONZERO_PATTERN... but I thought those times were all past us ;-) > > I'm using PETSc 3.5.1 on OSX ... underneath libMesh and MOOSE of course... > > Any ideas where to start digging? > > Thanks! > Derek From friedmud at gmail.com Sat Nov 29 20:58:50 2014 From: friedmud at gmail.com (Derek Gaston) Date: Sat, 29 Nov 2014 21:58:50 -0500 Subject: [petsc-users] wrong no of nonzeros In-Reply-To: <87zjb9h0b6.fsf@jedbrown.org> References: <87zjb9h0b6.fsf@jedbrown.org> Message-ID: I'll try tomorrow - thanks Jed! Derek On Saturday, November 29, 2014, Jed Brown wrote: > Derek Gaston > writes: > > > I am changing the stencil of my preconditioning matrix in the middle of a > > solve a solve... and I'm running into this: > > Can you PCReset() after changing the stencil? > > The implementation of MatGetSubMatrices_MPIAIJ caches some intermediate > structures to speed up subsequent calls. I'll grant that Mat should do > a better job of remembering what has changed so that it can > automatically invalidate caches in this case. There are only two things > hard in computer science... > -- Sent from my iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedmud at gmail.com Sat Nov 29 21:03:04 2014 From: friedmud at gmail.com (Derek Gaston) Date: Sat, 29 Nov 2014 22:03:04 -0500 Subject: [petsc-users] wrong no of nonzeros In-Reply-To: References: Message-ID: On Saturday, November 29, 2014, Barry Smith wrote: > > Can you please rerun with the maint branch of PETSc Will do. > What is suppose to have happened is that the pc->flag would have been set > to DIFFERENT_NONZERO_PATTERN and hence scall reset to to > MAT_INITIAIL_MATRIX (hence you should never get to the state you are in). Good to know! > BTW: Ignore Jed's gibberish about PCReset(), you've got no business > messing with that. Hah - will do. :-) Derek -- Sent from my iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sat Nov 29 22:24:56 2014 From: jed at jedbrown.org (Jed Brown) Date: Sat, 29 Nov 2014 21:24:56 -0700 Subject: [petsc-users] wrong no of nonzeros In-Reply-To: References: Message-ID: <87wq6dgvs7.fsf@jedbrown.org> Derek Gaston writes: >> BTW: Ignore Jed's gibberish about PCReset(), you've got no business >> messing with that. > > > Hah - will do. :-) Yeah, my reply was premature and crappy. This should be handled correctly without shenanigans. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From paulhuaizhang at uky.edu Sun Nov 30 11:33:09 2014 From: paulhuaizhang at uky.edu (UK) Date: Sun, 30 Nov 2014 12:33:09 -0500 Subject: [petsc-users] initial guess Message-ID: Hello, I am using KSPFGMRES as my linear solver for Ax=b system. I was wondering how the initial guess will influence the computation efficiency. Does PETSc always starts with xo=0 or some value else? If I use my previous solution as the initial guess for current computation is it going to take less effort for PETSc to solve the system? (my configuration attached) Thanks, Paul MatCreateMPIAIJ( PETSC_COMM_WORLD, grid[gid].cellCount*nVars, grid[gid].cellCount*nVars, grid[gid].globalCellCount*nVars, grid[gid].globalCellCount*nVars, 0,&diagonal_nonzeros[0], 0,&off_diagonal_nonzeros[0], &impOP); KSPSetOperators(ksp,impOP,impOP,SAME_NONZERO_PATTERN); KSPSetTolerances(ksp,rtol,abstol,1.e15,maxits); KSPSetInitialGuessKnoll(ksp,PETSC_TRUE); KSPSetType(ksp,KSPFGMRES); Huaibao (Paul) Zhang *Gas Surface Interactions Lab* Department of Mechanical Engineering University of Kentucky, Lexington, KY, 40506-0503 *Office*: 216 Ralph G. Anderson Building *Web*:gsil.engineering.uky.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Nov 30 11:48:41 2014 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 30 Nov 2014 11:48:41 -0600 Subject: [petsc-users] initial guess In-Reply-To: References: Message-ID: By default PETSc zeros x at the beginning of the linear solve*. If you call KSPSetInitialGuessNonzero() then PETSc does NOT zero x and uses whatever value is in x as the starting point. Note you cannot use both KSPSetInitialGuessKnoll() and KSPSetInitialGuessNonzero(). Whether using some "previous" solution as an initial guess depends on the problem you are solving and how you are solving it. For example, if you are using Newton's Method then using the previous initial guess is just plain silly because Newton's method solves for the correction. In general if what you are doing requires a linear solve for a "correction" then using the nonzero initial guess is not the way to go. Barry * Unless you are using KSPSetInitialGuessKnoll() then it starts with xo = preconditioner (applied to) b > On Nov 30, 2014, at 11:33 AM, UK wrote: > > Hello, > > I am using KSPFGMRES as my linear solver for Ax=b system. I was wondering how the initial guess will influence the computation efficiency. Does PETSc always starts with xo=0 or some value else? If I use my previous solution as the initial guess for current computation is it going to take less effort for PETSc to solve the system? (my configuration attached) > > Thanks, > Paul > > MatCreateMPIAIJ( > PETSC_COMM_WORLD, > grid[gid].cellCount*nVars, > grid[gid].cellCount*nVars, > grid[gid].globalCellCount*nVars, > grid[gid].globalCellCount*nVars, > 0,&diagonal_nonzeros[0], > 0,&off_diagonal_nonzeros[0], > &impOP); > > KSPSetOperators(ksp,impOP,impOP,SAME_NONZERO_PATTERN); > KSPSetTolerances(ksp,rtol,abstol,1.e15,maxits); > KSPSetInitialGuessKnoll(ksp,PETSC_TRUE); > KSPSetType(ksp,KSPFGMRES); > > > > Huaibao (Paul) Zhang > Gas Surface Interactions Lab > Department of Mechanical Engineering > University of Kentucky, > Lexington, > KY, 40506-0503 > Office: 216 Ralph G. Anderson Building > Web:gsil.engineering.uky.edu From paulhuaizhang at gmail.com Sun Nov 30 20:54:37 2014 From: paulhuaizhang at gmail.com (paul zhang) Date: Sun, 30 Nov 2014 21:54:37 -0500 Subject: [petsc-users] running error Message-ID: Hello, I compiled the newest version of petsc-3.5.2 and tried to test an example code with it. I could managed the compilation, but as I ran the sample, some error popped out. Could you please have a check? (files attached. I did search the issue online, but ends up with no valid solution.) -------------------------------------------------------------------------- Error obtaining unique transport key from ORTE (orte_precondition_transports not present in the environment). Thanks, Paul Huaibao (Paul) Zhang *Gas Surface Interactions Lab* Department of Mechanical Engineering University of Kentucky, Lexington, KY, 40506-0503 *Office*: 216 Ralph G. Anderson Building *Web*:gsil.engineering.uky.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- set (CMAKE_CXX_COMPILER mpiCC) set (CMAKE_CXX_FLAGS "-O3") set (PETSC_INCLUDE_DIRS1 /home/hzh225/LIB_CFD/nP/petsc-3.5.2/include) set (PETSC_INCLUDE_DIRS2 /home/hzh225/LIB_CFD/nP/petsc-3.5.2/linux-gnu-intel/include) set (PETSC_LIBRARY_DIRS /home/hzh225/LIB_CFD/nP/petsc-3.5.2/linux-gnu-intel/lib) cmake_minimum_required(VERSION 2.6) project(kats) set (kats_VERSION_MAJOR 2) set (kats_VERSION_MINOR 0) list (APPEND CMAKE_MODULE_PATH "${kats_SOURCE_DIR}/CMake") # Pass some CMake settings to source code through a header file configure_file ( "${PROJECT_SOURCE_DIR}/cmake_vars.h.in" "${PROJECT_BINARY_DIR}/cmake_vars.h" ) set (CMAKE_INSTALL_PREFIX ${PROJECT_SOURCE_DIR}/../) # add to the include search path include_directories("${PROJECT_SOURCE_DIR}") include_directories(${PETSC_INCLUDE_DIRS1}) include_directories(${PETSC_INCLUDE_DIRS2}) link_directories(${PETSC_LIBRARY_DIRS}) #set (EXTRA_LIBS parmetis metis cgns petsc)# imf m) set (EXTRA_LIBS petsc ) #add the executable set (SOURCES main.cc cmake_vars.h ) add_executable(kats ${SOURCES}) target_link_libraries (kats ${FCFD_LIBS} ${EXTRA_LIBS}) install (TARGETS kats RUNTIME DESTINATION bin) -------------- next part -------------- A non-text attachment was scrubbed... Name: cmake_vars.h Type: text/x-chdr Size: 61 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: main.cc Type: text/x-c++src Size: 2363 bytes Desc: not available URL: From jed at jedbrown.org Sun Nov 30 21:01:39 2014 From: jed at jedbrown.org (Jed Brown) Date: Sun, 30 Nov 2014 20:01:39 -0700 Subject: [petsc-users] running error In-Reply-To: References: Message-ID: <87lhmsgjjg.fsf@jedbrown.org> paul zhang writes: > Hello, > > I compiled the newest version of petsc-3.5.2 and tried to test an example > code with it. I could managed the compilation, but as I ran the sample, > some error popped out. Could you please have a check? (files attached. I > did search the issue online, but ends up with no valid solution.) > > > -------------------------------------------------------------------------- > Error obtaining unique transport key from ORTE > (orte_precondition_transports not present in > the environment). Looks like an Open MPI/resource manager configuration issue. This thread has a workaround, but your facility should be notified so that they can provide a system-wide fix. http://comments.gmane.org/gmane.comp.clustering.open-mpi.user/14871 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Sun Nov 30 21:28:50 2014 From: jed at jedbrown.org (Jed Brown) Date: Sun, 30 Nov 2014 20:28:50 -0700 Subject: [petsc-users] running error In-Reply-To: References: <87lhmsgjjg.fsf@jedbrown.org> Message-ID: <87iohwgia5.fsf@jedbrown.org> Please keep the discussion on-list. paul zhang writes: > I set some breakpoints, which shows the code breaks down at the > PetscInitialize(&argc,&argv,(char *)0,help); Run in a debugger and send a stack trace. This is most likely due to a misconfigured environment/MPI. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: