From recrusader at gmail.com Sat Mar 1 13:12:12 2008 From: recrusader at gmail.com (Yujie) Date: Sat, 1 Mar 2008 11:12:12 -0800 Subject: about Dense Matrix Re: several problems about codes Re: how to add a parallel MatMatSolve() function? In-Reply-To: References: <7ff0ee010803011038q6d0fb498ne8d65d97f5711992@mail.gmail.com> Message-ID: <7ff0ee010803011112xc7b6b0ejd4e3a1987923358e@mail.gmail.com> However, you have realized A*B(A is AIJ, B is dense matrix) to my knowledge. PETSc is the Portable, Extensible Toolkit for Scientific computation, not the Sparse Portable, Extensible Toolkit for Scientific computation :). It is easy to realize A*B(A is dense matrix, B is AIJ)? Thanks, Yujie On 3/1/08, Matthew Knepley wrote: > > If you want dense matrices, I would suggest PLAPACK/FLAME instead of > PETSc. We > specialize in sparse matrices. > > Thanks, > > Matt > > > On Sat, Mar 1, 2008 at 12:38 PM, Yujie wrote: > > Dear Barry: > > > > I just find MatMatMult() doesn't support Dense Matrix. I want to do A*B > (A > > is dense matrix, B is AIJ). I think a possible method is to convert A to > > AIJ, however, it will take much more memory than using dense matrix, I > > think. Because AIJ needs to save positions and corresponding values. Do > you > > have any good idea? Thanks a lot. > > > > Regards, > > Yujie > > > > On 3/1/08, Barry Smith wrote: > > > > > > > > > > > > > > > On Mar 1, 2008, at 2:01 AM, Yujie wrote: > > > > > > Dear Barry: > > > > > > I can't understand several problems as follows about codes. Could you > give > > me some explanations? thanks a lot. > > > > > > 1. About MatGetArray() > > > I check the codes of PETSc. I find only matrix types, SeqAij and > MPIDense, > > support this fnction. If I want to use SuperLU_dist or Spooles. I need > to > > convert matrix A into corresponding type > > > before I call MatMatSolve(). If it is, MatGetArray() doesn't work, I > > think. > > > > > > MatGetArray() is called on B and X, it is not called on A. B and A > must > > be dense or the whole thing doesn't make > > > sense, since X will always end up being dense. > > > > > > > > > > > > > > > 2. A->hdr.comm should be A->comm? > > > > > > Yes, for your version of PETSc > > > > > > > > > > > > 3. > > > for (i=0; i > > ierr = VecPlaceArray(b,bb + i*m);CHKERRQ(ierr); > > > ierr = VecPlaceArray(x,xx + i*m);CHKERRQ(ierr); > > > ierr = MatSolve(A,b,x);CHKERRQ(ierr); > > > } > > > > > > should be > > > > > > for (i=0; i > > ierr = VecPlaceArray(b,bb + i*m);CHKERRQ(ierr); > > > ierr = MatSolve(A,b,x);CHKERRQ(ierr); > > > ierr = VecPlaceArray(xx + i*m,x);CHKERRQ(ierr); > > > } > > > MatRestoreArray(B,&xx); > > > > > > First replace b with bb+i*m, and solve Ax=b, final replace xx+i*m with > x. > > > > > > No, you misunderstand what VecPlaceArray() does. The code above will > solve > > the columns in the wrong place. > > > > > > > > > Yes, I did forget the MatRestoreArray(), thanks for pointing that out. > > > > > > > > > Barry > > > > > > > > > > > > > > > After finishing the loop, the code should call MatRestoreArray() to > > restore matrix B. > > > > > > Regards, > > > Yujie > > > > > > > > > > > > On 3/1/08, Barry Smith wrote: > > > > > > > > Please edit the file src/mat/interface/matrix.c and remove the > > > > function MatMatSolve(). Replace it with the following two functions, > > > > then run "make lib shared" in that directory. Please let us know at > > petsc-maint at mcs.anl.gov > > > > if it crashes or produces incorrect > > > > results. > > > > > > > > Barry > > > > > > > > > > > > > > > > #undef __FUNCT__ > > > > #define __FUNCT__ "MatMatSolve_Basic" > > > > PetscErrorCode PETSCMAT_DLLEXPORT MatMatSolve_Basic(Mat A,Mat B,Mat > X) > > > > { > > > > PetscErrorCode ierr; > > > > Vec b,x; > > > > PetscInt m,N,i; > > > > PetscScalar *bb,*xx; > > > > > > > > PetscFunctionBegin; > > > > ierr = MatGetArray(B,&bb);CHKERRQ(ierr); > > > > ierr = MatGetArray(X,&xx);CHKERRQ(ierr); > > > > ierr = MatGetLocalSize(B,&m,PETSC_NULL);CHKERRQ(ierr); /* number > > > > local rows */ > > > > ierr = MatGetSize(B,PETSC_NULL,&N);CHKERRQ(ierr); /* total > > > > columns in dense matrix */ > > > > ierr = VecCreateMPIWithArray(A- > > > > >hdr.comm,m,PETSC_DETERMINE,PETSC_NULL,&b);CHKERRQ(ierr); > > > > ierr = VecCreateMPIWithArray(A- > > > > >hdr.comm,m,PETSC_DETERMINE,PETSC_NULL,&x);CHKERRQ(ierr); > > > > for (i=0; i > > > ierr = VecPlaceArray(b,bb + i*m);CHKERRQ(ierr); > > > > ierr = VecPlaceArray(x,xx + i*m);CHKERRQ(ierr); > > > > ierr = MatSolve(A,b,x);CHKERRQ(ierr); > > > > } > > > > ierr = VecDestroy(b);CHKERRQ(ierr); > > > > ierr = VecDestroy(x);CHKERRQ(ierr); > > > > PetscFunctionReturn(0); > > > > } > > > > > > > > #undef __FUNCT__ > > > > #define __FUNCT__ "MatMatSolve" > > > > /*@ > > > > MatMatSolve - Solves A X = B, given a factored matrix. > > > > > > > > Collective on Mat > > > > > > > > Input Parameters: > > > > + mat - the factored matrix > > > > - B - the right-hand-side matrix (dense matrix) > > > > > > > > Output Parameter: > > > > . B - the result matrix (dense matrix) > > > > > > > > Notes: > > > > The matrices b and x cannot be the same. I.e., one cannot > > > > call MatMatSolve(A,x,x). > > > > > > > > Notes: > > > > Most users should usually employ the simplified KSP interface > for > > > > linear solvers > > > > instead of working directly with matrix algebra routines such as > > > > this. > > > > See, e.g., KSPCreate(). However KSP can only solve for one > vector > > > > (column of X) > > > > at a time. > > > > > > > > Level: developer > > > > > > > > Concepts: matrices^triangular solves > > > > > > > > .seealso: MatMatSolveAdd(), MatMatSolveTranspose(), > > > > MatMatSolveTransposeAdd(), MatLUFactor(), MatCholeskyFactor() > > > > @*/ > > > > PetscErrorCode PETSCMAT_DLLEXPORT MatMatSolve(Mat A,Mat B,Mat X) > > > > { > > > > PetscErrorCode ierr; > > > > > > > > PetscFunctionBegin; > > > > PetscValidHeaderSpecific(A,MAT_COOKIE,1); > > > > PetscValidType(A,1); > > > > PetscValidHeaderSpecific(B,MAT_COOKIE,2); > > > > PetscValidHeaderSpecific(X,MAT_COOKIE,3); > > > > PetscCheckSameComm(A,1,B,2); > > > > PetscCheckSameComm(A,1,X,3); > > > > if (X == B) SETERRQ(PETSC_ERR_ARG_IDN,"X and B must be different > > > > matrices"); > > > > if (!A->factor) SETERRQ(PETSC_ERR_ARG_WRONGSTATE,"Unfactored > > > > matrix"); > > > > if (A->cmap.N != X->rmap.N) SETERRQ2(PETSC_ERR_ARG_SIZ,"Mat A,Mat > > > > X: global dim %D %D",A->cmap.N,X->rmap.N); > > > > if (A->rmap.N != B->rmap.N) SETERRQ2(PETSC_ERR_ARG_SIZ,"Mat A,Mat > > > > B: global dim %D %D",A->rmap.N,B->rmap.N); > > > > if (A->rmap.n != B->rmap.n) SETERRQ2(PETSC_ERR_ARG_SIZ,"Mat A,Mat > > > > B: local dim %D %D",A->rmap.n,B->rmap.n); > > > > if (!A->rmap.N && !A->cmap.N) PetscFunctionReturn(0); > > > > ierr = MatPreallocated(A);CHKERRQ(ierr); > > > > > > > > ierr = PetscLogEventBegin(MAT_MatSolve,A,B,X,0);CHKERRQ(ierr); > > > > if (!A->ops->matsolve) { > > > > ierr = PetscInfo1(A,"Mat type %s using basic MatMatSolve", > > > > ((PetscObject)A)->type_name);CHKERRQ(ierr); > > > > ierr = MatMatSolve_Basic(A,B,X);CHKERRQ(ierr); > > > > } else { > > > > ierr = (*A->ops->matsolve)(A,B,X);CHKERRQ(ierr); > > > > } > > > > ierr = PetscLogEventEnd(MAT_MatSolve,A,B,X,0);CHKERRQ(ierr); > > > > ierr = PetscObjectStateIncrease((PetscObject)X);CHKERRQ(ierr); > > > > PetscFunctionReturn(0); > > > > > > > > } > > > > > > > > On Feb 29, 2008, at 6:46 PM, Yujie wrote: > > > > > > > > > Hi, everyone > > > > > > > > > > I am considering to add a parallel MatMatSolve() into PETSc based > on > > > > > SuperLu_DIST or Spooles. If I want to use it like current > sequential > > > > > MatMatSolve() in the application codes, how to do it? Could you > give > > > > > me some examples about how to add a new function? > > > > > thanks a lot. > > > > > > > > > > Regards, > > > > > Yujie > > > > > > > > > > > > > > > > > > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafaelsantoscoelho at gmail.com Sun Mar 2 18:31:12 2008 From: rafaelsantoscoelho at gmail.com (Rafael Santos Coelho) Date: Sun, 2 Mar 2008 21:31:12 -0300 Subject: Doubts concerning matrix reorderings! Message-ID: <3b6f83d40803021631l70e5d94au4df41c245c4d4482@mail.gmail.com> Hello, folks! (: My name is Rafael and in order to get started with PETSc, I began writing a program which solves, in parallel, a simple fluid flow linear problem. Now, I want to experiment more by trying some matrix reordering schemes and observing the impact they might cause. So after reading PETSc's documentation, I couldn't find good examples to help me out. Say, for instance, I want to use RCM. How would it be? Will a simple call to the PCFactorSetMatOrdering() function do it? Or is there more? Just to remember you, it's a parallel code... Thanks in advance (: Rafael Santos Coelho -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Mar 2 20:04:15 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 2 Mar 2008 20:04:15 -0600 Subject: Doubts concerning matrix reorderings! In-Reply-To: <3b6f83d40803021631l70e5d94au4df41c245c4d4482@mail.gmail.com> References: <3b6f83d40803021631l70e5d94au4df41c245c4d4482@mail.gmail.com> Message-ID: <00AA4661-4DB5-43A4-A588-044A5F9A8B08@mcs.anl.gov> The built in PETSc factorization routines are only sequential, so to do parallel LU (of Cholesky) factorization and solve you need to build PETSc with one of the parallel solver packages: to do this you must config/configure.py PETSc with --download-spooles or --download-superlu_dist or --download-mumps --download-scalapack --download-blacs Now the people who write parallel factorization routines (for some reason I cannot understand) limit the reordering that may be used to a few they have hardwired for their package. Thus the PCFactorSetMatOrdering() doesn't do anything with the parallel factorization packages. If you look at the manual page for MATMPIAIJSPOOLES it shows the possibilities, MATAIJMUMPS and MATSUPERLU_DIST Hope this clarifies things a bit, Barry On Mar 2, 2008, at 6:31 PM, Rafael Santos Coelho wrote: > Hello, folks! (: > > My name is Rafael and in order to get started with PETSc, I began > writing a program which solves, in parallel, a simple fluid flow > linear problem. Now, I want to experiment more by trying some matrix > reordering schemes and observing the impact they might cause. So > after reading PETSc's documentation, I couldn't find good examples > to help me out. Say, for instance, I want to use RCM. How would it > be? Will a simple call to the PCFactorSetMatOrdering() function do > it? Or is there more? Just to remember you, it's a parallel code... > > Thanks in advance (: > > Rafael Santos Coelho From recrusader at gmail.com Sun Mar 2 23:47:23 2008 From: recrusader at gmail.com (Yujie) Date: Mon, 3 Mar 2008 13:47:23 +0800 Subject: about MatAssemblyBegin and MatAssemblyEnd Message-ID: <7ff0ee010803022147p6d779d8ej7ea7ef6f8078b891@mail.gmail.com> Hi, everyone I just check the codes added by Barry for MPIDENSE_MPIAIJ. The codes is as follows. I am wondering what context one may call MatAssemblyBegin and MatAssemblyEnd? >From the webpage, I can find some information about two functions: "Begins assembling the matrix. This routine should be called after completing all calls to MatSetValues()." I mean that I don't know how to judge whether I should call two functions. Could you give me some advice? thanks a lot. 4723 */ 4724 PetscErrorCode MatMatMultNumeric_MPIDense_MPIAIJ(Mat A,Mat B,Mat C) 4725 { 4726 PetscErrorCode ierr; 4727 Mat_MPIDense *sub_c = (Mat_MPIDense*)C->data; 4728 Mat At,Bt,Ct; 4729 4730 PetscFunctionBegin; 4731 ierr = MatTranspose(A,MAT_INITIAL_MATRIX,&At);CHKERRQ(ierr); 4732 ierr = MatTranspose(B,MAT_INITIAL_MATRIX,&Bt);CHKERRQ(ierr); 4733 ierr = MatMatMult(Bt,At,MAT_INITIAL_MATRIX,1.0,&Ct);CHKERRQ(ierr); 4734 ierr = MatDestroy(At);CHKERRQ(ierr); 4735 ierr = MatDestroy(Bt);CHKERRQ(ierr); 4736 ierr = MatTranspose(Ct,MAT_REUSE_MATRIX,&C);CHKERRQ(ierr); 4737 ierr = MatDestroy(Ct);CHKERRQ(ierr); 4738 4739 C->assembled = PETSC_TRUE; 4740 ierr = MatAssemblyBegin(sub_c->A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); 4741 ierr = MatAssemblyEnd(sub_c->A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); 4742 if (!C->was_assembled) { 4743 ierr = MatSetUpMultiply_MPIDense(C);CHKERRQ(ierr); 4744 } 4745 PetscFunctionReturn(0); 4746 } Regards, Yujie -------------- next part -------------- An HTML attachment was scrubbed... URL: From Amit.Itagi at seagate.com Mon Mar 3 08:30:47 2008 From: Amit.Itagi at seagate.com (Amit.Itagi at seagate.com) Date: Mon, 3 Mar 2008 09:30:47 -0500 Subject: Direct LU solver In-Reply-To: Message-ID: Thanks for your response. A couple of follow-up questions - I go over the steps MatCreate() MatSetType(MPIAIJ) MatMPIAIJSetPreallocation() MatSetType(MATSUPERLU_DIST). Even though I want to do the direct LU solves repeatedly (with the same matrix), I don't want the program to do LU factorization repeatedly. I hope I can get that functionality by using the "SAME_PRECONDITIONER" option (along with -ksppreonly), while defining the KSP. When the program does the factorization, does it do it in-place or does it allocate new storage ? After doing the LU factorization, are the other operations such as MatMult() preserved ? Thanks Rgds, Amit Satish Balay To Sent by: petsc-users at mcs.anl.gov owner-petsc-users cc @mcs.anl.gov No Phone Info Subject Available Re: Direct LU solver 02/29/2008 02:06 PM Please respond to petsc-users at mcs.a nl.gov On Fri, 29 Feb 2008, Amit.Itagi at seagate.com wrote: > > My woes continue. Based on the earlier discussions, I implemented the > matrix as > > //========================================================================= > > // Option 1 > ierr=MatCreate(PETSC_COMM_WORLD,&A); CHKERRQ(ierr); > ierr=MatSetSizes(A,1,1,2,2); CHKERRQ(ierr); > > > /* Option 2 > PetscInt d_nnz=1, o_nnz=1; > ierr=MatCreateMPIAIJ(PETSC_COMM_WORLD,1,1,2,2,0,&d_nnz,0,&o_nnz,&A); > CHKERRQ(ierr); > */ > > /* Option 3 > > ierr=MatCreateMPIAIJ(PETSC_COMM_WORLD,1,1,2,2,0,PETSC_NULL,0,PETSC_NULL,&A); > CHKERRQ(ierr); > */ > > ierr=MatSetType(A,MATSUPERLU_DIST); CHKERRQ(ierr); > ierr=MatSetFromOptions(A); CHKERRQ(ierr); > > // (After this, I set the values and do the assembly). I then use the > direct LU solver. > > //============================================================================ > > Note: I have a simple 2 by 2 matrix (with non-zero values in all 4 places). > If I use "option 1" (based on Satish's email), the program executes > successfully. If instead of "option 1", I use "option 2" or "option 3", I > get a crash. > If I am not mistaken, options 1 and 3 are the same. Option 2, additionally, > does a pre-allocation. Am I correct ? Nope - Option 3 is same as: MatCreate() MatSetType(MPIAIJ) MatMPIAIJSetPreallocation() MatSetType(MATSUPERLU_DIST) [i.e first you are setting type as MPIAIJ, and then changing to MATSUPERLU_DIST] What you want is: MatCreate() MatSetType(MATSUPERLU_DIST) MatMPIAIJSetPreallocation() [Ideally you need MatSuerLU_DistSetPreallocation() - but that would be same as MatMPIAIJSetPreallocation()] Satish From dalcinl at gmail.com Mon Mar 3 08:48:57 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 3 Mar 2008 11:48:57 -0300 Subject: Doubts concerning matrix reorderings! In-Reply-To: <3b6f83d40803021631l70e5d94au4df41c245c4d4482@mail.gmail.com> References: <3b6f83d40803021631l70e5d94au4df41c245c4d4482@mail.gmail.com> Message-ID: Dear Rafael, two questions: * What kind of flow problem are you trying to solve? compressible or incompressible? Or just an advection problem? What discretization method are you using? * Have you considered using iterative methods for this, or you are actually investigating direct methods for your problem at hand? On 3/2/08, Rafael Santos Coelho wrote: > Hello, folks! (: > > My name is Rafael and in order to get started with PETSc, I began writing a > program which solves, in parallel, a simple fluid flow linear problem. Now, > I want to experiment more by trying some matrix reordering schemes and > observing the impact they might cause. So after reading PETSc's > documentation, I couldn't find good examples to help me out. Say, for > instance, I want to use RCM. How would it be? Will a simple call to the > PCFactorSetMatOrdering() function do it? Or is there more? Just to remember > you, it's a parallel code... > > Thanks in advance (: > > Rafael Santos Coelho > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From hzhang at mcs.anl.gov Mon Mar 3 09:55:21 2008 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 3 Mar 2008 09:55:21 -0600 (CST) Subject: Direct LU solver In-Reply-To: References: Message-ID: Amit, > Thanks for your response. A couple of follow-up questions - > > I go over the steps > > MatCreate() > MatSetType(MPIAIJ) > MatMPIAIJSetPreallocation() > MatSetType(MATSUPERLU_DIST). > > Even though I want to do the direct LU solves repeatedly (with the same > matrix), I don't want the program to do LU factorization repeatedly. I hope > I can get that functionality by using the "SAME_PRECONDITIONER" option > (along with -ksppreonly), while defining the KSP. When the program does the > factorization, does it do it in-place or does it allocate new storage ? allocate new storage unless you call MatLUFactor() explicitly. It is done by KSPSetUp() which calls PCSetUp() -> PCSetUp_LU() -> MatLUFactorSymbolic() & MatLUFactorNumeric( You can do following for reusing factored L and U: KSPCreate() KSPSetOperators() KSPSetFromOptions() KSPSetUp() while ( num_rhs-- ) { KSPSolve(ksp,b,x); } See src/ksp/ksp/examples/tutorials/ex10.c > After doing the LU factorization, are the other operations such as > MatMult() preserved ? Yes. Hong > > Thanks > > Rgds, > Amit > > > > > > Satish Balay > v> To > Sent by: petsc-users at mcs.anl.gov > owner-petsc-users cc > @mcs.anl.gov > No Phone Info Subject > Available Re: Direct LU solver > > > 02/29/2008 02:06 > PM > > > Please respond to > petsc-users at mcs.a > nl.gov > > > > > > > On Fri, 29 Feb 2008, Amit.Itagi at seagate.com wrote: > >> >> My woes continue. Based on the earlier discussions, I implemented the >> matrix as >> >> > //========================================================================= >> >> // Option 1 >> ierr=MatCreate(PETSC_COMM_WORLD,&A); CHKERRQ(ierr); >> ierr=MatSetSizes(A,1,1,2,2); CHKERRQ(ierr); >> >> >> /* Option 2 >> PetscInt d_nnz=1, o_nnz=1; >> ierr=MatCreateMPIAIJ(PETSC_COMM_WORLD,1,1,2,2,0,&d_nnz,0,&o_nnz,&A); >> CHKERRQ(ierr); >> */ >> >> /* Option 3 >> >> > ierr=MatCreateMPIAIJ(PETSC_COMM_WORLD,1,1,2,2,0,PETSC_NULL,0,PETSC_NULL,&A); > >> CHKERRQ(ierr); >> */ >> >> ierr=MatSetType(A,MATSUPERLU_DIST); CHKERRQ(ierr); >> ierr=MatSetFromOptions(A); CHKERRQ(ierr); >> >> // (After this, I set the values and do the assembly). I then use the >> direct LU solver. >> >> > //============================================================================ > >> >> Note: I have a simple 2 by 2 matrix (with non-zero values in all 4 > places). >> If I use "option 1" (based on Satish's email), the program executes >> successfully. If instead of "option 1", I use "option 2" or "option 3", I >> get a crash. >> If I am not mistaken, options 1 and 3 are the same. Option 2, > additionally, >> does a pre-allocation. Am I correct ? > > Nope - Option 3 is same as: > > MatCreate() > MatSetType(MPIAIJ) > MatMPIAIJSetPreallocation() > MatSetType(MATSUPERLU_DIST) > > [i.e first you are setting type as MPIAIJ, and then changing to > MATSUPERLU_DIST] > > What you want is: > > MatCreate() > MatSetType(MATSUPERLU_DIST) > MatMPIAIJSetPreallocation() > > [Ideally you need MatSuerLU_DistSetPreallocation() - but that would be > same as MatMPIAIJSetPreallocation()] > > Satish > > > > From rafaelsantoscoelho at gmail.com Mon Mar 3 10:50:12 2008 From: rafaelsantoscoelho at gmail.com (Rafael Santos Coelho) Date: Mon, 3 Mar 2008 13:50:12 -0300 Subject: Doubts concerning matrix reorderings! In-Reply-To: References: <3b6f83d40803021631l70e5d94au4df41c245c4d4482@mail.gmail.com> Message-ID: <3b6f83d40803030850s7bdfe97bqd7288b778d21ff0e@mail.gmail.com> Hello, everyone! (: Thank you very much, Barry, for your reply. I'll attempt to reconfigure PETSc with such external packages, as you told me, and see how it goes... As for your questions, Lisandro: 1) Incompressible bidimensional flow problem, via finite differences (Darcy's law) resulting in a pentadiagonal coefficient matrix; 2) Currently, I've been working with GMRES and LCD and now I intend to analyse in what way matrix reorderings routines affect the solving process (fill-in reduction, stability, number of iterations, etc); I realize it's an easy problem and so matrix reorderings might not be too advantageous. But still, the main reason I want to pursue this it's because soon I'll be dealing with more complex nonlinear problems and when that time comes, I already plan to be familiarized with reorderings [: Regards, Rafael Santos Coelho -------------- next part -------------- An HTML attachment was scrubbed... URL: From recrusader at gmail.com Mon Mar 3 10:59:09 2008 From: recrusader at gmail.com (Yujie) Date: Mon, 3 Mar 2008 08:59:09 -0800 Subject: about MatAssemblyBegin and MatAssemblyEnd In-Reply-To: References: <7ff0ee010803022147p6d779d8ej7ea7ef6f8078b891@mail.gmail.com> Message-ID: <7ff0ee010803030859q35d35134pffc00b231cad37e0@mail.gmail.com> Dear Barry: Why do you call these two functions in your code? Because I have realized the same method with yours to calculate MPIDENSE_MPIAIJ. However, I didn't call MatAssemblyBegin and MatAssemblyEnd after I call MatTranspose for matrix C. In application, sometimes I don't know what do the functions I call in PETSc do? So, I don't know whether I should call these two functions although you give me these conditions. Regarding this example, which condition should it belong to? thanks a lot. Regards, Yujie On 3/3/08, Barry Smith wrote: > > > MatAssembly does > 1) moves any values set on one process that belongs on another > 2) finalizes the sparse matrix data structure (for example takes any > holes out of the CSR format) > doesn't need to do this for dense matrices > 3) sets up whatever VecScatters that are need to do matrix vector > products > 4) marks the matrix as assembled and ready to be used > > > Barry > > > On Mar 2, 2008, at 11:47 PM, Yujie wrote: > > > Hi, everyone > > > > I just check the codes added by Barry for MPIDENSE_MPIAIJ. The codes > > is as follows. I am wondering what context one may call > > MatAssemblyBegin and MatAssemblyEnd? > > From the webpage, I can find some information about two functions: > > "Begins assembling the matrix. This routine should be called after > > completing all calls to MatSetValues()." > > I mean that I don't know how to judge whether I should call two > > functions. Could you give me some advice? thanks a lot. > > > > 4723 */ > > 4724 PetscErrorCode MatMatMultNumeric_MPIDense_MPIAIJ(Mat A,Mat > > B,Mat C) > > 4725 { > > 4726 PetscErrorCode ierr; > > 4727 Mat_MPIDense *sub_c = (Mat_MPIDense*)C->data; > > 4728 Mat At,Bt,Ct; > > 4729 > > 4730 PetscFunctionBegin; > > 4731 ierr = MatTranspose(A,MAT_INITIAL_MATRIX,&At);CHKERRQ(ierr); > > 4732 ierr = MatTranspose(B,MAT_INITIAL_MATRIX,&Bt);CHKERRQ(ierr); > > 4733 ierr = MatMatMult(Bt,At,MAT_INITIAL_MATRIX, > > 1.0,&Ct);CHKERRQ(ierr); > > 4734 ierr = MatDestroy(At);CHKERRQ(ierr); > > 4735 ierr = MatDestroy(Bt);CHKERRQ(ierr); > > 4736 ierr = MatTranspose(Ct,MAT_REUSE_MATRIX,&C);CHKERRQ(ierr); > > 4737 ierr = MatDestroy(Ct);CHKERRQ(ierr); > > 4738 > > 4739 C->assembled = PETSC_TRUE; > > 4740 ierr = MatAssemblyBegin(sub_c- > > >A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > 4741 ierr = MatAssemblyEnd(sub_c->A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > 4742 if (!C->was_assembled) { > > 4743 ierr = MatSetUpMultiply_MPIDense(C);CHKERRQ(ierr); > > 4744 } > > 4745 PetscFunctionReturn(0); > > 4746 } > > > > > > > > Regards, > > Yujie > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From li76pan at yahoo.com Mon Mar 3 11:05:10 2008 From: li76pan at yahoo.com (li pan) Date: Mon, 3 Mar 2008 09:05:10 -0800 (PST) Subject: matrix free for linear solver Message-ID: <234726.14289.qm@web36807.mail.mud.yahoo.com> Dear all, If I want to solve a linear equation in matrix free scheme, how is the command line argument? I only know for nonlinear solver it is _snes_mf thanx pan ____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping From knepley at gmail.com Mon Mar 3 11:07:32 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 3 Mar 2008 11:07:32 -0600 Subject: matrix free for linear solver In-Reply-To: <234726.14289.qm@web36807.mail.mud.yahoo.com> References: <234726.14289.qm@web36807.mail.mud.yahoo.com> Message-ID: On Mon, Mar 3, 2008 at 11:05 AM, li pan wrote: > Dear all, > If I want to solve a linear equation in matrix free > scheme, how is the command line argument? I only know > for nonlinear solver it is _snes_mf What do you mean here? How do you have linear equations without a matrix? If you mean that you would only like to specify the action of the operator, you should use MATSHELL. Matt > thanx > > pan > > > > ____________________________________________________________________________________ > Looking for last minute shopping deals? > Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From recrusader at gmail.com Mon Mar 3 11:24:19 2008 From: recrusader at gmail.com (Yujie) Date: Mon, 3 Mar 2008 09:24:19 -0800 Subject: about MatAssemblyBegin and MatAssemblyEnd In-Reply-To: <25EE2354-62CD-451D-88C8-EF813E9B1121@mcs.anl.gov> References: <7ff0ee010803022147p6d779d8ej7ea7ef6f8078b891@mail.gmail.com> <7ff0ee010803030859q35d35134pffc00b231cad37e0@mail.gmail.com> <25EE2354-62CD-451D-88C8-EF813E9B1121@mcs.anl.gov> Message-ID: <7ff0ee010803030924k180d1c40i50d768afad64f4c2@mail.gmail.com> Dear Barry: Thank you for your reply. Generally, for security, after I do some matrix operations by calling some functions in PETSc, I should call the assembly functions based on your advice. Whether is there a mechanism/method that the users should always call assembly functions but the code will judge whether it should do the assembly? If there is, I think that it should save some time :). Regards, Yujie On 3/3/08, Barry Smith wrote: > > > Based on your comments I have simplified and improved the code slightly > it is nowPetscErrorCode MatMatMultNumeric_MPIDense_MPIAIJ(Mat A,Mat B,Mat > C) > { > PetscErrorCode ierr; > Mat At,Bt,Ct; > > PetscFunctionBegin; > ierr = MatTranspose(A,MAT_INITIAL_MATRIX,&At);CHKERRQ(ierr); > ierr = MatTranspose(B,MAT_INITIAL_MATRIX,&Bt);CHKERRQ(ierr); > ierr = MatMatMult(Bt,At,MAT_INITIAL_MATRIX,1.0,&Ct);CHKERRQ(ierr); > ierr = MatDestroy(At);CHKERRQ(ierr); > ierr = MatDestroy(Bt);CHKERRQ(ierr); > ierr = MatTranspose(Ct,MAT_REUSE_MATRIX,&C);CHKERRQ(ierr); > ierr = MatDestroy(Ct);CHKERRQ(ierr); > PetscFunctionReturn(0); > } > > #undef __FUNCT__ > #define __FUNCT__ "MatMatMultSymbolic_MPIDense_MPIAIJ" > PetscErrorCode MatMatMultSymbolic_MPIDense_MPIAIJ(Mat A,Mat B,PetscReal > fill,Mat *C) > { > PetscErrorCode ierr; > PetscInt m=A->rmap.n,n=B->cmap.n; > Mat Cmat; > > PetscFunctionBegin; > if (A->cmap.n != B->rmap.n) SETERRQ2(PETSC_ERR_ARG_SIZ,"A->cmap.n %d != > B->rmap.n %d\n",A->cmap.n,B->rmap.n); > ierr = MatCreate(A->hdr.comm,&Cmat);CHKERRQ(ierr); > ierr = > MatSetSizes(Cmat,m,n,PETSC_DETERMINE,PETSC_DETERMINE);CHKERRQ(ierr); > ierr = MatSetType(Cmat,MATMPIDENSE);CHKERRQ(ierr); > ierr = MatMPIDenseSetPreallocation(Cmat,PETSC_NULL);CHKERRQ(ierr); > ierr = MatAssemblyBegin(Cmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(Cmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > *C = Cmat; > PetscFunctionReturn(0); > } > > As you note there was no need to call the assembly after the transpose. It > is, needed, in general where I put it > in the symbolic portion of the code. (You could avoid it by directly > setting some flags in the Cmat object > but it is better to do the right thing and call the assembly (which is > free in terms of time anyways compare to the > numerical calculations). > > Barry > > On Mar 3, 2008, at 10:59 AM, Yujie wrote: > > Dear Barry: > > Why do you call these two functions in your code? Because I have realized > the same method with yours to calculate MPIDENSE_MPIAIJ. However, I didn't > call MatAssemblyBegin and > MatAssemblyEnd after I call MatTranspose for matrix C. > In application, sometimes I don't know what do the functions I call in > PETSc do? So, I don't know whether I should > call these two functions although you give me these conditions. Regarding this example, which condition should it belong to? thanks a lot. > > Regards, > Yujie > > On 3/3/08, Barry Smith wrote: > > > > > > MatAssembly does > > 1) moves any values set on one process that belongs on another > > 2) finalizes the sparse matrix data structure (for example takes any > > holes out of the CSR format) > > doesn't need to do this for dense matrices > > 3) sets up whatever VecScatters that are need to do matrix vector > > products > > 4) marks the matrix as assembled and ready to be used > > > > > > Barry > > > > > > On Mar 2, 2008, at 11:47 PM, Yujie wrote: > > > > > Hi, everyone > > > > > > I just check the codes added by Barry for MPIDENSE_MPIAIJ. The codes > > > is as follows. I am wondering what context one may call > > > MatAssemblyBegin and MatAssemblyEnd? > > > From the webpage, I can find some information about two functions: > > > "Begins assembling the matrix. This routine should be called after > > > completing all calls to MatSetValues()." > > > I mean that I don't know how to judge whether I should call two > > > functions. Could you give me some advice? thanks a lot. > > > > > > 4723 */ > > > 4724 PetscErrorCode MatMatMultNumeric_MPIDense_MPIAIJ(Mat A,Mat > > > B,Mat C) > > > 4725 { > > > 4726 PetscErrorCode ierr; > > > 4727 Mat_MPIDense *sub_c = (Mat_MPIDense*)C->data; > > > 4728 Mat At,Bt,Ct; > > > 4729 > > > 4730 PetscFunctionBegin; > > > 4731 ierr = MatTranspose(A,MAT_INITIAL_MATRIX,&At);CHKERRQ(ierr); > > > 4732 ierr = MatTranspose(B,MAT_INITIAL_MATRIX,&Bt);CHKERRQ(ierr); > > > 4733 ierr = MatMatMult(Bt,At,MAT_INITIAL_MATRIX, > > > 1.0,&Ct);CHKERRQ(ierr); > > > 4734 ierr = MatDestroy(At);CHKERRQ(ierr); > > > 4735 ierr = MatDestroy(Bt);CHKERRQ(ierr); > > > 4736 ierr = MatTranspose(Ct,MAT_REUSE_MATRIX,&C);CHKERRQ(ierr); > > > 4737 ierr = MatDestroy(Ct);CHKERRQ(ierr); > > > 4738 > > > 4739 C->assembled = PETSC_TRUE; > > > 4740 ierr = MatAssemblyBegin(sub_c- > > > >A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > > 4741 ierr = MatAssemblyEnd(sub_c->A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > > 4742 if (!C->was_assembled) { > > > 4743 ierr = MatSetUpMultiply_MPIDense(C);CHKERRQ(ierr); > > > 4744 } > > > 4745 PetscFunctionReturn(0); > > > 4746 } > > > > > > > > > > > > Regards, > > > Yujie > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Mon Mar 3 12:04:15 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 3 Mar 2008 15:04:15 -0300 Subject: matrix free for linear solver In-Reply-To: <234726.14289.qm@web36807.mail.mud.yahoo.com> References: <234726.14289.qm@web36807.mail.mud.yahoo.com> Message-ID: Well, if this helps you, you have attached a self-dontained example for sequentially solving a 3D Poisson problem by using MATSHELL and KSP. It IS NOT based on PETSc DA's. The actual routine implementing a finite difference scheme is in Fortran 90. It assumes the domain is the unit box, bc are all Dirichlet and zero, and that the each direction of the box is discretized with the same number of nodes. Additionally, some part of the code is lacking error checking, that is the ierr=... and CHKERRQ() macros. Hope this can be helpful to you... Regards, On 3/3/08, li pan wrote: > Dear all, > If I want to solve a linear equation in matrix free > scheme, how is the command line argument? I only know > for nonlinear solver it is _snes_mf > > thanx > > > pan > > > > ____________________________________________________________________________________ > Looking for last minute shopping deals? > Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: del2lib.f90 Type: application/octet-stream Size: 577 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: del2mat.h Type: application/octet-stream Size: 1229 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: solvesys.c Type: application/octet-stream Size: 1732 bytes Desc: not available URL: From li76pan at yahoo.com Mon Mar 3 12:16:34 2008 From: li76pan at yahoo.com (li pan) Date: Mon, 3 Mar 2008 10:16:34 -0800 (PST) Subject: matrix free for linear solver In-Reply-To: Message-ID: <826323.76252.qm@web36811.mail.mud.yahoo.com> hi Matt, I used to discuss with somebody about SNES for nonlinear equation. One can use -snes_mf_operator for matrix free calculation. As far as I know, SNES uses a Newton-Kryplov Method. In each Newton, one has to solve a linear equation, which is : residual = Jacobian * delt_u If I can solve the linear equation in a iterative way in Kryplov space, then I only need to optimise Jacobian*delt_u - residual The matrix vector multiplication can be expressed through the gradient calculation for residual. That's why we can forget the Jacobian matrix. The method is generally called Jacobian free Newton Kryplov mehtod (JFNK). You must know it. Actually, that's the way snes works with -snes_mf. Here, we define the function SNESSetFunction(). And as argument, we give the calculation of residual. I wrote so much only because of the continuity of the context. Now my question is, if the SNES method includes a JFNK method to solve the linear equation, why can't I use it for my linear equation directly? And how? thanx pan --- Matthew Knepley wrote: > On Mon, Mar 3, 2008 at 11:05 AM, li pan > wrote: > > Dear all, > > If I want to solve a linear equation in matrix > free > > scheme, how is the command line argument? I only > know > > for nonlinear solver it is _snes_mf > > What do you mean here? How do you have linear > equations without > a matrix? If you mean that you would only like to > specify the action > of the operator, you should use MATSHELL. > > Matt > > > thanx > > > > pan > > > > > > > > > ____________________________________________________________________________________ > > Looking for last minute shopping deals? > > Find them fast with Yahoo! Search. > http://tools.search.yahoo.com/newsearch/category.php?category=shopping > > > > > > > > -- > What most experimenters take for granted before they > begin their > experiments is infinitely more interesting than any > results to which > their experiments lead. > -- Norbert Wiener > > ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs From li76pan at yahoo.com Mon Mar 3 12:16:34 2008 From: li76pan at yahoo.com (li pan) Date: Mon, 3 Mar 2008 10:16:34 -0800 (PST) Subject: matrix free for linear solver In-Reply-To: Message-ID: <826323.76252.qm@web36811.mail.mud.yahoo.com> hi Matt, I used to discuss with somebody about SNES for nonlinear equation. One can use -snes_mf_operator for matrix free calculation. As far as I know, SNES uses a Newton-Kryplov Method. In each Newton, one has to solve a linear equation, which is : residual = Jacobian * delt_u If I can solve the linear equation in a iterative way in Kryplov space, then I only need to optimise Jacobian*delt_u - residual The matrix vector multiplication can be expressed through the gradient calculation for residual. That's why we can forget the Jacobian matrix. The method is generally called Jacobian free Newton Kryplov mehtod (JFNK). You must know it. Actually, that's the way snes works with -snes_mf. Here, we define the function SNESSetFunction(). And as argument, we give the calculation of residual. I wrote so much only because of the continuity of the context. Now my question is, if the SNES method includes a JFNK method to solve the linear equation, why can't I use it for my linear equation directly? And how? thanx pan --- Matthew Knepley wrote: > On Mon, Mar 3, 2008 at 11:05 AM, li pan > wrote: > > Dear all, > > If I want to solve a linear equation in matrix > free > > scheme, how is the command line argument? I only > know > > for nonlinear solver it is _snes_mf > > What do you mean here? How do you have linear > equations without > a matrix? If you mean that you would only like to > specify the action > of the operator, you should use MATSHELL. > > Matt > > > thanx > > > > pan > > > > > > > > > ____________________________________________________________________________________ > > Looking for last minute shopping deals? > > Find them fast with Yahoo! Search. > http://tools.search.yahoo.com/newsearch/category.php?category=shopping > > > > > > > > -- > What most experimenters take for granted before they > begin their > experiments is infinitely more interesting than any > results to which > their experiments lead. > -- Norbert Wiener > > ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs From knepley at gmail.com Mon Mar 3 12:22:25 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 3 Mar 2008 12:22:25 -0600 Subject: matrix free for linear solver In-Reply-To: <826323.76252.qm@web36811.mail.mud.yahoo.com> References: <826323.76252.qm@web36811.mail.mud.yahoo.com> Message-ID: On Mon, Mar 3, 2008 at 12:16 PM, li pan wrote: > hi Matt, > I used to discuss with somebody about SNES for > nonlinear equation. One can use -snes_mf_operator for > matrix free calculation. As far as I know, SNES uses a > Newton-Kryplov Method. In each Newton, one has to > solve a linear equation, which is : > residual = Jacobian * delt_u > If I can solve the linear equation in a iterative way > in Kryplov space, then I only need to optimise > Jacobian*delt_u - residual > The matrix vector multiplication can be expressed > through the gradient calculation for residual. That's > why we can forget the Jacobian matrix. The method is > generally called Jacobian free Newton Kryplov mehtod > (JFNK). You must know it. > Actually, that's the way snes works with -snes_mf. > Here, we define the function SNESSetFunction(). And as > argument, we give the calculation of residual. > I wrote so much only because of the continuity of the > context. > Now my question is, if the SNES method includes a JFNK > method to solve the linear equation, why can't I use > it for my linear equation directly? And how? The "matrix-free" you refer to above uses a finite difference approximation to the Jacobian of a nonlinear function. If you have a linear equation, the Jacobian is THE EQUATION ITSELF. Thus, what you propose makes no sense to me at all. Again, I recommend you look at MATSHELL since I think this is what you want. Matt > thanx > > pan > > > --- Matthew Knepley wrote: > > > On Mon, Mar 3, 2008 at 11:05 AM, li pan > > wrote: > > > Dear all, > > > If I want to solve a linear equation in matrix > > free > > > scheme, how is the command line argument? I only > > know > > > for nonlinear solver it is _snes_mf > > > > What do you mean here? How do you have linear > > equations without > > a matrix? If you mean that you would only like to > > specify the action > > of the operator, you should use MATSHELL. > > > > Matt > > > > > thanx > > > > > > pan > > > > > > > > > > > > > > > ____________________________________________________________________________________ > > > Looking for last minute shopping deals? > > > Find them fast with Yahoo! Search. > > > http://tools.search.yahoo.com/newsearch/category.php?category=shopping > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they > > begin their > > experiments is infinitely more interesting than any > > results to which > > their experiments lead. > > -- Norbert Wiener > > > > > > > > > ____________________________________________________________________________________ > Never miss a thing. Make Yahoo your home page. > http://www.yahoo.com/r/hs > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From dalcinl at gmail.com Mon Mar 3 12:21:25 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 3 Mar 2008 15:21:25 -0300 Subject: Doubts concerning matrix reorderings! In-Reply-To: <3b6f83d40803030850s7bdfe97bqd7288b778d21ff0e@mail.gmail.com> References: <3b6f83d40803021631l70e5d94au4df41c245c4d4482@mail.gmail.com> <3b6f83d40803030850s7bdfe97bqd7288b778d21ff0e@mail.gmail.com> Message-ID: On 3/3/08, Rafael Santos Coelho wrote: > I realize it's an easy problem and so matrix reorderings might not be too > advantageous. But still, the main reason I want to pursue this it's because > soon I'll be dealing with more complex nonlinear problems and when that time > comes, I already plan to be familiarized with reorderings [: OK, I see. Anyway, if you have the chance, I would recommend you to take a look at SNES if going to nonlinear problems, and specially its support for Eisenstand&Walker method for selecting the tolerance of the inner Krylov solver. You have good chances of obtaining better running times, even for moderately sized problems. Just a suggestion for further investigation :) ! -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon Mar 3 12:26:30 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 3 Mar 2008 15:26:30 -0300 Subject: matrix free for linear solver In-Reply-To: <826323.76252.qm@web36811.mail.mud.yahoo.com> References: <826323.76252.qm@web36811.mail.mud.yahoo.com> Message-ID: On 3/3/08, li pan wrote: > Now my question is, if the SNES method includes a JFNK > method to solve the linear equation, why can't I use > it for my linear equation directly? And how? Sorry, I'm a bit confused. Are you trying to solve a linear or a nonlinear problem? What are you exactly trying to do? > > thanx > > pan > > > > --- Matthew Knepley wrote: > > > On Mon, Mar 3, 2008 at 11:05 AM, li pan > > wrote: > > > Dear all, > > > If I want to solve a linear equation in matrix > > free > > > scheme, how is the command line argument? I only > > know > > > for nonlinear solver it is _snes_mf > > > > What do you mean here? How do you have linear > > equations without > > a matrix? If you mean that you would only like to > > specify the action > > of the operator, you should use MATSHELL. > > > > Matt > > > > > thanx > > > > > > pan > > > > > > > > > > > > > > > ____________________________________________________________________________________ > > > Looking for last minute shopping deals? > > > Find them fast with Yahoo! Search. > > > http://tools.search.yahoo.com/newsearch/category.php?category=shopping > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they > > begin their > > experiments is infinitely more interesting than any > > results to which > > their experiments lead. > > -- Norbert Wiener > > > > > > > > > ____________________________________________________________________________________ > Never miss a thing. Make Yahoo your home page. > http://www.yahoo.com/r/hs > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From Amit.Itagi at seagate.com Mon Mar 3 13:08:36 2008 From: Amit.Itagi at seagate.com (Amit.Itagi at seagate.com) Date: Mon, 3 Mar 2008 14:08:36 -0500 Subject: Direct LU solver In-Reply-To: Message-ID: Hong, Your comments were helpful. Now I can do the LU solves multiple times. However within the program, if I destroy the matrix and the KSP and create them again (for a different matrix), I get an error in superlu. Here is the log from a successful run. [1] 0.249931 Event begin: KSPSetup [1] 0.24994 Event end: KSPSetup [1] 0.249948 Event begin: PCSetUp [1] 0.249963 Event begin: MatGetOrdering [0] 0.151691 Event begin: KSPSetup [0] 0.151699 Event end: KSPSetup [0] PCSetUp(): Setting up new PC [0] 0.151713 Event begin: PCSetUp [0] 0.151722 Event begin: MatGetOrdering [1] PetscCommDuplicate(): returning tag 2147483493 [0] PetscCommDuplicate(): returning tag 2147483493 [1] PetscCommDuplicate(): returning tag 2147483492 [0] PetscCommDuplicate(): returning tag 2147483492 [1] 0.250169 Event end: MatGetOrdering [1] 0.250182 Event begin: MatLUFactorSym [0] 0.151885 Event end: MatGetOrdering [0] 0.151896 Event begin: MatLUFactorSym [1] PetscCommDuplicate(): returning tag 2147483491 [0] PetscCommDuplicate(): returning tag 2147483491 [1] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ LU factorization and solves. [0] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ LU factorization and solves. [1] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 143685848 [1] PetscCommDuplicate(): returning tag 2147483627 [1] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 143685848 [1] PetscCommDuplicate(): returning tag 2147483626 [0] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 144562608 [0] PetscCommDuplicate(): returning tag 2147483627 [0] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 144562608 [0] PetscCommDuplicate(): returning tag 2147483626 [1] 0.252072 Event end: MatLUFactorSym [1] 0.252098 Event begin: MatLUFactorNum [0] 0.153821 Event end: MatLUFactorSym [0] 0.153835 Event begin: MatLUFactorNum Nonzeros in L 171877 Nonzeros in U 171877 nonzeros in L+U-I 342195 nonzeros in LSUB 52137 Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 0.000296116 / 0.00028491 / 0.000290513 [1] 0.623389 Event end: MatLUFactorNum [1] 0.623425 Event end: PCSetUp EQUIL time 0.00 ROWPERM time 0.01 COLPERM time 0.01 SYMBFACT time 0.01 DISTRIBUTE time 0.02 FACTOR time 0.29 Factor flops 2.652935e+08 Mflops 924.82 SOLVE time 0.00 [0] 0.525149 Event end: MatLUFactorNum [0] 0.525167 Event end: PCSetUp Here is the log from a crash. [0] 1.41665 Event begin: KSPSetup [0] 1.41666 Event end: KSPSetup [0] PCSetUp(): Setting up new PC [0] 1.41668 Event begin: PCSetUp [0] 1.41669 Event begin: MatGetOrdering [1] 1.51505 Event begin: KSPSetup [1] 1.51506 Event end: KSPSetup [1] 1.51507 Event begin: PCSetUp [1] 1.51507 Event begin: MatGetOrdering [0] PetscCommDuplicate(): returning tag 2147483226 [1] PetscCommDuplicate(): returning tag 2147483226 [0] PetscCommDuplicate(): returning tag 2147483225 [1] PetscCommDuplicate(): returning tag 2147483225 [0] 1.4169 Event end: MatGetOrdering [0] 1.41692 Event begin: MatLUFactorSym [1] 1.51526 Event end: MatGetOrdering [1] 1.51526 Event begin: MatLUFactorSym [0] PetscCommDuplicate(): returning tag 2147483224 [1] PetscCommDuplicate(): returning tag 2147483224 [0] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ LU factorization and solves. [1] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ LU factorization and solves. [0] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 144562608 [0] PetscCommDuplicate(): returning tag 2147483627 [0] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 144562608 [0] PetscCommDuplicate(): returning tag 2147483626 [1] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 143685848 [1] PetscCommDuplicate(): returning tag 2147483627 [1] PetscCommDuplicate(): Using internal PETSc communicator 1083412768 143685848 [1] PetscCommDuplicate(): returning tag 2147483626 [0] 1.4187 Event end: MatLUFactorSym [0] 1.41872 Event begin: MatLUFactorNum [1] 1.51706 Event end: MatLUFactorSym [1] 1.51707 Event begin: MatLUFactorNum MPI_Alltoallv: invalid datatype argument: Invalid argument (rank 0, comm 5) Rank (0, MPI_COMM_WORLD): Call stack within LAM: Rank (0, MPI_COMM_WORLD): - MPI_Alltoallv() Rank (0, MPI_COMM_WORLD): - main() If I trace the place of the error in the debugger, the trace gives KSPSolve() -> KSPSetUp() -> PCSetUp() -> PCSetUp_LU() -> MatLuFactorNumeric() -> MatLuFactorNumeric_SuperLU_Dist() -> pzgssvx() -> pzCompRow_loc_to_CompCol_global() -> MPI_Alltoallv I could not make much head-way by looking at the errors. Could you give me some tips on what might be causing this error, and what to look for ? Thanks Rgds, Amit owner-petsc-users at mcs.anl.gov wrote on 03/03/2008 10:55:21 AM: > > Amit, > > > Thanks for your response. A couple of follow-up questions - > > > > I go over the steps > > > > MatCreate() > > MatSetType(MPIAIJ) > > MatMPIAIJSetPreallocation() > > MatSetType(MATSUPERLU_DIST). > > > > Even though I want to do the direct LU solves repeatedly (with the same > > matrix), I don't want the program to do LU factorization repeatedly. I hope > > I can get that functionality by using the "SAME_PRECONDITIONER" option > > (along with -ksppreonly), while defining the KSP. When the program does the > > factorization, does it do it in-place or does it allocate new storage ? > > allocate new storage unless you call > MatLUFactor() explicitly. > > It is done by > KSPSetUp() > which calls PCSetUp() -> PCSetUp_LU() > -> MatLUFactorSymbolic() & MatLUFactorNumeric( > > You can do following for reusing factored L and U: > KSPCreate() > KSPSetOperators() > KSPSetFromOptions() > KSPSetUp() > while ( num_rhs-- ) { > KSPSolve(ksp,b,x); > } > > See src/ksp/ksp/examples/tutorials/ex10.c > > > After doing the LU factorization, are the other operations such as > > MatMult() preserved ? > > Yes. > > Hong > > > > > Thanks > > > > Rgds, > > Amit > > > > From recrusader at gmail.com Tue Mar 4 01:47:40 2008 From: recrusader at gmail.com (Yujie) Date: Tue, 4 Mar 2008 15:47:40 +0800 Subject: MatGetSubMatrix() and MatGetSubMatrices() Message-ID: <7ff0ee010803032347v44221b12g1f499e4992bc4bc@mail.gmail.com> Hi, everyone #include "petscmat.h" PetscErrorCode PETSCMAT_DLLEXPORT MatGetSubMatrix(Mat mat,IS isrow,IS iscol,PetscInt csize,MatReuse cll,Mat *newmat) #include "petscmat.h" PetscErrorCode PETSCMAT_DLLEXPORT MatGetSubMatrices(Mat mat,PetscInt n,const IS irow[],const IS icol[],MatReuse scall,Mat *submat[]) To my knowledge, I can't get different matrices at different nodes by setting different isrows and iscol with MatGetSubMatrix(). Because isrow is rows current processor should obtain and the iscol argument must be the same on each processor. Furthermore, I can't get different parallel matrices at different nodes by setting different irow[] and icolp[] with MatGetSubMatrices() assuming I only want to get a matrix at each node. Because this function only gets different sequential matrices at different nodes. Is it right? Another equestion is how to partition the matrix when calling MatConvert() to convert a sequential matrix to a parallel one? Thanks a lot. Regards, Yujie -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Mar 4 06:53:05 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 4 Mar 2008 06:53:05 -0600 Subject: MatGetSubMatrix() and MatGetSubMatrices() In-Reply-To: <7ff0ee010803032347v44221b12g1f499e4992bc4bc@mail.gmail.com> References: <7ff0ee010803032347v44221b12g1f499e4992bc4bc@mail.gmail.com> Message-ID: <2BB26D6A-1233-4544-9DD6-6D082B980F45@mcs.anl.gov> On Mar 4, 2008, at 1:47 AM, Yujie wrote: > Hi, everyone > > #include "petscmat.h" > PetscErrorCode PETSCMAT_DLLEXPORT MatGetSubMatrix(Mat mat,IS > isrow,IS iscol,PetscInt csize,MatReuse cll,Mat *newmat) > #include "petscmat.h" > PetscErrorCode PETSCMAT_DLLEXPORT MatGetSubMatrices(Mat mat,PetscInt > n,const IS irow[],const IS icol[],MatReuse scall,Mat *submat[]) > > To my knowledge, I can't get different matrices at different nodes > by setting different isrows and iscol with MatGetSubMatrix(). > Because isrow is rows current processor should obtain and the iscol > argument must be the same on each processor. > Furthermore, I can't get different parallel matrices at different > nodes by setting different irow[] and icolp[] with > MatGetSubMatrices() assuming I only want to get a matrix at each node. > Because this function only gets different sequential matrices at > different nodes. Is it right? > MatGetSubMatrices() gets one or more sequential matrix on each process. MatGetSubMatrix() gets ONE parallel matrix that lives on the same communicator has the original matrix > Another equestion is how to partition the matrix when calling > MatConvert() to convert a sequential matrix to a parallel one? MatConvert() is not for converting sequential matrix to parallel; the communicator of a the new matrix from MatConvert() is always the same as the old one. In petsc-dev there is a utility for AIJ matrices that takes a given sequential AIJ matrix and makes it parallel, called /* Distributes a SeqAIJ matrix across a set of processes. Code stolen from MatLoad_MPIAIJ(). Horrible lack of reuse. Should be a routine for each matrix type. Only for square matrices */ PetscErrorCode MatDistribute_MPIAIJ(MPI_Comm comm,Mat gmat,PetscInt m,MatReuse reuse,Mat *inmat) Barry > > > Thanks a lot. > > Regards, > Yujie From recrusader at gmail.com Tue Mar 4 11:22:14 2008 From: recrusader at gmail.com (Yujie) Date: Tue, 4 Mar 2008 09:22:14 -0800 Subject: MatGetSubMatrix() and MatGetSubMatrices() In-Reply-To: <2BB26D6A-1233-4544-9DD6-6D082B980F45@mcs.anl.gov> References: <7ff0ee010803032347v44221b12g1f499e4992bc4bc@mail.gmail.com> <2BB26D6A-1233-4544-9DD6-6D082B980F45@mcs.anl.gov> Message-ID: <7ff0ee010803040922m482351ddr6e7f261023e7bc14@mail.gmail.com> Dear Barry: That is, if one wants to generate different sub-matrices at different nodes of the cluster from the same matrix, only MatGetSubMatrices() is available? If I use MatDistribute_MPIAIJ to convert a sequential matrix to parallel one, how to control which nodes of the cluster the submatrix of the parallel matrix is distributed into? For example, in a cluster, there are 10 nodes, I only want to use the specified 5 nodes for distributing the matrix. thanks a lot. Regards, Yujie On 3/4/08, Barry Smith wrote: > > > On Mar 4, 2008, at 1:47 AM, Yujie wrote: > > > Hi, everyone > > > > #include "petscmat.h" > > PetscErrorCode PETSCMAT_DLLEXPORT MatGetSubMatrix(Mat mat,IS > > isrow,IS iscol,PetscInt csize,MatReuse cll,Mat *newmat) > > #include "petscmat.h" > > PetscErrorCode PETSCMAT_DLLEXPORT MatGetSubMatrices(Mat mat,PetscInt > > n,const IS irow[],const IS icol[],MatReuse scall,Mat *submat[]) > > > > To my knowledge, I can't get different matrices at different nodes > > by setting different isrows and iscol with MatGetSubMatrix(). > > Because isrow is rows current processor should obtain and the iscol > > argument must be the same on each processor. > > Furthermore, I can't get different parallel matrices at different > > nodes by setting different irow[] and icolp[] with > > MatGetSubMatrices() assuming I only want to get a matrix at each node. > > Because this function only gets different sequential matrices at > > different nodes. Is it right? > > > > MatGetSubMatrices() gets one or more sequential matrix on each > process. > MatGetSubMatrix() gets ONE parallel matrix that lives on the same > communicator has the original matrix > > > > Another equestion is how to partition the matrix when calling > > MatConvert() to convert a sequential matrix to a parallel one? > > > MatConvert() is not for converting sequential matrix to parallel; > the communicator of a the new matrix from MatConvert() > is always the same as the old one. > > In petsc-dev there is a utility for AIJ matrices that takes a > given sequential AIJ matrix and makes it parallel, called > > /* > Distributes a SeqAIJ matrix across a set of processes. Code > stolen from > MatLoad_MPIAIJ(). Horrible lack of reuse. Should be a routine for > each matrix type. > > Only for square matrices > */ > PetscErrorCode MatDistribute_MPIAIJ(MPI_Comm comm,Mat gmat,PetscInt > m,MatReuse reuse,Mat *inmat) > > > > Barry > > > > > > > > > Thanks a lot. > > > > Regards, > > Yujie > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Amit.Itagi at seagate.com Tue Mar 4 11:36:04 2008 From: Amit.Itagi at seagate.com (Amit.Itagi at seagate.com) Date: Tue, 4 Mar 2008 12:36:04 -0500 Subject: Direct LU solver Message-ID: > > Are you using petsc-dev or a released version? > If petsc-dev with the newly released superlu_dist_2.2, > there is a bug in reusing parallel LU symbolic factorization > which I've reported to the developer of superlu_dist_2.2. > I am using petsc-2.3.3-p8 with the download_superlu=1 option. Looking at the externalPackages folder, I have superlu_dist 2.0. > Otherwise, > can you send me a simplified code that display the crash? > Then I may run it and see if petsc superlu_dist interface has bug. > You may also test other parallel direct LU solvers, e.g., > mumps or spooles. > I will create some simplified code that captures the problem. Thanks a lot, Hong. From Amit.Itagi at seagate.com Tue Mar 4 11:36:36 2008 From: Amit.Itagi at seagate.com (Amit.Itagi at seagate.com) Date: Tue, 4 Mar 2008 12:36:36 -0500 Subject: Direct LU solver Message-ID: Here is an example. I am creating the matrices, the vectors and the solver. I can do an LU solve on two processes just fine. However, when I destroy the entities and repeat the process (without any change), I get a crash. #include #include #include #include "petsc.h" #include "petscmat.h" #include "petscvec.h" #include "petscksp.h" using namespace std; int main( int argc, char *argv[] ) { int rank, size; Mat A; PetscErrorCode ierr; PetscInt loc; PetscScalar val; Vec x, y; KSP solver; PC prec; MPI_Comm comm; // Number of non-zeros in each row ierr=PetscInitialize(&argc,&argv,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); // Initialization (including MPI) comm=PETSC_COMM_WORLD; ierr=MPI_Comm_size(PETSC_COMM_WORLD,&size); CHKERRQ(ierr); ierr=MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr); //================== LU : First Time ============================= // Assemble matrix A PetscInt d_nnz=1, o_nnz=1; ierr=MatCreate(comm,&A); CHKERRQ(ierr); ierr=MatSetSizes(A,1,1,2,2); CHKERRQ(ierr); ierr=MatMPIAIJSetPreallocation(A,0,&d_nnz,0,&o_nnz); CHKERRQ(ierr); ierr=MatSetType(A,MATSUPERLU_DIST); CHKERRQ(ierr); ierr=MatSetFromOptions(A); CHKERRQ(ierr); if(rank==0) { val=complex(1.0,0.0); ierr=MatSetValue(A,0,0,val,ADD_VALUES);CHKERRQ(ierr); val=complex(0.0,1.0); ierr=MatSetValue(A,0,1,val,ADD_VALUES);CHKERRQ(ierr); } else { val=complex(1.0,1.0); ierr=MatSetValue(A,1,1,val,ADD_VALUES);CHKERRQ(ierr); val=complex(0.0,1.0); ierr=MatSetValue(A,1,0,val,ADD_VALUES);CHKERRQ(ierr); } ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); cout << "============ Mat A ==================" << endl; ierr=MatView(A,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); cout << "======================================" << endl; // Direct LU solver ierr=KSPCreate(comm,&solver); CHKERRQ(ierr); ierr=KSPSetOperators(solver,A,A,SAME_PRECONDITIONER); CHKERRQ(ierr); ierr=KSPSetType(solver,KSPPREONLY); CHKERRQ(ierr); ierr=KSPGetPC(solver,&prec); CHKERRQ(ierr); ierr=PCSetType(prec,PCLU); CHKERRQ(ierr); ierr=PCFactorSetShiftNonzero(prec,PETSC_DECIDE); ierr=KSPSetFromOptions(solver); CHKERRQ(ierr); //============ Vector assembly ======================== if(rank==0) { ierr=VecCreateMPI(PETSC_COMM_WORLD,1,2,&x); CHKERRQ(ierr); val=complex(1.0,0.0); loc=0; ierr=VecSetValues(x,1,&loc,&val,ADD_VALUES);CHKERRQ(ierr); } else { ierr=VecCreateMPI(PETSC_COMM_WORLD,1,2,&x); CHKERRQ(ierr); val=complex(-1.0,0.0); loc=1; ierr=VecSetValues(x,1,&loc,&val,ADD_VALUES);CHKERRQ(ierr); } ierr=VecAssemblyBegin(x); CHKERRQ(ierr); ierr=VecAssemblyEnd(x); CHKERRQ(ierr); cout << "============== Vec x ==================" << endl; ierr=VecView(x,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); cout << "======================================" << endl; VecDuplicate(x,&y); // Duplicate the matrix storage // Solve the matrix equation ierr=KSPSolve(solver,x,y); CHKERRQ(ierr); cout << "============== Vec y =================" << endl; ierr=VecView(y,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); cout << "======================================" << endl; // Destructors ierr=KSPDestroy(solver); CHKERRQ(ierr); ierr=VecDestroy(x); CHKERRQ(ierr); ierr=VecDestroy(y); CHKERRQ(ierr); ierr=MatDestroy(A); CHKERRQ(ierr); //============== LU Second time ============================== //======= Everything is exactly the same ===================== ierr=MatCreate(comm,&A); CHKERRQ(ierr); ierr=MatSetSizes(A,1,1,2,2); CHKERRQ(ierr); ierr=MatMPIAIJSetPreallocation(A,0,&d_nnz,0,&o_nnz); CHKERRQ(ierr); ierr=MatSetType(A,MATSUPERLU_DIST); CHKERRQ(ierr); ierr=MatSetFromOptions(A); CHKERRQ(ierr); if(rank==0) { val=complex(1.0,0.0); ierr=MatSetValue(A,0,0,val,ADD_VALUES);CHKERRQ(ierr); val=complex(0.0,1.0); ierr=MatSetValue(A,0,1,val,ADD_VALUES);CHKERRQ(ierr); } else { val=complex(1.0,1.0); ierr=MatSetValue(A,1,1,val,ADD_VALUES);CHKERRQ(ierr); val=complex(0.0,1.0); ierr=MatSetValue(A,1,0,val,ADD_VALUES);CHKERRQ(ierr); } ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); cout << "============ Mat A ==================" << endl; ierr=MatView(A,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); cout << "======================================" << endl; // Direct LU solver ierr=KSPCreate(comm,&solver); CHKERRQ(ierr); ierr=KSPSetOperators(solver,A,A,SAME_PRECONDITIONER); CHKERRQ(ierr); ierr=KSPSetType(solver,KSPPREONLY); CHKERRQ(ierr); ierr=KSPGetPC(solver,&prec); CHKERRQ(ierr); ierr=PCSetType(prec,PCLU); CHKERRQ(ierr); ierr=PCFactorSetShiftNonzero(prec,PETSC_DECIDE); ierr=KSPSetFromOptions(solver); CHKERRQ(ierr); //============ Vector assembly ======================== if(rank==0) { ierr=VecCreateMPI(PETSC_COMM_WORLD,1,2,&x); CHKERRQ(ierr); val=complex(1.0,0.0); loc=0; ierr=VecSetValues(x,1,&loc,&val,ADD_VALUES);CHKERRQ(ierr); } else { ierr=VecCreateMPI(PETSC_COMM_WORLD,1,2,&x); CHKERRQ(ierr); val=complex(-1.0,0.0); loc=1; ierr=VecSetValues(x,1,&loc,&val,ADD_VALUES);CHKERRQ(ierr); } ierr=VecAssemblyBegin(x); CHKERRQ(ierr); ierr=VecAssemblyEnd(x); CHKERRQ(ierr); cout << "============== Vec x ==================" << endl; ierr=VecView(x,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); cout << "======================================" << endl; VecDuplicate(x,&y); // Duplicate the matrix storage // Solve the matrix equation ierr=KSPSolve(solver,x,y); CHKERRQ(ierr); cout << "============== Vec y =================" << endl; ierr=VecView(y,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); cout << "======================================" << endl; // Destructors ierr=KSPDestroy(solver); CHKERRQ(ierr); ierr=VecDestroy(x); CHKERRQ(ierr); ierr=VecDestroy(y); CHKERRQ(ierr); ierr=MatDestroy(A); CHKERRQ(ierr); // Finalize ierr=PetscFinalize(); CHKERRQ(ierr); return 0; } Thanks Rgds, Amit Hong Zhang wrote on 03/03/2008 09:55:43 PM: > > Are you using petsc-dev or a released version? > If petsc-dev with the newly released superlu_dist_2.2, > there is a bug in reusing parallel LU symbolic factorization > which I've reported to the developer of superlu_dist_2.2. > > > > > Your comments were helpful. Now I can do the LU solves multiple times. > > However within the program, if I destroy the matrix and the KSP and create > > them again (for a different matrix), I get an error in superlu. > > Otherwise, > can you send me a simplified code that display the crash? > Then I may run it and see if petsc superlu_dist interface has bug. > You may also test other parallel direct LU solvers, e.g., > mumps or spooles. > > Hong > > > > > Here is the log from a successful run. > > > > [1] 0.249931 Event begin: KSPSetup > > [1] 0.24994 Event end: KSPSetup > > [1] 0.249948 Event begin: PCSetUp > > [1] 0.249963 Event begin: MatGetOrdering > > [0] 0.151691 Event begin: KSPSetup > > [0] 0.151699 Event end: KSPSetup > > [0] PCSetUp(): Setting up new PC > > [0] 0.151713 Event begin: PCSetUp > > [0] 0.151722 Event begin: MatGetOrdering > > [1] PetscCommDuplicate(): returning tag 2147483493 > > [0] PetscCommDuplicate(): returning tag 2147483493 > > [1] PetscCommDuplicate(): returning tag 2147483492 > > [0] PetscCommDuplicate(): returning tag 2147483492 > > [1] 0.250169 Event end: MatGetOrdering > > [1] 0.250182 Event begin: MatLUFactorSym > > [0] 0.151885 Event end: MatGetOrdering > > [0] 0.151896 Event begin: MatLUFactorSym > > [1] PetscCommDuplicate(): returning tag 2147483491 > > [0] PetscCommDuplicate(): returning tag 2147483491 > > [1] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ > LU factorization and solves. > > [0] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ > LU factorization and solves. > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1083412768 143685848 > > [1] PetscCommDuplicate(): returning tag 2147483627 > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1083412768 143685848 > > [1] PetscCommDuplicate(): returning tag 2147483626 > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1083412768 144562608 > > [0] PetscCommDuplicate(): returning tag 2147483627 > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1083412768 144562608 > > [0] PetscCommDuplicate(): returning tag 2147483626 > > [1] 0.252072 Event end: MatLUFactorSym > > [1] 0.252098 Event begin: MatLUFactorNum > > [0] 0.153821 Event end: MatLUFactorSym > > [0] 0.153835 Event begin: MatLUFactorNum > > Nonzeros in L 171877 > > Nonzeros in U 171877 > > nonzeros in L+U-I 342195 > > nonzeros in LSUB 52137 > > Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): > > 0.000296116 / 0.00028491 / 0.000290513 > > [1] 0.623389 Event end: MatLUFactorNum > > [1] 0.623425 Event end: PCSetUp > > EQUIL time 0.00 > > ROWPERM time 0.01 > > COLPERM time 0.01 > > SYMBFACT time 0.01 > > DISTRIBUTE time 0.02 > > FACTOR time 0.29 > > Factor flops 2.652935e+08 Mflops 924.82 > > SOLVE time 0.00 > > [0] 0.525149 Event end: MatLUFactorNum > > [0] 0.525167 Event end: PCSetUp > > > > Here is the log from a crash. > > > > [0] 1.41665 Event begin: KSPSetup > > [0] 1.41666 Event end: KSPSetup > > [0] PCSetUp(): Setting up new PC > > [0] 1.41668 Event begin: PCSetUp > > [0] 1.41669 Event begin: MatGetOrdering > > [1] 1.51505 Event begin: KSPSetup > > [1] 1.51506 Event end: KSPSetup > > [1] 1.51507 Event begin: PCSetUp > > [1] 1.51507 Event begin: MatGetOrdering > > [0] PetscCommDuplicate(): returning tag 2147483226 > > [1] PetscCommDuplicate(): returning tag 2147483226 > > [0] PetscCommDuplicate(): returning tag 2147483225 > > [1] PetscCommDuplicate(): returning tag 2147483225 > > [0] 1.4169 Event end: MatGetOrdering > > [0] 1.41692 Event begin: MatLUFactorSym > > [1] 1.51526 Event end: MatGetOrdering > > [1] 1.51526 Event begin: MatLUFactorSym > > [0] PetscCommDuplicate(): returning tag 2147483224 > > [1] PetscCommDuplicate(): returning tag 2147483224 > > [0] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ > LU factorization and solves. > > [1] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ > LU factorization and solves. > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1083412768 144562608 > > [0] PetscCommDuplicate(): returning tag 2147483627 > > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1083412768 144562608 > > [0] PetscCommDuplicate(): returning tag 2147483626 > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1083412768 143685848 > > [1] PetscCommDuplicate(): returning tag 2147483627 > > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1083412768 143685848 > > [1] PetscCommDuplicate(): returning tag 2147483626 > > [0] 1.4187 Event end: MatLUFactorSym > > [0] 1.41872 Event begin: MatLUFactorNum > > [1] 1.51706 Event end: MatLUFactorSym > > [1] 1.51707 Event begin: MatLUFactorNum > > MPI_Alltoallv: invalid datatype argument: Invalid argument (rank 0, comm 5) > > Rank (0, MPI_COMM_WORLD): Call stack within LAM: > > Rank (0, MPI_COMM_WORLD): - MPI_Alltoallv() > > Rank (0, MPI_COMM_WORLD): - main() > > > > If I trace the place of the error in the debugger, the trace gives > > > > KSPSolve() -> KSPSetUp() -> PCSetUp() -> PCSetUp_LU() -> > MatLuFactorNumeric() -> MatLuFactorNumeric_SuperLU_Dist() > > -> pzgssvx() -> pzCompRow_loc_to_CompCol_global() -> MPI_Alltoallv > > > > > > I could not make much head-way by looking at the errors. Could you > give me some tips on what might be causing this error, and what to look for ? > > > > > > Thanks > > > > Rgds, > > Amit > > > > owner-petsc-users at mcs.anl.gov wrote on 03/03/2008 10:55:21 AM: > > > >> > >> Amit, > >> > >>> Thanks for your response. A couple of follow-up questions - > >>> > >>> I go over the steps > >>> > >>> MatCreate() > >>> MatSetType(MPIAIJ) > >>> MatMPIAIJSetPreallocation() > >>> MatSetType(MATSUPERLU_DIST). > >>> > >>> Even though I want to do the direct LU solves repeatedly (with the same > >>> matrix), I don't want the program to do LU factorization repeatedly. I > > hope > >>> I can get that functionality by using the "SAME_PRECONDITIONER" option > >>> (along with -ksppreonly), while defining the KSP. When the program does > > the > >>> factorization, does it do it in-place or does it allocate new storage ? > >> > >> allocate new storage unless you call > >> MatLUFactor() explicitly. > >> > >> It is done by > >> KSPSetUp() > >> which calls PCSetUp() -> PCSetUp_LU() > >> -> MatLUFactorSymbolic() & MatLUFactorNumeric( > >> > >> You can do following for reusing factored L and U: > >> KSPCreate() > >> KSPSetOperators() > >> KSPSetFromOptions() > >> KSPSetUp() > >> while ( num_rhs-- ) { > >> KSPSolve(ksp,b,x); > >> } > >> > >> See src/ksp/ksp/examples/tutorials/ex10.c > >> > >>> After doing the LU factorization, are the other operations such as > >>> MatMult() preserved ? > >> > >> Yes. > >> > >> Hong > >> > >>> > >>> Thanks > >>> > >>> Rgds, > >>> Amit > >>> > >>> > > > > From hzhang at mcs.anl.gov Tue Mar 4 11:51:31 2008 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 4 Mar 2008 11:51:31 -0600 (CST) Subject: Direct LU solver In-Reply-To: References: Message-ID: Amit, I'll test it and let you know the result. Hong On Tue, 4 Mar 2008, Amit.Itagi at seagate.com wrote: > > > > Here is an example. I am creating the matrices, the vectors and the solver. > I can do an LU solve on two processes just fine. However, when I destroy > the entities and repeat the process (without any change), I get a crash. > > #include > #include > #include > #include "petsc.h" > #include "petscmat.h" > #include "petscvec.h" > #include "petscksp.h" > > using namespace std; > > int main( int argc, char *argv[] ) { > > int rank, size; > Mat A; > PetscErrorCode ierr; > PetscInt loc; > PetscScalar val; > Vec x, y; > KSP solver; > PC prec; > MPI_Comm comm; > > // Number of non-zeros in each row > > ierr=PetscInitialize(&argc,&argv,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); > // Initialization (including MPI) > > comm=PETSC_COMM_WORLD; > > ierr=MPI_Comm_size(PETSC_COMM_WORLD,&size); CHKERRQ(ierr); > ierr=MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr); > > > //================== LU : First Time ============================= > > // Assemble matrix A > > PetscInt d_nnz=1, o_nnz=1; > ierr=MatCreate(comm,&A); CHKERRQ(ierr); > ierr=MatSetSizes(A,1,1,2,2); CHKERRQ(ierr); > ierr=MatMPIAIJSetPreallocation(A,0,&d_nnz,0,&o_nnz); CHKERRQ(ierr); > ierr=MatSetType(A,MATSUPERLU_DIST); CHKERRQ(ierr); > ierr=MatSetFromOptions(A); CHKERRQ(ierr); > > > if(rank==0) { > val=complex(1.0,0.0); > ierr=MatSetValue(A,0,0,val,ADD_VALUES);CHKERRQ(ierr); > val=complex(0.0,1.0); > ierr=MatSetValue(A,0,1,val,ADD_VALUES);CHKERRQ(ierr); > } > else { > val=complex(1.0,1.0); > ierr=MatSetValue(A,1,1,val,ADD_VALUES);CHKERRQ(ierr); > val=complex(0.0,1.0); > ierr=MatSetValue(A,1,0,val,ADD_VALUES);CHKERRQ(ierr); > } > > ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > > cout << "============ Mat A ==================" << endl; > ierr=MatView(A,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > cout << "======================================" << endl; > > // Direct LU solver > ierr=KSPCreate(comm,&solver); CHKERRQ(ierr); > ierr=KSPSetOperators(solver,A,A,SAME_PRECONDITIONER); CHKERRQ(ierr); > ierr=KSPSetType(solver,KSPPREONLY); CHKERRQ(ierr); > ierr=KSPGetPC(solver,&prec); CHKERRQ(ierr); > ierr=PCSetType(prec,PCLU); CHKERRQ(ierr); > ierr=PCFactorSetShiftNonzero(prec,PETSC_DECIDE); > ierr=KSPSetFromOptions(solver); CHKERRQ(ierr); > > //============ Vector assembly ======================== > > if(rank==0) { > ierr=VecCreateMPI(PETSC_COMM_WORLD,1,2,&x); CHKERRQ(ierr); > val=complex(1.0,0.0); > loc=0; > ierr=VecSetValues(x,1,&loc,&val,ADD_VALUES);CHKERRQ(ierr); > } > else { > ierr=VecCreateMPI(PETSC_COMM_WORLD,1,2,&x); CHKERRQ(ierr); > val=complex(-1.0,0.0); > loc=1; > ierr=VecSetValues(x,1,&loc,&val,ADD_VALUES);CHKERRQ(ierr); > } > > ierr=VecAssemblyBegin(x); CHKERRQ(ierr); > ierr=VecAssemblyEnd(x); CHKERRQ(ierr); > > cout << "============== Vec x ==================" << endl; > ierr=VecView(x,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > cout << "======================================" << endl; > > VecDuplicate(x,&y); // Duplicate the matrix storage > > // Solve the matrix equation > ierr=KSPSolve(solver,x,y); CHKERRQ(ierr); > > cout << "============== Vec y =================" << endl; > ierr=VecView(y,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > cout << "======================================" << endl; > > > // Destructors > ierr=KSPDestroy(solver); CHKERRQ(ierr); > ierr=VecDestroy(x); CHKERRQ(ierr); > ierr=VecDestroy(y); CHKERRQ(ierr); > ierr=MatDestroy(A); CHKERRQ(ierr); > > > //============== LU Second time ============================== > //======= Everything is exactly the same ===================== > > ierr=MatCreate(comm,&A); CHKERRQ(ierr); > ierr=MatSetSizes(A,1,1,2,2); CHKERRQ(ierr); > ierr=MatMPIAIJSetPreallocation(A,0,&d_nnz,0,&o_nnz); CHKERRQ(ierr); > ierr=MatSetType(A,MATSUPERLU_DIST); CHKERRQ(ierr); > ierr=MatSetFromOptions(A); CHKERRQ(ierr); > > > if(rank==0) { > val=complex(1.0,0.0); > ierr=MatSetValue(A,0,0,val,ADD_VALUES);CHKERRQ(ierr); > val=complex(0.0,1.0); > ierr=MatSetValue(A,0,1,val,ADD_VALUES);CHKERRQ(ierr); > } > else { > val=complex(1.0,1.0); > ierr=MatSetValue(A,1,1,val,ADD_VALUES);CHKERRQ(ierr); > val=complex(0.0,1.0); > ierr=MatSetValue(A,1,0,val,ADD_VALUES);CHKERRQ(ierr); > } > > ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > > cout << "============ Mat A ==================" << endl; > ierr=MatView(A,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > cout << "======================================" << endl; > > // Direct LU solver > ierr=KSPCreate(comm,&solver); CHKERRQ(ierr); > ierr=KSPSetOperators(solver,A,A,SAME_PRECONDITIONER); CHKERRQ(ierr); > ierr=KSPSetType(solver,KSPPREONLY); CHKERRQ(ierr); > ierr=KSPGetPC(solver,&prec); CHKERRQ(ierr); > ierr=PCSetType(prec,PCLU); CHKERRQ(ierr); > ierr=PCFactorSetShiftNonzero(prec,PETSC_DECIDE); > ierr=KSPSetFromOptions(solver); CHKERRQ(ierr); > > //============ Vector assembly ======================== > > if(rank==0) { > ierr=VecCreateMPI(PETSC_COMM_WORLD,1,2,&x); CHKERRQ(ierr); > val=complex(1.0,0.0); > loc=0; > ierr=VecSetValues(x,1,&loc,&val,ADD_VALUES);CHKERRQ(ierr); > } > else { > ierr=VecCreateMPI(PETSC_COMM_WORLD,1,2,&x); CHKERRQ(ierr); > val=complex(-1.0,0.0); > loc=1; > ierr=VecSetValues(x,1,&loc,&val,ADD_VALUES);CHKERRQ(ierr); > } > > ierr=VecAssemblyBegin(x); CHKERRQ(ierr); > ierr=VecAssemblyEnd(x); CHKERRQ(ierr); > > cout << "============== Vec x ==================" << endl; > ierr=VecView(x,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > cout << "======================================" << endl; > > VecDuplicate(x,&y); // Duplicate the matrix storage > > // Solve the matrix equation > ierr=KSPSolve(solver,x,y); CHKERRQ(ierr); > > cout << "============== Vec y =================" << endl; > ierr=VecView(y,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > cout << "======================================" << endl; > > > // Destructors > ierr=KSPDestroy(solver); CHKERRQ(ierr); > ierr=VecDestroy(x); CHKERRQ(ierr); > ierr=VecDestroy(y); CHKERRQ(ierr); > ierr=MatDestroy(A); CHKERRQ(ierr); > > > // Finalize > ierr=PetscFinalize(); CHKERRQ(ierr); > > > return 0; > > } > > Thanks > > Rgds, > Amit > > Hong Zhang wrote on 03/03/2008 09:55:43 PM: > >> >> Are you using petsc-dev or a released version? >> If petsc-dev with the newly released superlu_dist_2.2, >> there is a bug in reusing parallel LU symbolic factorization >> which I've reported to the developer of superlu_dist_2.2. >> >>> >>> Your comments were helpful. Now I can do the LU solves multiple times. >>> However within the program, if I destroy the matrix and the KSP and > create >>> them again (for a different matrix), I get an error in superlu. >> >> Otherwise, >> can you send me a simplified code that display the crash? >> Then I may run it and see if petsc superlu_dist interface has bug. >> You may also test other parallel direct LU solvers, e.g., >> mumps or spooles. >> >> Hong >> >>> >>> Here is the log from a successful run. >>> >>> [1] 0.249931 Event begin: KSPSetup >>> [1] 0.24994 Event end: KSPSetup >>> [1] 0.249948 Event begin: PCSetUp >>> [1] 0.249963 Event begin: MatGetOrdering >>> [0] 0.151691 Event begin: KSPSetup >>> [0] 0.151699 Event end: KSPSetup >>> [0] PCSetUp(): Setting up new PC >>> [0] 0.151713 Event begin: PCSetUp >>> [0] 0.151722 Event begin: MatGetOrdering >>> [1] PetscCommDuplicate(): returning tag 2147483493 >>> [0] PetscCommDuplicate(): returning tag 2147483493 >>> [1] PetscCommDuplicate(): returning tag 2147483492 >>> [0] PetscCommDuplicate(): returning tag 2147483492 >>> [1] 0.250169 Event end: MatGetOrdering >>> [1] 0.250182 Event begin: MatLUFactorSym >>> [0] 0.151885 Event end: MatGetOrdering >>> [0] 0.151896 Event begin: MatLUFactorSym >>> [1] PetscCommDuplicate(): returning tag 2147483491 >>> [0] PetscCommDuplicate(): returning tag 2147483491 >>> [1] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ >> LU factorization and solves. >>> [0] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ >> LU factorization and solves. >>> [1] PetscCommDuplicate(): Using internal PETSc communicator >> 1083412768 143685848 >>> [1] PetscCommDuplicate(): returning tag 2147483627 >>> [1] PetscCommDuplicate(): Using internal PETSc communicator >> 1083412768 143685848 >>> [1] PetscCommDuplicate(): returning tag 2147483626 >>> [0] PetscCommDuplicate(): Using internal PETSc communicator >> 1083412768 144562608 >>> [0] PetscCommDuplicate(): returning tag 2147483627 >>> [0] PetscCommDuplicate(): Using internal PETSc communicator >> 1083412768 144562608 >>> [0] PetscCommDuplicate(): returning tag 2147483626 >>> [1] 0.252072 Event end: MatLUFactorSym >>> [1] 0.252098 Event begin: MatLUFactorNum >>> [0] 0.153821 Event end: MatLUFactorSym >>> [0] 0.153835 Event begin: MatLUFactorNum >>> Nonzeros in L 171877 >>> Nonzeros in U 171877 >>> nonzeros in L+U-I 342195 >>> nonzeros in LSUB 52137 >>> Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): >>> 0.000296116 / 0.00028491 / 0.000290513 >>> [1] 0.623389 Event end: MatLUFactorNum >>> [1] 0.623425 Event end: PCSetUp >>> EQUIL time 0.00 >>> ROWPERM time 0.01 >>> COLPERM time 0.01 >>> SYMBFACT time 0.01 >>> DISTRIBUTE time 0.02 >>> FACTOR time 0.29 >>> Factor flops 2.652935e+08 Mflops 924.82 >>> SOLVE time 0.00 >>> [0] 0.525149 Event end: MatLUFactorNum >>> [0] 0.525167 Event end: PCSetUp >>> >>> Here is the log from a crash. >>> >>> [0] 1.41665 Event begin: KSPSetup >>> [0] 1.41666 Event end: KSPSetup >>> [0] PCSetUp(): Setting up new PC >>> [0] 1.41668 Event begin: PCSetUp >>> [0] 1.41669 Event begin: MatGetOrdering >>> [1] 1.51505 Event begin: KSPSetup >>> [1] 1.51506 Event end: KSPSetup >>> [1] 1.51507 Event begin: PCSetUp >>> [1] 1.51507 Event begin: MatGetOrdering >>> [0] PetscCommDuplicate(): returning tag 2147483226 >>> [1] PetscCommDuplicate(): returning tag 2147483226 >>> [0] PetscCommDuplicate(): returning tag 2147483225 >>> [1] PetscCommDuplicate(): returning tag 2147483225 >>> [0] 1.4169 Event end: MatGetOrdering >>> [0] 1.41692 Event begin: MatLUFactorSym >>> [1] 1.51526 Event end: MatGetOrdering >>> [1] 1.51526 Event begin: MatLUFactorSym >>> [0] PetscCommDuplicate(): returning tag 2147483224 >>> [1] PetscCommDuplicate(): returning tag 2147483224 >>> [0] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ >> LU factorization and solves. >>> [1] MatConvert_AIJ_SuperLU_DIST(): Using SuperLU_DIST for SeqAIJ >> LU factorization and solves. >>> [0] PetscCommDuplicate(): Using internal PETSc communicator >> 1083412768 144562608 >>> [0] PetscCommDuplicate(): returning tag 2147483627 >>> [0] PetscCommDuplicate(): Using internal PETSc communicator >> 1083412768 144562608 >>> [0] PetscCommDuplicate(): returning tag 2147483626 >>> [1] PetscCommDuplicate(): Using internal PETSc communicator >> 1083412768 143685848 >>> [1] PetscCommDuplicate(): returning tag 2147483627 >>> [1] PetscCommDuplicate(): Using internal PETSc communicator >> 1083412768 143685848 >>> [1] PetscCommDuplicate(): returning tag 2147483626 >>> [0] 1.4187 Event end: MatLUFactorSym >>> [0] 1.41872 Event begin: MatLUFactorNum >>> [1] 1.51706 Event end: MatLUFactorSym >>> [1] 1.51707 Event begin: MatLUFactorNum >>> MPI_Alltoallv: invalid datatype argument: Invalid argument (rank 0, > comm 5) >>> Rank (0, MPI_COMM_WORLD): Call stack within LAM: >>> Rank (0, MPI_COMM_WORLD): - MPI_Alltoallv() >>> Rank (0, MPI_COMM_WORLD): - main() >>> >>> If I trace the place of the error in the debugger, the trace gives >>> >>> KSPSolve() -> KSPSetUp() -> PCSetUp() -> PCSetUp_LU() -> >> MatLuFactorNumeric() -> MatLuFactorNumeric_SuperLU_Dist() >>> -> pzgssvx() -> pzCompRow_loc_to_CompCol_global() -> MPI_Alltoallv >>> >>> >>> I could not make much head-way by looking at the errors. Could you >> give me some tips on what might be causing this error, and what to look > for ? >>> >>> >>> Thanks >>> >>> Rgds, >>> Amit >>> >>> owner-petsc-users at mcs.anl.gov wrote on 03/03/2008 10:55:21 AM: >>> >>>> >>>> Amit, >>>> >>>>> Thanks for your response. A couple of follow-up questions - >>>>> >>>>> I go over the steps >>>>> >>>>> MatCreate() >>>>> MatSetType(MPIAIJ) >>>>> MatMPIAIJSetPreallocation() >>>>> MatSetType(MATSUPERLU_DIST). >>>>> >>>>> Even though I want to do the direct LU solves repeatedly (with the > same >>>>> matrix), I don't want the program to do LU factorization repeatedly. > I >>> hope >>>>> I can get that functionality by using the "SAME_PRECONDITIONER" > option >>>>> (along with -ksppreonly), while defining the KSP. When the program > does >>> the >>>>> factorization, does it do it in-place or does it allocate new storage > ? >>>> >>>> allocate new storage unless you call >>>> MatLUFactor() explicitly. >>>> >>>> It is done by >>>> KSPSetUp() >>>> which calls PCSetUp() -> PCSetUp_LU() >>>> -> MatLUFactorSymbolic() & MatLUFactorNumeric( >>>> >>>> You can do following for reusing factored L and U: >>>> KSPCreate() >>>> KSPSetOperators() >>>> KSPSetFromOptions() >>>> KSPSetUp() >>>> while ( num_rhs-- ) { >>>> KSPSolve(ksp,b,x); >>>> } >>>> >>>> See src/ksp/ksp/examples/tutorials/ex10.c >>>> >>>>> After doing the LU factorization, are the other operations such as >>>>> MatMult() preserved ? >>>> >>>> Yes. >>>> >>>> Hong >>>> >>>>> >>>>> Thanks >>>>> >>>>> Rgds, >>>>> Amit >>>>> >>>>> >>> >>> > > From bsmith at mcs.anl.gov Tue Mar 4 13:48:21 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 4 Mar 2008 13:48:21 -0600 Subject: MatGetSubMatrix() and MatGetSubMatrices() In-Reply-To: <7ff0ee010803040922m482351ddr6e7f261023e7bc14@mail.gmail.com> References: <7ff0ee010803032347v44221b12g1f499e4992bc4bc@mail.gmail.com> <2BB26D6A-1233-4544-9DD6-6D082B980F45@mcs.anl.gov> <7ff0ee010803040922m482351ddr6e7f261023e7bc14@mail.gmail.com> Message-ID: <2B1BE7F1-6FC4-4154-BA44-480DF142F69F@mcs.anl.gov> On Mar 4, 2008, at 11:22 AM, Yujie wrote: > Dear Barry: > > That is, if one wants to generate different sub-matrices at > different nodes of the cluster from the same matrix, only > MatGetSubMatrices() is available? Yes > > > If I use MatDistribute_MPIAIJ to convert a sequential matrix to > parallel one, how to control which nodes of the cluster the > submatrix of the parallel matrix is distributed into? For example, > in a cluster, there are 10 nodes, I only want to use the specified 5 > nodes for > distributing the matrix. thanks a lot. That is the comm argument to the function. In PETSc all objects are laid out on an MPI_Comm, thus using subsets of nodes for various objects is very easy, just create the appropriate MPI_Comm and use it to create the object. Barry > > > Regards, > Yujie > > > On 3/4/08, Barry Smith wrote: > > On Mar 4, 2008, at 1:47 AM, Yujie wrote: > > > Hi, everyone > > > > #include "petscmat.h" > > PetscErrorCode PETSCMAT_DLLEXPORT MatGetSubMatrix(Mat mat,IS > > isrow,IS iscol,PetscInt csize,MatReuse cll,Mat *newmat) > > #include "petscmat.h" > > PetscErrorCode PETSCMAT_DLLEXPORT MatGetSubMatrices(Mat mat,PetscInt > > n,const IS irow[],const IS icol[],MatReuse scall,Mat *submat[]) > > > > To my knowledge, I can't get different matrices at different nodes > > by setting different isrows and iscol with MatGetSubMatrix(). > > Because isrow is rows current processor should obtain and the iscol > > argument must be the same on each processor. > > Furthermore, I can't get different parallel matrices at different > > nodes by setting different irow[] and icolp[] with > > MatGetSubMatrices() assuming I only want to get a matrix at each > node. > > Because this function only gets different sequential matrices at > > different nodes. Is it right? > > > > MatGetSubMatrices() gets one or more sequential matrix on each > process. > MatGetSubMatrix() gets ONE parallel matrix that lives on the same > communicator has the original matrix > > > > Another equestion is how to partition the matrix when calling > > MatConvert() to convert a sequential matrix to a parallel one? > > > MatConvert() is not for converting sequential matrix to parallel; > the communicator of a the new matrix from MatConvert() > is always the same as the old one. > > In petsc-dev there is a utility for AIJ matrices that takes a > given sequential AIJ matrix and makes it parallel, called > > /* > Distributes a SeqAIJ matrix across a set of processes. Code > stolen from > MatLoad_MPIAIJ(). Horrible lack of reuse. Should be a routine for > each matrix type. > > Only for square matrices > */ > PetscErrorCode MatDistribute_MPIAIJ(MPI_Comm comm,Mat gmat,PetscInt > m,MatReuse reuse,Mat *inmat) > > > > Barry > > > > > > > > > Thanks a lot. > > > > Regards, > > Yujie > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From recrusader at gmail.com Tue Mar 4 18:04:09 2008 From: recrusader at gmail.com (Yujie) Date: Tue, 4 Mar 2008 16:04:09 -0800 Subject: MatMultConstrained() is canclled? Message-ID: <7ff0ee010803041604h7a5262b4kd44cf46cc683f8d7@mail.gmail.com> Hi, everyone I am trying to use MatMultConstrained() function. When I want to check its realization. I can't find any functions like MatMultConstrained_*****(). However there is MatMultConstrained() in "src/mat/interface/matrix.c". Regards, Yujie -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Mar 4 20:23:04 2008 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 4 Mar 2008 20:23:04 -0600 Subject: MatMultConstrained() is canclled? In-Reply-To: <7ff0ee010803041604h7a5262b4kd44cf46cc683f8d7@mail.gmail.com> References: <7ff0ee010803041604h7a5262b4kd44cf46cc683f8d7@mail.gmail.com> Message-ID: It no longer exists. I forgot to remove it. Matt On Tue, Mar 4, 2008 at 6:04 PM, Yujie wrote: > Hi, everyone > > I am trying to use MatMultConstrained() function. When I want to check its > realization. I can't find any functions like MatMultConstrained_*****(). > However there is MatMultConstrained() in "src/mat/interface/matrix.c". > > Regards, > Yujie > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From etienne.perchat at transvalor.com Wed Mar 5 03:43:52 2008 From: etienne.perchat at transvalor.com (Etienne PERCHAT) Date: Wed, 5 Mar 2008 10:43:52 +0100 Subject: win64 question Message-ID: <9113A52E1096EB41B1F88DD94C4369D511FD96@EXCHSRV.transvalor.com> Hello, I would like to compile Petsc on x64 platforms under windows. I guess that win32fe might not be used but I didn't find something like win64fe. Does somebody have any tips? I have browsed the Installation informations as well as the mailing list archives without success (sorry to bother everybody if I missed something). Thanks in advance, Etienne Perchat -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Mar 5 08:33:20 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 5 Mar 2008 08:33:20 -0600 (CST) Subject: win64 question In-Reply-To: <9113A52E1096EB41B1F88DD94C4369D511FD96@EXCHSRV.transvalor.com> References: <9113A52E1096EB41B1F88DD94C4369D511FD96@EXCHSRV.transvalor.com> Message-ID: Win32fe [& cygwintools] are used only as part of the build enviornment. So 32bit version of these tools don't prevent 64bit builds. Internally win32fe will invoke 'cl' with correct options. Here 'cl' will run in 64 bit mode - thus producing the correct 64bit object files. Note: you might have to follow the instructions in the "Using Visual Studio 2005" section of the installation insctuctions. Make sure when you use the 'VisualStudio CMD' - its the 64bit compiler version [and then use win32fe --nodetect' options. Satish On Wed, 5 Mar 2008, Etienne PERCHAT wrote: > Hello, > > > > I would like to compile Petsc on x64 platforms under windows. > > I guess that win32fe might not be used but I didn't find something like > win64fe. > > > > Does somebody have any tips? > > > > I have browsed the Installation informations as well as the mailing list > archives without success (sorry to bother everybody if I missed > something). > > > > Thanks in advance, > > > > Etienne Perchat > > From etienne.perchat at transvalor.com Wed Mar 5 08:44:02 2008 From: etienne.perchat at transvalor.com (Etienne PERCHAT) Date: Wed, 5 Mar 2008 15:44:02 +0100 Subject: win64 question Message-ID: <9113A52E1096EB41B1F88DD94C4369D511FDBC@EXCHSRV.transvalor.com> OK, Thank you very much Satish for this very fast answer. Best regards, Etienne -----Message d'origine----- De?: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] De la part de Satish Balay Envoy??: mercredi 5 mars 2008 15:33 ??: petsc-users at mcs.anl.gov Objet?: Re: win64 question Win32fe [& cygwintools] are used only as part of the build enviornment. So 32bit version of these tools don't prevent 64bit builds. Internally win32fe will invoke 'cl' with correct options. Here 'cl' will run in 64 bit mode - thus producing the correct 64bit object files. Note: you might have to follow the instructions in the "Using Visual Studio 2005" section of the installation insctuctions. Make sure when you use the 'VisualStudio CMD' - its the 64bit compiler version [and then use win32fe --nodetect' options. Satish On Wed, 5 Mar 2008, Etienne PERCHAT wrote: > Hello, > > > > I would like to compile Petsc on x64 platforms under windows. > > I guess that win32fe might not be used but I didn't find something like > win64fe. > > > > Does somebody have any tips? > > > > I have browsed the Installation informations as well as the mailing list > archives without success (sorry to bother everybody if I missed > something). > > > > Thanks in advance, > > > > Etienne Perchat > > From grs2103 at columbia.edu Wed Mar 5 18:00:47 2008 From: grs2103 at columbia.edu (Gideon Simpson) Date: Wed, 5 Mar 2008 19:00:47 -0500 Subject: out of memory error Message-ID: I'm getting the following error with some code, running on a serial machine with, i think, 3 gigs of memory. Is there anyway to circumvent this error by being clever, or do I really need to go to a distributed memory machine? -gideon Out of memory trying to allocate 112032952 bytes [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly illegal memory access [0]PETSC ERROR: Try option -start_in_debugger or - on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal [0]PETSC ERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PCSetUp_HYPRE line 95 src/ksp/pc/impls/hypre/hypre.c [0]PETSC ERROR: [0] PCSetUp line 764 src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSPSetUp line 183 src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] KSPSolve line 305 src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 8, Fri Nov 16 17:03:40 CST 2007 HG revision: 414581156e67e55c761739b0deb119f7590d0f4b [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./bulk3d on a darwin8.1 named valkyrie.appmath.columbia.edu by gideon Wed Mar 5 15:53:26 2008 [0]PETSC ERROR: Libraries linked from /Users/gideon/software/ petsc-2.3.3-p8/lib/darwin8.11.1-cxx-debug [0]PETSC ERROR: Configure run at Thu Jan 10 11:00:45 2008 [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-mpi-dir=/ Users/gideon/software --with-mpi-shared=1 --with-x11 --download- hypre=1 --with-hypre=1 --download-umfpack=1 --with-umfpack=1 --with- shared=1 --with-dynamic=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file From aja2111 at columbia.edu Wed Mar 5 19:04:19 2008 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Wed, 5 Mar 2008 20:04:19 -0500 Subject: out of memory error In-Reply-To: References: Message-ID: <37604ab40803051704g1dc69af8y5547658189cb3d18@mail.gmail.com> 112032952 bytes is about 100 MB. Are you really running out of memory or is something else going on? ~A On Wed, Mar 5, 2008 at 7:00 PM, Gideon Simpson wrote: > I'm getting the following error with some code, running on a serial > machine with, i think, 3 gigs of memory. Is there anyway to > circumvent this error by being clever, or do I really need to go to a > distributed memory machine? > > -gideon > > Out of memory trying to allocate 112032952 bytes > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly > illegal memory access > [0]PETSC ERROR: Try option -start_in_debugger or - > on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal > [0]PETSC ERROR: or try http://valgrind.org on linux or man libgmalloc > on Apple to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the > function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PCSetUp_HYPRE line 95 src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: [0] PCSetUp line 764 src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPSetUp line 183 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] KSPSolve line 305 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 8, Fri Nov 16 > 17:03:40 CST 2007 HG revision: 414581156e67e55c761739b0deb119f7590d0f4b > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./bulk3d on a darwin8.1 named > valkyrie.appmath.columbia.edu by gideon Wed Mar 5 15:53:26 2008 > [0]PETSC ERROR: Libraries linked from /Users/gideon/software/ > petsc-2.3.3-p8/lib/darwin8.11.1-cxx-debug > [0]PETSC ERROR: Configure run at Thu Jan 10 11:00:45 2008 > [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-mpi-dir=/ > Users/gideon/software --with-mpi-shared=1 --with-x11 --download- > hypre=1 --with-hypre=1 --download-umfpack=1 --with-umfpack=1 --with- > shared=1 --with-dynamic=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > > > > From bsmith at mcs.anl.gov Wed Mar 5 19:20:16 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 5 Mar 2008 19:20:16 -0600 Subject: out of memory error In-Reply-To: References: Message-ID: You could run with -start_in_debugger noxterm then type continue to see exactly why/where it is crashing. Barry On Mar 5, 2008, at 6:00 PM, Gideon Simpson wrote: > I'm getting the following error with some code, running on a serial > machine with, i think, 3 gigs of memory. Is there anyway to > circumvent this error by being clever, or do I really need to go to > a distributed memory machine? > > -gideon > > Out of memory trying to allocate 112032952 bytes > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly > illegal memory access > [0]PETSC ERROR: Try option -start_in_debugger or - > on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal > [0]PETSC ERROR: or try http://valgrind.org on linux or man > libgmalloc on Apple to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the > function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PCSetUp_HYPRE line 95 src/ksp/pc/impls/hypre/ > hypre.c > [0]PETSC ERROR: [0] PCSetUp line 764 src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPSetUp line 183 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] KSPSolve line 305 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 8, Fri Nov 16 > 17:03:40 CST 2007 HG revision: > 414581156e67e55c761739b0deb119f7590d0f4b > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./bulk3d on a darwin8.1 named > valkyrie.appmath.columbia.edu by gideon Wed Mar 5 15:53:26 2008 > [0]PETSC ERROR: Libraries linked from /Users/gideon/software/ > petsc-2.3.3-p8/lib/darwin8.11.1-cxx-debug > [0]PETSC ERROR: Configure run at Thu Jan 10 11:00:45 2008 > [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-mpi- > dir=/Users/gideon/software --with-mpi-shared=1 --with-x11 --download- > hypre=1 --with-hypre=1 --download-umfpack=1 --with-umfpack=1 --with- > shared=1 --with-dynamic=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > > > From balay at mcs.anl.gov Wed Mar 5 19:50:53 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 5 Mar 2008 19:50:53 -0600 (CST) Subject: out of memory error In-Reply-To: <37604ab40803051704g1dc69af8y5547658189cb3d18@mail.gmail.com> References: <37604ab40803051704g1dc69af8y5547658189cb3d18@mail.gmail.com> Message-ID: > Out of memory trying to allocate 112032952 bytes You have 2-3 GB memory limit [per proc] in 32bit mode. If you really need more than 3GB of memory for this app, you can move to a 64bit OS/machine with sufficient memory. [you can do 64bit compiles on OSX 10.4-ppc and perhaps 10.5-intel] Aron, 100 MB is the request for current malloc - that failed. [Previous mallocs might have already allocated close to 2-3GB memory] Satish On Wed, 5 Mar 2008, Aron Ahmadia wrote: > 112032952 bytes is about 100 MB. > > Are you really running out of memory or is something else going on? > > ~A > > On Wed, Mar 5, 2008 at 7:00 PM, Gideon Simpson wrote: > > I'm getting the following error with some code, running on a serial > > machine with, i think, 3 gigs of memory. Is there anyway to > > circumvent this error by being clever, or do I really need to go to a > > distributed memory machine? > > > > -gideon > > > > Out of memory trying to allocate 112032952 bytes > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly > > illegal memory access > > [0]PETSC ERROR: Try option -start_in_debugger or - > > on_error_attach_debugger > > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal > > [0]PETSC ERROR: or try http://valgrind.org on linux or man libgmalloc > > on Apple to find memory corruption errors > > [0]PETSC ERROR: likely location of problem given in stack below > > [0]PETSC ERROR: --------------------- Stack Frames > > ------------------------------------ > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > > available, > > [0]PETSC ERROR: INSTEAD the line number of the start of the > > function > > [0]PETSC ERROR: is given. > > [0]PETSC ERROR: [0] PCSetUp_HYPRE line 95 src/ksp/pc/impls/hypre/hypre.c > > [0]PETSC ERROR: [0] PCSetUp line 764 src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: [0] KSPSetUp line 183 src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: [0] KSPSolve line 305 src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: --------------------- Error Message > > ------------------------------------ > > [0]PETSC ERROR: Signal received! > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 8, Fri Nov 16 > > 17:03:40 CST 2007 HG revision: 414581156e67e55c761739b0deb119f7590d0f4b > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: ./bulk3d on a darwin8.1 named > > valkyrie.appmath.columbia.edu by gideon Wed Mar 5 15:53:26 2008 > > [0]PETSC ERROR: Libraries linked from /Users/gideon/software/ > > petsc-2.3.3-p8/lib/darwin8.11.1-cxx-debug > > [0]PETSC ERROR: Configure run at Thu Jan 10 11:00:45 2008 > > [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-mpi-dir=/ > > Users/gideon/software --with-mpi-shared=1 --with-x11 --download- > > hypre=1 --with-hypre=1 --download-umfpack=1 --with-umfpack=1 --with- > > shared=1 --with-dynamic=1 > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: User provided function() line 0 in unknown directory > > unknown file > > > > > > > > > > From li76pan at yahoo.com Thu Mar 6 01:36:04 2008 From: li76pan at yahoo.com (li pan) Date: Wed, 5 Mar 2008 23:36:04 -0800 (PST) Subject: out of memory error In-Reply-To: Message-ID: <935346.4764.qm@web36805.mail.mud.yahoo.com> hi Gideon, can I ask you how big is the nonzero allocation of your sparse matrix? pan --- Gideon Simpson wrote: > I'm getting the following error with some code, > running on a serial > machine with, i think, 3 gigs of memory. Is there > anyway to > circumvent this error by being clever, or do I > really need to go to a > distributed memory machine? > > -gideon > > Out of memory trying to allocate 112032952 bytes > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 10 BUS: Bus > Error, possibly > illegal memory access > [0]PETSC ERROR: Try option -start_in_debugger or - > on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal > > [0]PETSC ERROR: or try http://valgrind.org on linux > or man libgmalloc > on Apple to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in > stack below > [0]PETSC ERROR: --------------------- Stack Frames > > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the > stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the > start of the > function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PCSetUp_HYPRE line 95 > src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: [0] PCSetUp line 764 > src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPSetUp line 183 > src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] KSPSolve line 305 > src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message > > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch > 8, Fri Nov 16 > 17:03:40 CST 2007 HG revision: > 414581156e67e55c761739b0deb119f7590d0f4b > [0]PETSC ERROR: See docs/changes/index.html for > recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about > trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual > pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./bulk3d on a darwin8.1 named > valkyrie.appmath.columbia.edu by gideon Wed Mar 5 > 15:53:26 2008 > [0]PETSC ERROR: Libraries linked from > /Users/gideon/software/ > petsc-2.3.3-p8/lib/darwin8.11.1-cxx-debug > [0]PETSC ERROR: Configure run at Thu Jan 10 11:00:45 > 2008 > [0]PETSC ERROR: Configure options > --with-clanguage=cxx --with-mpi-dir=/ > Users/gideon/software --with-mpi-shared=1 --with-x11 > --download- > hypre=1 --with-hypre=1 --download-umfpack=1 > --with-umfpack=1 --with- > shared=1 --with-dynamic=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in > unknown directory > unknown file > > > > ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs From grs2103 at columbia.edu Thu Mar 6 13:47:35 2008 From: grs2103 at columbia.edu (Gideon Simpson) Date: Thu, 6 Mar 2008 14:47:35 -0500 Subject: out of memory error In-Reply-To: <935346.4764.qm@web36805.mail.mud.yahoo.com> References: <935346.4764.qm@web36805.mail.mud.yahoo.com> Message-ID: <6C105388-2EBA-4FFD-8338-B2570CC74E0C@columbia.edu> Matrix Object: type=seqaij, rows=218898, cols=218898 total: nonzeros=20389756, allocated nonzeros=20389756 not using I-node routines On Mar 6, 2008, at 2:36 AM, li pan wrote: > hi Gideon, > can I ask you how big is the nonzero allocation of > your sparse matrix? > > pan > > > --- Gideon Simpson wrote: > >> I'm getting the following error with some code, >> running on a serial >> machine with, i think, 3 gigs of memory. Is there >> anyway to >> circumvent this error by being clever, or do I >> really need to go to a >> distributed memory machine? >> >> -gideon >> >> Out of memory trying to allocate 112032952 bytes >> [0]PETSC ERROR: >> > ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 10 BUS: Bus >> Error, possibly >> illegal memory access >> [0]PETSC ERROR: Try option -start_in_debugger or - >> on_error_attach_debugger >> [0]PETSC ERROR: or see >> > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal >> >> [0]PETSC ERROR: or try http://valgrind.org on linux >> or man libgmalloc >> on Apple to find memory corruption errors >> [0]PETSC ERROR: likely location of problem given in >> stack below >> [0]PETSC ERROR: --------------------- Stack Frames >> >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the >> stack are not >> available, >> [0]PETSC ERROR: INSTEAD the line number of the >> start of the >> function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] PCSetUp_HYPRE line 95 >> src/ksp/pc/impls/hypre/hypre.c >> [0]PETSC ERROR: [0] PCSetUp line 764 >> src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: [0] KSPSetUp line 183 >> src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: [0] KSPSolve line 305 >> src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: --------------------- Error Message >> >> ------------------------------------ >> [0]PETSC ERROR: Signal received! >> [0]PETSC ERROR: >> > ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch >> 8, Fri Nov 16 >> 17:03:40 CST 2007 HG revision: >> 414581156e67e55c761739b0deb119f7590d0f4b >> [0]PETSC ERROR: See docs/changes/index.html for >> recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about >> trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual >> pages. >> [0]PETSC ERROR: >> > ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./bulk3d on a darwin8.1 named >> valkyrie.appmath.columbia.edu by gideon Wed Mar 5 >> 15:53:26 2008 >> [0]PETSC ERROR: Libraries linked from >> /Users/gideon/software/ >> petsc-2.3.3-p8/lib/darwin8.11.1-cxx-debug >> [0]PETSC ERROR: Configure run at Thu Jan 10 11:00:45 >> 2008 >> [0]PETSC ERROR: Configure options >> --with-clanguage=cxx --with-mpi-dir=/ >> Users/gideon/software --with-mpi-shared=1 --with-x11 >> --download- >> hypre=1 --with-hypre=1 --download-umfpack=1 >> --with-umfpack=1 --with- >> shared=1 --with-dynamic=1 >> [0]PETSC ERROR: >> > ------------------------------------------------------------------------ >> [0]PETSC ERROR: User provided function() line 0 in >> unknown directory >> unknown file >> >> >> >> > > > > > ____________________________________________________________________________________ > Never miss a thing. Make Yahoo your home page. > http://www.yahoo.com/r/hs > From li76pan at yahoo.com Fri Mar 7 04:10:38 2008 From: li76pan at yahoo.com (li pan) Date: Fri, 7 Mar 2008 02:10:38 -0800 (PST) Subject: out of memory error In-Reply-To: <6C105388-2EBA-4FFD-8338-B2570CC74E0C@columbia.edu> Message-ID: <182234.53881.qm@web36806.mail.mud.yahoo.com> hi Gideon, is the matrix big? I used to calculate one with more than 1 million rows on a PC equiped with 3 GB memory. pan --- Gideon Simpson wrote: > Matrix Object: > type=seqaij, rows=218898, cols=218898 > total: nonzeros=20389756, allocated > nonzeros=20389756 > not using I-node routines > > On Mar 6, 2008, at 2:36 AM, li pan wrote: > > > hi Gideon, > > can I ask you how big is the nonzero allocation of > > your sparse matrix? > > > > pan > > > > > > --- Gideon Simpson wrote: > > > >> I'm getting the following error with some code, > >> running on a serial > >> machine with, i think, 3 gigs of memory. Is > there > >> anyway to > >> circumvent this error by being clever, or do I > >> really need to go to a > >> distributed memory machine? > >> > >> -gideon > >> > >> Out of memory trying to allocate 112032952 bytes > >> [0]PETSC ERROR: > >> > > > ------------------------------------------------------------------------ > >> [0]PETSC ERROR: Caught signal number 10 BUS: Bus > >> Error, possibly > >> illegal memory access > >> [0]PETSC ERROR: Try option -start_in_debugger or > - > >> on_error_attach_debugger > >> [0]PETSC ERROR: or see > >> > > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal > >> > >> [0]PETSC ERROR: or try http://valgrind.org on > linux > >> or man libgmalloc > >> on Apple to find memory corruption errors > >> [0]PETSC ERROR: likely location of problem given > in > >> stack below > >> [0]PETSC ERROR: --------------------- Stack > Frames > >> > >> ------------------------------------ > >> [0]PETSC ERROR: Note: The EXACT line numbers in > the > >> stack are not > >> available, > >> [0]PETSC ERROR: INSTEAD the line number of > the > >> start of the > >> function > >> [0]PETSC ERROR: is given. > >> [0]PETSC ERROR: [0] PCSetUp_HYPRE line 95 > >> src/ksp/pc/impls/hypre/hypre.c > >> [0]PETSC ERROR: [0] PCSetUp line 764 > >> src/ksp/pc/interface/precon.c > >> [0]PETSC ERROR: [0] KSPSetUp line 183 > >> src/ksp/ksp/interface/itfunc.c > >> [0]PETSC ERROR: [0] KSPSolve line 305 > >> src/ksp/ksp/interface/itfunc.c > >> [0]PETSC ERROR: --------------------- Error > Message > >> > >> ------------------------------------ > >> [0]PETSC ERROR: Signal received! > >> [0]PETSC ERROR: > >> > > > ------------------------------------------------------------------------ > >> [0]PETSC ERROR: Petsc Release Version 2.3.3, > Patch > >> 8, Fri Nov 16 > >> 17:03:40 CST 2007 HG revision: > >> 414581156e67e55c761739b0deb119f7590d0f4b > >> [0]PETSC ERROR: See docs/changes/index.html for > >> recent updates. > >> [0]PETSC ERROR: See docs/faq.html for hints about > >> trouble shooting. > >> [0]PETSC ERROR: See docs/index.html for manual > >> pages. > >> [0]PETSC ERROR: > >> > > > ------------------------------------------------------------------------ > >> [0]PETSC ERROR: ./bulk3d on a darwin8.1 named > >> valkyrie.appmath.columbia.edu by gideon Wed Mar > 5 > >> 15:53:26 2008 > >> [0]PETSC ERROR: Libraries linked from > >> /Users/gideon/software/ > >> petsc-2.3.3-p8/lib/darwin8.11.1-cxx-debug > >> [0]PETSC ERROR: Configure run at Thu Jan 10 > 11:00:45 > >> 2008 > >> [0]PETSC ERROR: Configure options > >> --with-clanguage=cxx --with-mpi-dir=/ > >> Users/gideon/software --with-mpi-shared=1 > --with-x11 > >> --download- > >> hypre=1 --with-hypre=1 --download-umfpack=1 > >> --with-umfpack=1 --with- > >> shared=1 --with-dynamic=1 > >> [0]PETSC ERROR: > >> > > > ------------------------------------------------------------------------ > >> [0]PETSC ERROR: User provided function() line 0 > in > >> unknown directory > >> unknown file > >> > >> > >> > >> > > > > > > > > > > > ____________________________________________________________________________________ > > Never miss a thing. Make Yahoo your home page. > > http://www.yahoo.com/r/hs > > > > ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs From tribur at vision.ee.ethz.ch Fri Mar 7 05:50:44 2008 From: tribur at vision.ee.ethz.ch (Kathrin Burckhardt) Date: Fri, 7 Mar 2008 12:50:44 +0100 (CET) Subject: Schur complement system Message-ID: Dear nice people, Did you ever try to solve a Schur complement system using PETSc? From balay at mcs.anl.gov Fri Mar 7 08:26:48 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 7 Mar 2008 08:26:48 -0600 (CST) Subject: out of memory error In-Reply-To: <182234.53881.qm@web36806.mail.mud.yahoo.com> References: <182234.53881.qm@web36806.mail.mud.yahoo.com> Message-ID: This matrix should take approximately 20389756*(8+4)=244677072, i.e app 250MB ram. i.e the memory usage [upto 3GB] must be somewere else. One issue coud be memory leak [due to object not being destroyed properly]. This can be checked with the option -malloc_dump Satish On Fri, 7 Mar 2008, li pan wrote: > hi Gideon, > is the matrix big? I used to calculate one with more > than 1 million rows on a PC equiped with 3 GB memory. > > pan > > > --- Gideon Simpson wrote: > > > Matrix Object: > > type=seqaij, rows=218898, cols=218898 > > total: nonzeros=20389756, allocated > > nonzeros=20389756 > > not using I-node routines > > > > On Mar 6, 2008, at 2:36 AM, li pan wrote: > > > > > hi Gideon, > > > can I ask you how big is the nonzero allocation of > > > your sparse matrix? > > > > > > pan > > > > > > > > > --- Gideon Simpson wrote: > > > > > >> I'm getting the following error with some code, > > >> running on a serial > > >> machine with, i think, 3 gigs of memory. Is > > there > > >> anyway to > > >> circumvent this error by being clever, or do I > > >> really need to go to a > > >> distributed memory machine? > > >> > > >> -gideon > > >> > > >> Out of memory trying to allocate 112032952 bytes > > >> [0]PETSC ERROR: > > >> > > > > > > ------------------------------------------------------------------------ > > >> [0]PETSC ERROR: Caught signal number 10 BUS: Bus > > >> Error, possibly > > >> illegal memory access > > >> [0]PETSC ERROR: Try option -start_in_debugger or > > - > > >> on_error_attach_debugger > > >> [0]PETSC ERROR: or see > > >> > > > > > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal > > >> > > >> [0]PETSC ERROR: or try http://valgrind.org on > > linux > > >> or man libgmalloc > > >> on Apple to find memory corruption errors > > >> [0]PETSC ERROR: likely location of problem given > > in > > >> stack below > > >> [0]PETSC ERROR: --------------------- Stack > > Frames > > >> > > >> ------------------------------------ > > >> [0]PETSC ERROR: Note: The EXACT line numbers in > > the > > >> stack are not > > >> available, > > >> [0]PETSC ERROR: INSTEAD the line number of > > the > > >> start of the > > >> function > > >> [0]PETSC ERROR: is given. > > >> [0]PETSC ERROR: [0] PCSetUp_HYPRE line 95 > > >> src/ksp/pc/impls/hypre/hypre.c > > >> [0]PETSC ERROR: [0] PCSetUp line 764 > > >> src/ksp/pc/interface/precon.c > > >> [0]PETSC ERROR: [0] KSPSetUp line 183 > > >> src/ksp/ksp/interface/itfunc.c > > >> [0]PETSC ERROR: [0] KSPSolve line 305 > > >> src/ksp/ksp/interface/itfunc.c > > >> [0]PETSC ERROR: --------------------- Error > > Message > > >> > > >> ------------------------------------ > > >> [0]PETSC ERROR: Signal received! > > >> [0]PETSC ERROR: > > >> > > > > > > ------------------------------------------------------------------------ > > >> [0]PETSC ERROR: Petsc Release Version 2.3.3, > > Patch > > >> 8, Fri Nov 16 > > >> 17:03:40 CST 2007 HG revision: > > >> 414581156e67e55c761739b0deb119f7590d0f4b > > >> [0]PETSC ERROR: See docs/changes/index.html for > > >> recent updates. > > >> [0]PETSC ERROR: See docs/faq.html for hints about > > >> trouble shooting. > > >> [0]PETSC ERROR: See docs/index.html for manual > > >> pages. > > >> [0]PETSC ERROR: > > >> > > > > > > ------------------------------------------------------------------------ > > >> [0]PETSC ERROR: ./bulk3d on a darwin8.1 named > > >> valkyrie.appmath.columbia.edu by gideon Wed Mar > > 5 > > >> 15:53:26 2008 > > >> [0]PETSC ERROR: Libraries linked from > > >> /Users/gideon/software/ > > >> petsc-2.3.3-p8/lib/darwin8.11.1-cxx-debug > > >> [0]PETSC ERROR: Configure run at Thu Jan 10 > > 11:00:45 > > >> 2008 > > >> [0]PETSC ERROR: Configure options > > >> --with-clanguage=cxx --with-mpi-dir=/ > > >> Users/gideon/software --with-mpi-shared=1 > > --with-x11 > > >> --download- > > >> hypre=1 --with-hypre=1 --download-umfpack=1 > > >> --with-umfpack=1 --with- > > >> shared=1 --with-dynamic=1 > > >> [0]PETSC ERROR: > > >> > > > > > > ------------------------------------------------------------------------ > > >> [0]PETSC ERROR: User provided function() line 0 > > in > > >> unknown directory > > >> unknown file > > >> > > >> > > >> > > >> > > > > > > > > > > > > > > > > > > ____________________________________________________________________________________ > > > Never miss a thing. Make Yahoo your home page. > > > http://www.yahoo.com/r/hs > > > > > > > > > > > ____________________________________________________________________________________ > Never miss a thing. Make Yahoo your home page. > http://www.yahoo.com/r/hs > > From knepley at gmail.com Fri Mar 7 09:25:02 2008 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 7 Mar 2008 09:25:02 -0600 Subject: Schur complement system In-Reply-To: References: Message-ID: Yes, you can certainly solve this system. Normally, you would just create a simple MATSHELL to apply the Schur complement, which does two MatMult()s and one KSPSolve(). We are integrating specific support for this, but it will not be released until the summer I think. Thanks, Matt On Fri, Mar 7, 2008 at 5:50 AM, Kathrin Burckhardt wrote: > Dear nice people, > > Did you ever try to solve a Schur complement system using PETSc? > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From dalcinl at gmail.com Fri Mar 7 09:34:35 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 7 Mar 2008 12:34:35 -0300 Subject: Schur complement system In-Reply-To: References: Message-ID: Well, I did. I have a preliminar implementation of this inside petsc4py (PETSc for Python). However, it is fully implemented in C as a standard preconditioner intended to be used with KSPPREONLY. The preconditioner has a sub-KSP iterating only on global interface nodes. It should be almost trivial to incorporate it inside other codes. A warning: all this needs a MATIS matrix to work, that is each processor have is local portion of the global matrix in an unassembled fashion. In order to define the Schur complement system, the local process matrix is splitted in four submatrices. At each processor, the 'subdomain' can be further partitioned, this is implemented with a simple minded graph partitioning implemented by reusing the some core routines used inside PETSc for sparse matrix reordering. In order to precondition the Schur complement system, a subsidiary problem can defined by taking some layers of nodes around interface nodes and it is globally iterated with typically a few loops or sub-sub KSP. In short, the implementation is a really involved, and comparable in dificulty to the Newmann-Newman preconditioner (which is also implemented to work with MATIS). At the user level, the only parameter to tweack is the 'sub-subdomain' size and of course tolerances of the inner KSP solver inside the 'SCHUR' preconditioner. Why I never added this to PETSc? Time, lack of serious testing, etc. but mainly because I'm not sure of it is really a good choice compared to PCASM. In fact, some day I would implement my trics for sub-subdamain partitioning (or perhaps use metis for this) inside ASM. Then the sub-subdomain problems would have a reasonable size for using LU, and finally I would compare PCASM with my PCSCHUR. Currently, we are using this for solving incompressible NS equations with monolithic stabilized FEM formulation, and other problems with badly conditioned global systems. We have found no better way to solve our linear systems for our application, probably because we are not smart enough. Hope you understood me... On 3/7/08, Kathrin Burckhardt wrote: > Dear nice people, > > Did you ever try to solve a Schur complement system using PETSc? > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From grs2103 at columbia.edu Fri Mar 7 12:21:55 2008 From: grs2103 at columbia.edu (Gideon Simpson) Date: Fri, 7 Mar 2008 13:21:55 -0500 Subject: .info files Message-ID: <8BB9C195-219C-4A11-A3A6-487B3259B502@columbia.edu> When I dump some matrices and vectors to binary, I seem to be generating some empty files with .info extensions. What's the deal with tehse? -gideon From bsmith at mcs.anl.gov Fri Mar 7 12:33:20 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 7 Mar 2008 12:33:20 -0600 Subject: .info files In-Reply-To: <8BB9C195-219C-4A11-A3A6-487B3259B502@columbia.edu> References: <8BB9C195-219C-4A11-A3A6-487B3259B502@columbia.edu> Message-ID: <0943ADD9-97D0-40CD-A8AC-BCF5B8F2A6F2@mcs.anl.gov> PETSc binary viewers put some additional information into .info files like matrix block size; it is harmless but if you really don't like it you can use -viewer_binary_skip_info or PetscViewerBinarySkipInfo() note you need to call PetscViewerBinarySkipInfo() before PetscViewerFileSetName(). In other words you cannot use PetscViewerBinaryOpen() directly. Barry On Mar 7, 2008, at 12:21 PM, Gideon Simpson wrote: > When I dump some matrices and vectors to binary, I seem to be > generating some empty files with .info extensions. What's the deal > with tehse? > > -gideon > From grs2103 at columbia.edu Fri Mar 7 18:02:57 2008 From: grs2103 at columbia.edu (Gideon Simpson) Date: Fri, 7 Mar 2008 19:02:57 -0500 Subject: reading matrices in the smart way Message-ID: Suppose I've dumped a seqaij matrix to binary, and I now want to read it in and do something with it, perhaps in a distributed manner. When I execute MatLoad, I need to give it a matrix type, and I am wondering what the smart thing is to do. I'd like it to work both when I am testing it serially and running it on a cluster. Assuming I still want AIJ format, should I set it to MATAIJ? Can I feed it a command line option to override whatever I hardcode it to? i.e., if I code MatLoad(viewer, MATSEQAIJ, &A); but run the executable ./ex -mattload_type mpaij will it run as though it were with MATMPIAIJ? -gideon From knepley at gmail.com Fri Mar 7 18:09:47 2008 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 7 Mar 2008 18:09:47 -0600 Subject: reading matrices in the smart way In-Reply-To: References: Message-ID: On Fri, Mar 7, 2008 at 6:02 PM, Gideon Simpson wrote: > Suppose I've dumped a seqaij matrix to binary, and I now want to read > it in and do something with it, perhaps in a distributed manner. When > I execute MatLoad, I need to give it a matrix type, and I am wondering > what the smart thing is to do. I'd like it to work both when I am > testing it serially and running it on a cluster. Assuming I still > want AIJ format, should I set it to MATAIJ? Can I feed it a command Yes, this will just do the right thing. > line option to override whatever I hardcode it to? > > i.e., if I code > > MatLoad(viewer, MATSEQAIJ, &A); > > but run the executable > ./ex -mattload_type mpaij > > will it run as though it were with MATMPIAIJ? Yes. Matt > -gideon > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From GRS2103 at columbia.edu Sat Mar 8 00:00:40 2008 From: GRS2103 at columbia.edu (Gideon Simpson) Date: Sat, 8 Mar 2008 01:00:40 -0500 Subject: PetscViewerBinaryOpen getting stuck Message-ID: <4E2720C8-23CC-4E3B-B967-393FF025F42E@columbia.edu> I have some code that wants to read in a previously binary dumped and gzipped matrix. The first time I run this code, it opens the file with PetscViewerBinaryOpen, loads the data into the matrix and processes as expected. However, if, for some reason, I want to execute it a second time, when it tries to open the file, it gets stuck. Investigating a bit, it seems to be that the first time it runs, when it gunzips the file, it leaves the file in /tmp. If that file is still in this folder, it gets stuck. If I remove it from / tmp, the code executes without a problem, as it did the first time. Obviously, I could avoid using zipped files. Is this a bug (or a feature)? Is there an obvious way to clean up /tmp after the code executes? -gideon From knepley at gmail.com Sat Mar 8 09:31:25 2008 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 8 Mar 2008 09:31:25 -0600 Subject: PetscViewerBinaryOpen getting stuck In-Reply-To: <4E2720C8-23CC-4E3B-B967-393FF025F42E@columbia.edu> References: <4E2720C8-23CC-4E3B-B967-393FF025F42E@columbia.edu> Message-ID: On Sat, Mar 8, 2008 at 12:00 AM, Gideon Simpson wrote: > I have some code that wants to read in a previously binary dumped and > gzipped matrix. The first time I run this code, it opens the file > with PetscViewerBinaryOpen, loads the data into the matrix and > processes as expected. However, if, for some reason, I want to > execute it a second time, when it tries to open the file, it gets > stuck. Investigating a bit, it seems to be that the first time it 1) Is this serial or parallel? 2) What do you mean by "gets stuck"? 3) Is there a small example which illustrates this? I am looking at the code, and do not see an obvious error. Thanks, Matt > runs, when it gunzips the file, it leaves the file in /tmp. If that > file is still in this folder, it gets stuck. If I remove it from / > tmp, the code executes without a problem, as it did the first time. > > Obviously, I could avoid using zipped files. Is this a bug (or a > feature)? Is there an obvious way to clean up /tmp after the code > executes? > > -gideon > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From bsmith at mcs.anl.gov Sat Mar 8 11:15:28 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 8 Mar 2008 11:15:28 -0600 Subject: PetscViewerBinaryOpen getting stuck In-Reply-To: <4E2720C8-23CC-4E3B-B967-393FF025F42E@columbia.edu> References: <4E2720C8-23CC-4E3B-B967-393FF025F42E@columbia.edu> Message-ID: <1A703128-E537-4570-8F21-BD249E6A9488@mcs.anl.gov> Is your original file in the local filesystem or does it start with a url like http: or ftp: Barry On Mar 8, 2008, at 12:00 AM, Gideon Simpson wrote: > I have some code that wants to read in a previously binary dumped > and gzipped matrix. The first time I run this code, it opens the > file with PetscViewerBinaryOpen, loads the data into the matrix and > processes as expected. However, if, for some reason, I want to > execute it a second time, when it tries to open the file, it gets > stuck. Investigating a bit, it seems to be that the first time it > runs, when it gunzips the file, it leaves the file in /tmp. If that > file is still in this folder, it gets stuck. If I remove it from / > tmp, the code executes without a problem, as it did the first time. > > Obviously, I could avoid using zipped files. Is this a bug (or a > feature)? Is there an obvious way to clean up /tmp after the code > executes? > > -gideon > From grs2103 at columbia.edu Sat Mar 8 11:19:30 2008 From: grs2103 at columbia.edu (Gideon Simpson) Date: Sat, 8 Mar 2008 12:19:30 -0500 Subject: PetscViewerBinaryOpen getting stuck In-Reply-To: References: <4E2720C8-23CC-4E3B-B967-393FF025F42E@columbia.edu> Message-ID: <1D4C9549-950F-4FE2-B234-E73C38C8C06F@columbia.edu> 1. Certainly a problem in serial. Also a problem on a shared memory machine with MPI. Have not tested on a distributed memory machine 2. When loading the matrix with MatLoad, you can see in top that it starts gunzip, and then gunzip idles,using no memory or cpu. 3. Here's an update. This is NOT a petsc problem, or at least not obviously a petsc problem. I am using petsc in conjunction with fenics software. I decided that I wanted to do problems that would need to be run on distributed memory machines, so I decoupled the three processes. I run dolfin code that generates the matrices and vectors and saves them as petsc binaries. Then I have pure petsc code that solves the problem. Finally, there is another dolfin program that interprets the output in terms of my variables. I discovered that if I do not run this third segment, I can run the petsc solver piece as many times as I want, and there is no idling of gunzip. Maybe dolfin has a memory leak or something. 4. Also, if I choose not to zip my files, and just dump the uncompressed binary data, the problem also disappears. Related question: when saving compressed binaries, if there is already something in the directory with the filename, gzip prompts to overwrite. Is there a way to circumvent this need for user intervention? On Mar 8, 2008, at 10:31 AM, Matthew Knepley wrote: > On Sat, Mar 8, 2008 at 12:00 AM, Gideon Simpson > wrote: >> I have some code that wants to read in a previously binary dumped and >> gzipped matrix. The first time I run this code, it opens the file >> with PetscViewerBinaryOpen, loads the data into the matrix and >> processes as expected. However, if, for some reason, I want to >> execute it a second time, when it tries to open the file, it gets >> stuck. Investigating a bit, it seems to be that the first time it > > 1) Is this serial or parallel? > > 2) What do you mean by "gets stuck"? > > 3) Is there a small example which illustrates this? > > I am looking at the code, and do not see an obvious error. > > Thanks, > > Matt > >> runs, when it gunzips the file, it leaves the file in /tmp. If that >> file is still in this folder, it gets stuck. If I remove it from / >> tmp, the code executes without a problem, as it did the first time. >> >> Obviously, I could avoid using zipped files. Is this a bug (or a >> feature)? Is there an obvious way to clean up /tmp after the code >> executes? >> >> -gideon >> >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > From bsmith at mcs.anl.gov Sat Mar 8 13:30:25 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 8 Mar 2008 13:30:25 -0600 Subject: PetscViewerBinaryOpen getting stuck In-Reply-To: <1D4C9549-950F-4FE2-B234-E73C38C8C06F@columbia.edu> References: <4E2720C8-23CC-4E3B-B967-393FF025F42E@columbia.edu> <1D4C9549-950F-4FE2-B234-E73C38C8C06F@columbia.edu> Message-ID: <965BEF1E-9B06-4572-B181-08A8E776FA49@mcs.anl.gov> On Mar 8, 2008, at 11:19 AM, Gideon Simpson wrote: > 1. Certainly a problem in serial. Also a problem on a shared > memory machine with MPI. Have not tested on a distributed memory > machine > > 2. When loading the matrix with MatLoad, you can see in top that it > starts gunzip, and then gunzip idles,using no memory or cpu. > Sounds like gunzip is waiting for input. Edit bin/urlget and locate the gzip like that looks like gunzip -f $file > /dev/null 2>&1 replace it with gunzip -f $file Now maybe it will print a message when it "hangs" and you can see what it wants. The -f is suppose to force it to run without asking the user anything but > 3. Here's an update. This is NOT a petsc problem, or at least not > obviously a petsc problem. I am using petsc in conjunction with > fenics software. I decided that I wanted to do problems that would > need to be run on distributed memory machines, so I decoupled the > three processes. I run dolfin code that generates the matrices and > vectors and saves them as petsc binaries. Then I have pure petsc > code that solves the problem. Finally, there is another dolfin > program that interprets the output in terms of my variables. I > discovered that if I do not run this third segment, I can run the > petsc solver piece as many times as I want, and there is no idling > of gunzip. Maybe dolfin has a memory leak or something. > > 4. Also, if I choose not to zip my files, and just dump the > uncompressed binary data, the problem also disappears. > > Related question: when saving compressed binaries, if there is > already something in the directory with the filename, gzip prompts > to overwrite. Is there a way to circumvent this need for user > intervention? I have added the -f option to gzip in src/sys/viewer/impls/binary/ binv.c so it will force the overwrite without request. You can hg pull and then run make in that one directory for this update. > > > On Mar 8, 2008, at 10:31 AM, Matthew Knepley wrote: > >> On Sat, Mar 8, 2008 at 12:00 AM, Gideon Simpson >> wrote: >>> I have some code that wants to read in a previously binary dumped >>> and >>> gzipped matrix. The first time I run this code, it opens the file >>> with PetscViewerBinaryOpen, loads the data into the matrix and >>> processes as expected. However, if, for some reason, I want to >>> execute it a second time, when it tries to open the file, it gets >>> stuck. Investigating a bit, it seems to be that the first time it >> >> 1) Is this serial or parallel? >> >> 2) What do you mean by "gets stuck"? >> >> 3) Is there a small example which illustrates this? >> >> I am looking at the code, and do not see an obvious error. >> >> Thanks, >> >> Matt >> >>> runs, when it gunzips the file, it leaves the file in /tmp. If that >>> file is still in this folder, it gets stuck. If I remove it from / >>> tmp, the code executes without a problem, as it did the first time. >>> >>> Obviously, I could avoid using zipped files. Is this a bug (or a >>> feature)? Is there an obvious way to clean up /tmp after the code >>> executes? >>> >>> -gideon >>> >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener >> > From aja2111 at columbia.edu Sat Mar 8 16:01:17 2008 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Sat, 8 Mar 2008 17:01:17 -0500 Subject: Checking my assumptions on the use of DMMG in SNES ex19.c Message-ID: <37604ab40803081401i4afb7581hd7ff46634fa03c1e@mail.gmail.com> Hi all, I'm preparing a multigrid lecture for Tuesday that will be motivated by a demonstration of PETSc using Multigrid to solve the thermally and lid-driven cavity flow problem using the DMMG solver framework. Since only about 20-30 minutes of time will be spent to describing this, I want to make sure my descriptions are both succinct and accurate. I'd like to describe how DMMG allows a user to rapidly solve (some) non-linear problems using multigrid if she's willing to use finite differencing to automatically generate her Jacobian matrix at each stage. Is the following statement correct? If not, how would you fix it, particularly in terms of describing the techniques used in ex19.c "To use DMMG to solve a given non-linear PDE using multigrid and finite-differencing to automatically generate the Jacobian operator, the user is responsible for specifying the coarsest grid that the problem will be solved on as well as the number of levels of multigrid which will be applied, with the resolution doubling for each level of multigrid. For example, to solve a non-linear problem on a 2-D grid with a fine-grid resolution of 256x256 and 3 levels of multigrid, the user creates a DMMG object with 3 levels, then calls DMMGSetDM with a DA object of resolution 64x64. The user is also responsible for providing SNESLocal, a programmatic function that evalutes the mathematical function F(u) locally using field values from the stencil specified in the DMMG creation routine. When DMMGSolve is called, PETSc will use its default non-linear solver to attempt to solve F(u) = 0. " ~A From bsmith at mcs.anl.gov Sat Mar 8 19:10:43 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 8 Mar 2008 19:10:43 -0600 Subject: Checking my assumptions on the use of DMMG in SNES ex19.c In-Reply-To: <37604ab40803081401i4afb7581hd7ff46634fa03c1e@mail.gmail.com> References: <37604ab40803081401i4afb7581hd7ff46634fa03c1e@mail.gmail.com> Message-ID: Aron, This is 100% correct. To prevent confusion I would suggest the final sentence be: When DMMGSolve is called, PETSc will use its default non-linear solver with its default geometric multigrid solver for the linear system to attempt to solve F(u) = 0. Barry On Mar 8, 2008, at 4:01 PM, Aron Ahmadia wrote: > Hi all, > > I'm preparing a multigrid lecture for Tuesday that will be motivated > by a demonstration of PETSc using Multigrid to solve the thermally and > lid-driven cavity flow problem using the DMMG solver framework. > > Since only about 20-30 minutes of time will be spent to describing > this, I want to make sure my descriptions are both succinct and > accurate. > > I'd like to describe how DMMG allows a user to rapidly solve (some) > non-linear problems using multigrid if she's willing to use finite > differencing to automatically generate her Jacobian matrix at each > stage. > > Is the following statement correct? If not, how would you fix it, > particularly in terms of describing the techniques used in ex19.c > > "To use DMMG to solve a given non-linear PDE using multigrid and > finite-differencing to automatically generate the Jacobian operator, > the user is responsible for specifying the coarsest grid that the > problem will be solved on as well as the number of levels of multigrid > which will be applied, with the resolution doubling for each level of > multigrid. > > For example, to solve a non-linear problem on a 2-D grid with a > fine-grid resolution of 256x256 and 3 levels of multigrid, the user > creates a DMMG object with 3 levels, then calls DMMGSetDM with a DA > object of resolution 64x64. > > The user is also responsible for providing SNESLocal, a programmatic > function that evalutes the mathematical function F(u) locally using > field values from the stencil specified in the DMMG creation routine. > When DMMGSolve is called, PETSc will use its default non-linear solver > to attempt to solve F(u) = 0. > " > ~A > From tribur at vision.ee.ethz.ch Sun Mar 9 16:25:59 2008 From: tribur at vision.ee.ethz.ch (Kathrin Burckhardt) Date: Sun, 9 Mar 2008 22:25:59 +0100 (CET) Subject: Schur complement system In-Reply-To: References: Message-ID: Hi Matt, 1. Where do I integrate the application of the Schur system using MATSHELL? Can I use KSP? 2. I already have the local Schur complements computed. Is it advisable to once setup the global Schur matrix using MATIS and then use KSP, instead of the application of the Schur complement, i.e. the two MatMult() and the KSPSolve(), at each iteration? Thank you, Kathrin > Yes, you can certainly solve this system. Normally, you would just create a > simple MATSHELL to apply the Schur complement, which does two MatMult()s > and one KSPSolve(). We are integrating specific support for this, but it will > not be released until the summer I think. > > Thanks, > > Matt > > On Fri, Mar 7, 2008 at 5:50 AM, Kathrin Burckhardt > wrote: >> Dear nice people, >> >> Did you ever try to solve a Schur complement system using PETSc? >> >> > > > > From tribur at vision.ee.ethz.ch Sun Mar 9 16:39:12 2008 From: tribur at vision.ee.ethz.ch (Kathrin Burckhardt) Date: Sun, 9 Mar 2008 22:39:12 +0100 (CET) Subject: Schur complement system In-Reply-To: References: Message-ID: Dear Lisandro, uff, sounds in fact quite complicated. In particular, I don't understand why you split the "local process matrix" (the local Schur complement, right?) in four (sub?)submatrices? Is it because of the structure of you domain? On Fri, 7 Mar 2008, Lisandro Dalcin wrote: > Well, I did. I have a preliminar implementation of this inside > petsc4py (PETSc for Python). However, it is fully implemented in C as > a standard preconditioner intended to be used with KSPPREONLY. The > preconditioner has a sub-KSP iterating only on global interface > nodes. It should be almost trivial to incorporate it inside other > codes. > > A warning: all this needs a MATIS matrix to work, that is each > processor have is local portion of the global matrix in an unassembled > fashion. In order to define the Schur complement system, the local > process matrix is splitted in four submatrices. At each processor, the > 'subdomain' can be further partitioned, this is implemented with a > simple minded graph partitioning implemented by reusing the some core > routines used inside PETSc for sparse matrix reordering. In order to > precondition the Schur complement system, a subsidiary problem can > defined by taking some layers of nodes around interface nodes and it > is globally iterated with typically a few loops or sub-sub KSP. > > In short, the implementation is a really involved, and comparable in > dificulty to the Newmann-Newman preconditioner (which is also > implemented to work with MATIS). At the user level, the only parameter > to tweack is the 'sub-subdomain' size and of course tolerances of the > inner KSP solver inside the 'SCHUR' preconditioner. > > Why I never added this to PETSc? Time, lack of serious testing, etc. > but mainly because I'm not sure of it is really a good choice compared > to PCASM. In fact, some day I would implement my trics for > sub-subdamain partitioning (or perhaps use metis for this) inside ASM. > Then the sub-subdomain problems would have a reasonable size for using > LU, and finally I would compare PCASM with my PCSCHUR. > > Currently, we are using this for solving incompressible NS equations > with monolithic stabilized FEM formulation, and other problems with > badly conditioned global systems. We have found no better way to solve > our linear systems for our application, probably because we are not > smart enough. > > Hope you understood me... > > On 3/7/08, Kathrin Burckhardt wrote: >> Dear nice people, >> >> Did you ever try to solve a Schur complement system using PETSc? >> >> > > > From knepley at gmail.com Sun Mar 9 17:47:59 2008 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 9 Mar 2008 16:47:59 -0600 Subject: Schur complement system In-Reply-To: References: Message-ID: On Sun, Mar 9, 2008 at 3:25 PM, Kathrin Burckhardt wrote: > Hi Matt, > > 1. Where do I integrate the application of the Schur system using > MATSHELL? Can I use KSP? Here is a sketch: MatShellSetLOperation(S, MATOP_MULT, myApply); PetscErrorCode myApply(Mat S, Vec x, Vec y) { MatMult(B, x, tmp1); KSPSetOperators(ksp, A, A); KSPSolve(ksp, tmp1, tmp2); MatMultTranspose(B, tmp2, y); } > 2. I already have the local Schur complements computed. Is it advisable > to once setup the global Schur matrix using MATIS and then use KSP, > instead of the application of the Schur complement, i.e. the two > MatMult() and the KSPSolve(), at each iteration? It depends on how dense the Schur complement is. Matt > Thank you, > Kathrin > > > > Yes, you can certainly solve this system. Normally, you would just create a > > simple MATSHELL to apply the Schur complement, which does two MatMult()s > > and one KSPSolve(). We are integrating specific support for this, but it will > > not be released until the summer I think. > > > > Thanks, > > > > Matt > > > > On Fri, Mar 7, 2008 at 5:50 AM, Kathrin Burckhardt > > wrote: > >> Dear nice people, > >> > >> Did you ever try to solve a Schur complement system using PETSc? > >> > >> > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From rafaelsantoscoelho at gmail.com Mon Mar 10 07:24:08 2008 From: rafaelsantoscoelho at gmail.com (Rafael Santos Coelho) Date: Mon, 10 Mar 2008 09:24:08 -0300 Subject: Problem with SNESSolve() Message-ID: <3b6f83d40803100524g642ea717g6eb7bf4880bfc154@mail.gmail.com> Hello, guys! (: I've been trying to code a program that solves a simple 3D diffusion-convection problem, but there's something weird going on. Whenever I run it, I get the following error message: $ mpirun -np 1 ./ex8 -x 8 -y 8 -z 8 [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Invalid argument! [0]PETSC ERROR: Wrong type of object: Parameter # 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: VecMaxPointwiseDivide() line 35 in src/vec/vec/interface/rvector.c [0]PETSC ERROR: SNESLineSearchCubic() line 544 in src/snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve_LS() line 211 in src/snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve() line 1871 in src/snes/interface/snes.c [0]PETSC ERROR: main() line 135 in src/snes/examples/tutorials/ex8.c Well, line 135 is: ierr = SNESSolve(snes, PETSC_NULL, x); CHKERRQ(ierr); Here's the main function: (I removed several comments of mine to make it smaller) typedef struct { DA da; } appContext; PetscScalar Hx, Hy, Hz, Hx2, Hy2, Hz2; PetscInt M = 9, N = 9, P = 9; #undef __FUNCT__ #define __FUNCT__ "main" int main(int argc, char **argv) { SNES snes; Mat J; appContext ctx; Vec x, r; PetscErrorCode ierr; PetscScalar hx, hy, hz; PetscInitialize(&argc, &argv, (char*) 0, help); ierr = PetscOptionsGetInt(PETSC_NULL, "-y", &M, PETSC_NULL); CHKERRQ(ierr); ierr = PetscOptionsGetInt(PETSC_NULL, "-x", &N, PETSC_NULL); CHKERRQ(ierr); ierr = PetscOptionsGetInt(PETSC_NULL, "-z", &P, PETSC_NULL); CHKERRQ(ierr); hx = 1.0 / (N+1); hy = 1.0 / (M+1); hz = 1.0 / (P+1); // global variables Hx2 = 1.0 / (2.0*hx*hx); Hy2 = 1.0 / (2.0*hy*hy); Hz2 = 1.0 / (2.0*hz*hz ); Hx = 1.0 / (hx*hx); Hy = 1.0 / (hy*hy); Hz = 1.0 / (hz*hz); ierr = SNESCreate(PETSC_COMM_WORLD, &snes); CHKERRQ(ierr); ierr = DACreate3d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_STAR, M, N, P, PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, 1, 1, PETSC_NULL, PETSC_NULL, PETSC_NULL, &ctx.da); ierr = DACreateGlobalVector(ctx.da, &x); CHKERRQ(ierr); ierr = VecDuplicate(x, &r); CHKERRQ(ierr); ierr = DAGetMatrix(ctx.da, MATAIJ, &J); CHKERRQ(ierr); ierr = SNESSetFunction(snes, r, computeFVector, &ctx); CHKERRQ(ierr); ierr = SNESSetJacobian(snes, J, J, computeJacobian, &ctx); CHKERRQ(ierr); ierr = SNESSetFromOptions(snes); CHKERRQ(ierr); ierr = formInitialGuess(x); CHKERRQ(ierr); ierr = SNESSolve(snes, PETSC_NULL, x); CHKERRQ(ierr); ierr = VecDestroy(x); CHKERRQ(ierr); ierr = VecDestroy(r); CHKERRQ(ierr); ierr = MatDestroy(J); CHKERRQ(ierr); ierr = SNESDestroy(snes); CHKERRQ(ierr); ierr = DADestroy(ctx.da); CHKERRQ(ierr); ierr = PetscFinalize(); CHKERRQ(ierr); PetscFunctionReturn(0); } What am I missing here? Regards, Rafael Santos Coelho -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 10 07:39:54 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 10 Mar 2008 06:39:54 -0600 Subject: Problem with SNESSolve() In-Reply-To: <3b6f83d40803100524g642ea717g6eb7bf4880bfc154@mail.gmail.com> References: <3b6f83d40803100524g642ea717g6eb7bf4880bfc154@mail.gmail.com> Message-ID: The error says the the object 'x', the solution vector, has an incorrect header in the call. Since I see you have created the vector, the most likely explanation is overwriting memory in computeFunction or computeJacobian, however these functions are not in the mail. Also, I would suggest that you move to the local fnuction paradigm if you are using DAs. See SNES ex5 for an example. This is much easier to code and results in fewer memory overwrites. Thanks, Matt On Mon, Mar 10, 2008 at 6:24 AM, Rafael Santos Coelho wrote: > Hello, guys! (: > > I've been trying to code a program that solves a simple 3D > diffusion-convection problem, but there's something weird going on. Whenever > I run it, I get the following error message: > > $ mpirun -np 1 ./ex8 -x 8 -y 8 -z 8 > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Invalid argument! > [0]PETSC ERROR: Wrong type of object: Parameter # 2! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: VecMaxPointwiseDivide() line 35 in > src/vec/vec/interface/rvector.c > [0]PETSC ERROR: SNESLineSearchCubic() line 544 in src/snes/impls/ls/ls.c > [0]PETSC ERROR: SNESSolve_LS() line 211 in src/snes/impls/ls/ls.c > [0]PETSC ERROR: SNESSolve() line 1871 in src/snes/interface/snes.c > [0]PETSC ERROR: main() line 135 in src/snes/examples/tutorials/ex8.c > > Well, line 135 is: > ierr = SNESSolve(snes, PETSC_NULL, x); CHKERRQ(ierr); > > Here's the main function: (I removed several comments of mine to make it > smaller) > > typedef struct { > DA da; > } appContext; > > PetscScalar Hx, Hy, Hz, Hx2, Hy2, Hz2; > PetscInt M = 9, N = 9, P = 9; > > #undef __FUNCT__ > #define __FUNCT__ "main" > > int main(int argc, char **argv) { > SNES snes; > Mat J; > appContext ctx; > Vec x, r; > PetscErrorCode ierr; > PetscScalar hx, hy, hz; > > PetscInitialize(&argc, &argv, (char*) 0, help); > > ierr = PetscOptionsGetInt(PETSC_NULL, "-y", &M, PETSC_NULL); > CHKERRQ(ierr); > ierr = PetscOptionsGetInt(PETSC_NULL, "-x", &N, PETSC_NULL); > CHKERRQ(ierr); > ierr = PetscOptionsGetInt(PETSC_NULL, "-z", &P, PETSC_NULL); > CHKERRQ(ierr); > > hx = 1.0 / (N+1); hy = 1.0 / (M+1); hz = 1.0 / (P+1); > > // global variables > Hx2 = 1.0 / (2.0*hx*hx); Hy2 = 1.0 / (2.0*hy*hy); Hz2 = 1.0 / > (2.0*hz*hz); > Hx = 1.0 / (hx*hx); Hy = 1.0 / (hy*hy); Hz = 1.0 / (hz*hz); > > ierr = SNESCreate(PETSC_COMM_WORLD, &snes); CHKERRQ(ierr); > > ierr = DACreate3d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_STAR, M, > N, P, PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, 1, 1, PETSC_NULL, > PETSC_NULL, PETSC_NULL, &ctx.da); > > ierr = DACreateGlobalVector(ctx.da, &x); CHKERRQ(ierr); > ierr = VecDuplicate(x, &r); CHKERRQ(ierr); > > ierr = DAGetMatrix(ctx.da, MATAIJ, &J); CHKERRQ(ierr); > > ierr = SNESSetFunction(snes, r, computeFVector, &ctx); CHKERRQ(ierr); > ierr = SNESSetJacobian(snes, J, J, computeJacobian, &ctx); CHKERRQ(ierr); > > ierr = SNESSetFromOptions(snes); CHKERRQ(ierr); > > ierr = formInitialGuess(x); CHKERRQ(ierr); > ierr = SNESSolve(snes, PETSC_NULL, x); CHKERRQ(ierr); > > ierr = VecDestroy(x); CHKERRQ(ierr); > ierr = VecDestroy(r); CHKERRQ(ierr); > ierr = MatDestroy(J); CHKERRQ(ierr); > ierr = SNESDestroy(snes); CHKERRQ(ierr); > ierr = DADestroy(ctx.da); CHKERRQ(ierr); > ierr = PetscFinalize(); CHKERRQ(ierr); > > PetscFunctionReturn(0); > } > > > What am I missing here? > > Regards, > > Rafael Santos Coelho > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From dalcinl at gmail.com Mon Mar 10 09:00:40 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 10 Mar 2008 11:00:40 -0300 Subject: Schur complement system In-Reply-To: References: Message-ID: On 3/9/08, Kathrin Burckhardt wrote: > uff, sounds in fact quite complicated. Yes, it is :-( > In particular, I don't > understand why you split the "local process matrix" (the local Schur > complement, right?) in four (sub?)submatrices? Is it because of the > structure of you domain? Well, perhaps I did not understood your. My implementation is intended to work with a MATIS global matrix. In this matrix, each process assembles its local subdomain matrix (this is somewhat natural in finite element methods). For a finite element partitioning (as described in Y. Saad book on iterative methods, http://www-users.cs.umn.edu/~saad/books.html), the 'local' Schur complement is defined by four submatrices obtained from the local subdomain matrix. Those 'local' Schur complements in turn define the 'global' one, which is never explicitely assembled, but implicitely handled by defining the matrix-vector product. > > > > On Fri, 7 Mar 2008, Lisandro Dalcin wrote: > > > Well, I did. I have a preliminar implementation of this inside > > petsc4py (PETSc for Python). However, it is fully implemented in C as > > a standard preconditioner intended to be used with KSPPREONLY. The > > preconditioner has a sub-KSP iterating only on global interface > > nodes. It should be almost trivial to incorporate it inside other > > codes. > > > > A warning: all this needs a MATIS matrix to work, that is each > > processor have is local portion of the global matrix in an unassembled > > fashion. In order to define the Schur complement system, the local > > process matrix is splitted in four submatrices. At each processor, the > > 'subdomain' can be further partitioned, this is implemented with a > > simple minded graph partitioning implemented by reusing the some core > > routines used inside PETSc for sparse matrix reordering. In order to > > precondition the Schur complement system, a subsidiary problem can > > defined by taking some layers of nodes around interface nodes and it > > is globally iterated with typically a few loops or sub-sub KSP. > > > > In short, the implementation is a really involved, and comparable in > > dificulty to the Newmann-Newman preconditioner (which is also > > implemented to work with MATIS). At the user level, the only parameter > > to tweack is the 'sub-subdomain' size and of course tolerances of the > > inner KSP solver inside the 'SCHUR' preconditioner. > > > > Why I never added this to PETSc? Time, lack of serious testing, etc. > > but mainly because I'm not sure of it is really a good choice compared > > to PCASM. In fact, some day I would implement my trics for > > sub-subdamain partitioning (or perhaps use metis for this) inside ASM. > > Then the sub-subdomain problems would have a reasonable size for using > > LU, and finally I would compare PCASM with my PCSCHUR. > > > > Currently, we are using this for solving incompressible NS equations > > with monolithic stabilized FEM formulation, and other problems with > > badly conditioned global systems. We have found no better way to solve > > our linear systems for our application, probably because we are not > > smart enough. > > > > Hope you understood me... > > > > On 3/7/08, Kathrin Burckhardt wrote: > >> Dear nice people, > >> > >> Did you ever try to solve a Schur complement system using PETSc? > >> > >> > > > > > > > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From knepley at gmail.com Mon Mar 10 10:53:55 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 10 Mar 2008 09:53:55 -0600 Subject: Schur complement system In-Reply-To: References: Message-ID: On Mon, Mar 10, 2008 at 8:00 AM, Lisandro Dalcin wrote: > On 3/9/08, Kathrin Burckhardt wrote: > > uff, sounds in fact quite complicated. > > Yes, it is :-( > > > In particular, I don't > > understand why you split the "local process matrix" (the local Schur > > complement, right?) in four (sub?)submatrices? Is it because of the > > structure of you domain? > > Well, perhaps I did not understood your. > > My implementation is intended to work with a MATIS global matrix. In > this matrix, each process assembles its local subdomain matrix (this > is somewhat natural in finite element methods). For a finite element > partitioning (as described in Y. Saad book on iterative methods, > http://www-users.cs.umn.edu/~saad/books.html), the 'local' Schur > complement is defined by four submatrices obtained from the local > subdomain matrix. Those 'local' Schur complements in turn define the > 'global' one, which is never explicitely assembled, but implicitely > handled by defining the matrix-vector product. By 4 matrices, I assume you mean A, B, C where S = B^T A^{-1} B + C right? Thanks, Matt > > > > > > > > On Fri, 7 Mar 2008, Lisandro Dalcin wrote: > > > > > Well, I did. I have a preliminar implementation of this inside > > > petsc4py (PETSc for Python). However, it is fully implemented in C as > > > a standard preconditioner intended to be used with KSPPREONLY. The > > > preconditioner has a sub-KSP iterating only on global interface > > > nodes. It should be almost trivial to incorporate it inside other > > > codes. > > > > > > A warning: all this needs a MATIS matrix to work, that is each > > > processor have is local portion of the global matrix in an unassembled > > > fashion. In order to define the Schur complement system, the local > > > process matrix is splitted in four submatrices. At each processor, the > > > 'subdomain' can be further partitioned, this is implemented with a > > > simple minded graph partitioning implemented by reusing the some core > > > routines used inside PETSc for sparse matrix reordering. In order to > > > precondition the Schur complement system, a subsidiary problem can > > > defined by taking some layers of nodes around interface nodes and it > > > is globally iterated with typically a few loops or sub-sub KSP. > > > > > > In short, the implementation is a really involved, and comparable in > > > dificulty to the Newmann-Newman preconditioner (which is also > > > implemented to work with MATIS). At the user level, the only parameter > > > to tweack is the 'sub-subdomain' size and of course tolerances of the > > > inner KSP solver inside the 'SCHUR' preconditioner. > > > > > > Why I never added this to PETSc? Time, lack of serious testing, etc. > > > but mainly because I'm not sure of it is really a good choice compared > > > to PCASM. In fact, some day I would implement my trics for > > > sub-subdamain partitioning (or perhaps use metis for this) inside ASM. > > > Then the sub-subdomain problems would have a reasonable size for using > > > LU, and finally I would compare PCASM with my PCSCHUR. > > > > > > Currently, we are using this for solving incompressible NS equations > > > with monolithic stabilized FEM formulation, and other problems with > > > badly conditioned global systems. We have found no better way to solve > > > our linear systems for our application, probably because we are not > > > smart enough. > > > > > > Hope you understood me... > > > > > > > On 3/7/08, Kathrin Burckhardt wrote: > > >> Dear nice people, > > >> > > >> Did you ever try to solve a Schur complement system using PETSc? > > >> > > >> > > > > > > > > > > > > > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From dalcinl at gmail.com Mon Mar 10 12:00:58 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 10 Mar 2008 14:00:58 -0300 Subject: Schur complement system In-Reply-To: References: Message-ID: On 3/10/08, Matthew Knepley wrote: > By 4 matrices, I assume you mean A, B, C where > > S = B^T A^{-1} B + C > > right? Yes, but actually four matrices, [[A_II, A_IB], [A_BI, A_BB]], were S = (A_BB - A_BI * A_II^{-1} * A_IB) 'B' are interface nodes and 'I' are interior nodes. The operator does not need to be symmetric. However, I have the strong feeling (from therory as well as practice) that my implementation is not actually better than ASM, except for the fact that I'm iterating over interface nodes only. I need to work a bit on ASM to enable partitioning subdomains (or use the support for setting local subdomain indices) to experiment and be completely sure about this. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From z.sheng at ewi.tudelft.nl Tue Mar 11 09:38:57 2008 From: z.sheng at ewi.tudelft.nl (Zhifeng Sheng) Date: Tue, 11 Mar 2008 15:38:57 +0100 Subject: Can I set verbosity level in Petsc? In-Reply-To: References: Message-ID: <47D69981.9070807@ewi.tudelft.nl> Dear all I uses Petsc in my programme and My programme also takes some arguments. The arguments are first treated by my programme and then past to Petsc whenever needed. So when the Petsc terminated, it would always warn me about some Options that are used (these options have been treated by my code).... Can I set the verbosity level of Petsc and get ride of these warnings? Thanks Best regards Zhifeng Sheng From knepley at gmail.com Tue Mar 11 10:10:31 2008 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 11 Mar 2008 10:10:31 -0500 Subject: Can I set verbosity level in Petsc? In-Reply-To: <47D69981.9070807@ewi.tudelft.nl> References: <47D69981.9070807@ewi.tudelft.nl> Message-ID: On Tue, Mar 11, 2008 at 9:38 AM, Zhifeng Sheng wrote: > Dear all > > I uses Petsc in my programme and My programme also takes some arguments. > > The arguments are first treated by my programme and then past to Petsc > whenever needed. So when the Petsc terminated, it would always warn me > about some Options that are used (these options have been treated by my > code).... > > Can I set the verbosity level of Petsc and get ride of these warnings? -options_left no Thanks, Matt > Thanks > Best regards > Zhifeng Sheng > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From aldo.bonfiglioli at unibas.it Tue Mar 11 11:31:29 2008 From: aldo.bonfiglioli at unibas.it (Aldo Bonfiglioli) Date: Tue, 11 Mar 2008 17:31:29 +0100 Subject: Non repeatability issue In-Reply-To: <4715D89B.8070005@unibas.it> References: <4715D89B.8070005@unibas.it> Message-ID: <47D6B3E1.5070606@unibas.it> Dear all, this is a follow-up to an old mail concerning non-repeatibility issues in a parallel environment. We are solving the steady 3D RANS eqns using Newtons's algorithm. All equations (turbulence included) are fully coupled. Our non-linear convergence history shows remarkable non-repeatibility among subsequent parallel runs (see the enclosed plot referred to as MeTis partitioning). We believe (but of course cannot rule out other reasons) that this is due to the fact that within those "ghosted" nodes that are shared by more than two sub-domains the rhs will be slightly different among subsequent runs depending on the order by which contributions are accumulated in these interfacial nodes. Since we push convergence down to machine accuracy, this may not be irrelevant. We then devised an alternative partitioning (though applicable only on very simple geometries) that guarantees that "ghosted" nodes are shared by not more than two procs. This guarantees that not more than two contributions will be accumulated in the ghosted nodes. Using this partitioning (referred to as geometric in the enclosed plots) subsequent runs give indeed identical results. In all cases, of course, we start from identical initial solutions. A colleague of ours has suggested the following: > This could be a problem of GMRES least-squares > system being poorly conditioned. I would try set a higher (very high in > fact) the tolerance for the inner (linear system) solve and see whether > the outer (Newton) iterations become more predictable. > The other possibility is indeed in the communication "inexectness". Are > there any asynchronous communications used in PETSc or in your code. > Change all you can to synchronous communications and see whether results > become more stable. Could you comment in particular on this latter? Is there a way we can modify PETSc's behaviour via command line options or otherwise? Regards, Aldo -- Dr. Aldo Bonfiglioli Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) Universita' della Basilicata V.le dell'Ateneo lucano, 10 85100 Potenza ITALY tel:+39.0971.205203 fax:+39.0971.205160 -------------- next part -------------- A non-text attachment was scrubbed... Name: geometricPartitioning.pdf Type: application/x-download Size: 3634 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MetisPartitioning.pdf Type: application/x-download Size: 8193 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Mar 12 08:41:58 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 Mar 2008 08:41:58 -0500 Subject: Non repeatability issue In-Reply-To: <47D6B3E1.5070606@unibas.it> References: <4715D89B.8070005@unibas.it> <47D6B3E1.5070606@unibas.it> Message-ID: <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> Aldo, 1) Have you made runs where you require, say -ksp_rtol 1.e-12 to eliminate the effects of not solving the linear systems accurately? 2) Have you run the exact example that you ran with geometric decomposition also with the parmetis decomposition? Is that what you sent? (This is too eliminate any fundamental differences with the two problems.) 3) In your plots you plot L_2 norm of the mass residual while Newton is running on all equations. This means the Newton's criteria for progress is based on || u,v,m .....|| as it chugs along. What do plots of || u,v, m....|| (that is what Newton calls the residual when you use -snes_monitor) look like, are they also unstable? Are they decreasing? Sometimes people scale some equations stronger than others if those are the residuals they are most interested in pushing down. What happens if you scale the mass residual equations by some factor (say 100 or 1000) in your FormFunction? 4) Getting MPI to do the updates the same every run is not trivial; I'll think about how it might be done. Barry On Mar 11, 2008, at 11:31 AM, Aldo Bonfiglioli wrote: > Dear all, > this is a follow-up to an old mail concerning non-repeatibilityIn > issues in a parallel environment. > > We are solving the steady 3D RANS eqns using > Newtons's algorithm. All equations (turbulence included) > are fully coupled. > > Our non-linear convergence history shows remarkable > non-repeatibility among subsequent parallel runs > (see the enclosed plot referred to as MeTis partitioning). > > We believe (but of course cannot rule out other > reasons) that this is due to the fact that > within those "ghosted" nodes that are shared by more > than two sub-domains the rhs will be slightly > different among subsequent runs > depending on the order by which contributions > are accumulated in these interfacial nodes. > Since we push convergence down to machine accuracy, > this may not be irrelevant. > > We then devised an alternative partitioning > (though applicable only on very simple geometries) > that guarantees that "ghosted" nodes are shared > by not more than two procs. This guarantees that > not more than two contributions will be accumulated > in the ghosted nodes. > Using this partitioning (referred to as geometric > in the enclosed plots) > subsequent runs give indeed identical results. > > In all cases, of course, we start from identical initial > solutions. > > A colleague of ours has suggested the following: > >> This could be a problem of GMRES least-squares >> system being poorly conditioned. I would try set a higher (very >> high in >> fact) the tolerance for the inner (linear system) solve and see >> whether >> the outer (Newton) iterations become more predictable. > > >> The other possibility is indeed in the communication "inexectness". >> Are >> there any asynchronous communications used in PETSc or in your code. >> Change all you can to synchronous communications and see whether >> results >> become more stable. > > Could you comment in particular on this latter? > Is there a way we can modify PETSc's behaviour via command line > options or otherwise? > > Regards, > Aldo > > -- > Dr. Aldo Bonfiglioli > Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) > Universita' della Basilicata > V.le dell'Ateneo lucano, 10 85100 Potenza ITALY > tel:+39.0971.205203 fax:+39.0971.205160 > > From aldo.bonfiglioli at unibas.it Wed Mar 12 11:02:59 2008 From: aldo.bonfiglioli at unibas.it (Aldo Bonfiglioli) Date: Wed, 12 Mar 2008 17:02:59 +0100 Subject: Non repeatability issue In-Reply-To: <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> References: <4715D89B.8070005@unibas.it> <47D6B3E1.5070606@unibas.it> <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> Message-ID: <47D7FEB3.8050204@unibas.it> Barry, thanks for your prompt reply. > 1) Have you made runs where you require, say -ksp_rtol 1.e-12 to > eliminate the effects of > not solving the linear systems accurately? I am trying this right now, although it may require very many linear iterations, particularly in the last Newton steps, when the CFL number becomes infinite. The systems arising from the fully coupled RANS eqns appear to be fairly ill-conditioned: even in the 2D cases we generally need to increase the level of fill beyond 0 to obtain reasonable convergence of the linear solver. In the 3D case (internal flow through an elbow) the previous plots refer to, there are about 200,000 nodes times 5 dof per node and I have been using BJ+ILU(1) with GMRES(60). > 2) Have you run the exact example that you ran with geometric > decomposition also with > the parmetis decomposition? Is that what you sent? (This is too > eliminate any fundamental > differences with the two problems.) Precisely. > 3) In your plots you plot L_2 norm of the mass residual while Newton > is running on all > equations. This means the Newton's criteria for progress is based on > || u,v,m .....|| > as it chugs along. What do plots of || u,v, m....|| (that is what > Newton calls the residual > when you use -snes_monitor) look like, are they also unstable? Are > they decreasing? All residuals appear to be fairly un-stable. This is shown in the enclosed plot. The strategy to ramp the CFL number looks at the ratio btw the current non-linear residual _of one of the conservation eqns_ and the initial residual. The Newton strategy adopted in the code is NOT implemented through PETSc's TS (just for "historical" reasons). > Sometimes people scale some equations stronger than others if those > are the residuals > they are most interested in pushing down. What happens if you scale > the mass residual > equations by some factor (say 100 or 1000) in your FormFunction? Never tried. We had once tested on a different problem row scaling without seeing noticeable differences. But I guess this may be different from what you mention. Regards, Aldo -- Dr. Aldo Bonfiglioli Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) Universita' della Basilicata V.le dell'Ateneo lucano, 10 85100 Potenza ITALY tel:+39.0971.205203 fax:+39.0971.205160 -------------- next part -------------- A non-text attachment was scrubbed... Name: convhst.pdf Type: application/pdf Size: 8054 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Mar 12 12:26:28 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 Mar 2008 12:26:28 -0500 Subject: Non repeatability issue In-Reply-To: <47D7FEB3.8050204@unibas.it> References: <4715D89B.8070005@unibas.it> <47D6B3E1.5070606@unibas.it> <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> <47D7FEB3.8050204@unibas.it> Message-ID: <929484FD-22FD-4134-99D3-6D0ABB3CBC73@mcs.anl.gov> Aldo, I would scale the turbulence equation to match the scaling of the rest of the equations. I suspect you are ramping up the CFL number too quickly. For a "continuation" type Newton method to work well (in my mindset anyways), you need to have the solution well converged before moving on to the next "continuation" point. Worth experimenting anyways. Barry On Mar 12, 2008, at 11:02 AM, Aldo Bonfiglioli wrote: > Barry, > thanks for your prompt reply. > >> 1) Have you made runs where you require, say -ksp_rtol 1.e-12 >> to eliminate the effects of >> not solving the linear systems accurately? > > I am trying this right now, although it may require very many linear > iterations, particularly in the last Newton steps, when the CFL number > becomes infinite. > The systems arising from the fully coupled RANS eqns appear to be > fairly ill-conditioned: > even in the 2D cases we generally need to increase the level of fill > beyond 0 > to obtain reasonable convergence of the linear solver. > > In the 3D case (internal flow through an elbow) the previous plots > refer to, there are > about 200,000 nodes times 5 dof per node and > I have been using BJ+ILU(1) with GMRES(60). > >> 2) Have you run the exact example that you ran with geometric >> decomposition also with >> the parmetis decomposition? Is that what you sent? (This is too >> eliminate any fundamental >> differences with the two problems.) > > Precisely. > >> 3) In your plots you plot L_2 norm of the mass residual while >> Newton is running on all >> equations. This means the Newton's criteria for progress is based >> on || u,v,m .....|| >> as it chugs along. What do plots of || u,v, m....|| (that is what >> Newton calls the residual >> when you use -snes_monitor) look like, are they also unstable? Are >> they decreasing? > > All residuals appear to be fairly un-stable. This is shown in the > enclosed plot. > The strategy to ramp the CFL number looks at the ratio btw the > current non-linear > residual _of one of the conservation eqns_ and the initial residual. > The Newton strategy adopted in the code is NOT implemented through > PETSc's TS > (just for "historical" reasons). > >> Sometimes people scale some equations stronger than others if >> those are the residuals >> they are most interested in pushing down. What happens if you >> scale the mass residual >> equations by some factor (say 100 or 1000) in your FormFunction? > > Never tried. We had once tested on a different problem row scaling > without seeing noticeable differences. But I guess this may be > different from what you mention. > > > Regards, > Aldo > > -- > Dr. Aldo Bonfiglioli > Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) > Universita' della Basilicata > V.le dell'Ateneo lucano, 10 85100 Potenza ITALY > tel:+39.0971.205203 fax:+39.0971.205160 > > From bsmith at mcs.anl.gov Wed Mar 12 16:37:38 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 Mar 2008 16:37:38 -0500 Subject: Non repeatability issue In-Reply-To: <47D7FEB3.8050204@unibas.it> References: <4715D89B.8070005@unibas.it> <47D6B3E1.5070606@unibas.it> <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> <47D7FEB3.8050204@unibas.it> Message-ID: Aldo, Actually it is far easier than I thought. I have added the argument -vecscatter_reproduce that will cause the receives to always be processed in the same order (though order or operations in the MPI reductions may still result in slightly different convergence histories.) to petsc-dev http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html If you are using petsc-2.3.3 and do no wish to change to using petsc-dev you can edit src/vec/vec/utils/vpscat.h and replace the line ierr = MPI_Waitany(nrecvs,rwaits,&imdex,&xrstatus);CHKERRQ(ierr); with the two lines imdex = count - 1; ierr = MPI_Wait(rwaits+imdex,&xrstatus);CHKERRQ(ierr); then run "make libfast" in that directory. Hope this helps clear things up, Barry On Mar 12, 2008, at 11:02 AM, Aldo Bonfiglioli wrote: > Barry, > thanks for your prompt reply. > >> 1) Have you made runs where you require, say -ksp_rtol 1.e-12 >> to eliminate the effects of >> not solving the linear systems accurately? > > I am trying this right now, although it may require very many linear > iterations, particularly in the last Newton steps, when the CFL number > becomes infinite. > The systems arising from the fully coupled RANS eqns appear to be > fairly ill-conditioned: > even in the 2D cases we generally need to increase the level of fill > beyond 0 > to obtain reasonable convergence of the linear solver. > > In the 3D case (internal flow through an elbow) the previous plots > refer to, there are > about 200,000 nodes times 5 dof per node and > I have been using BJ+ILU(1) with GMRES(60). > >> 2) Have you run the exact example that you ran with geometric >> decomposition also with >> the parmetis decomposition? Is that what you sent? (This is too >> eliminate any fundamental >> differences with the two problems.) > > Precisely. > >> 3) In your plots you plot L_2 norm of the mass residual while >> Newton is running on all >> equations. This means the Newton's criteria for progress is based >> on || u,v,m .....|| >> as it chugs along. What do plots of || u,v, m....|| (that is what >> Newton calls the residual >> when you use -snes_monitor) look like, are they also unstable? Are >> they decreasing? > > All residuals appear to be fairly un-stable. This is shown in the > enclosed plot. > The strategy to ramp the CFL number looks at the ratio btw the > current non-linear > residual _of one of the conservation eqns_ and the initial residual. > The Newton strategy adopted in the code is NOT implemented through > PETSc's TS > (just for "historical" reasons). > >> Sometimes people scale some equations stronger than others if >> those are the residuals >> they are most interested in pushing down. What happens if you >> scale the mass residual >> equations by some factor (say 100 or 1000) in your FormFunction? > > Never tried. We had once tested on a different problem row scaling > without seeing noticeable differences. But I guess this may be > different from what you mention. > > > Regards, > Aldo > > -- > Dr. Aldo Bonfiglioli > Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) > Universita' della Basilicata > V.le dell'Ateneo lucano, 10 85100 Potenza ITALY > tel:+39.0971.205203 fax:+39.0971.205160 > > From tribur at vision.ee.ethz.ch Mon Mar 17 07:20:09 2008 From: tribur at vision.ee.ethz.ch (Kathrin Burckhardt) Date: Mon, 17 Mar 2008 13:20:09 +0100 (CET) Subject: still Schur complement Message-ID: Dear all, My problem, once again and more concrete: I have given the local Schur matrices Si and I would like to set-up the global matrix S without assembling the Si on one processor. I thought "MATIS" is the suitable matrix type, but I don't succeed in using it. My code is: IS is; ISCreateGeneral(PETSC_COMM_SELF, NPb, &uBId_global[0], &is); ISLocalToGlobalMapping mapping; ISLocalToGlobalMappingCreateIS(is, &mapping); Mat Stot; MatCreate(PETSC_COMM_WORLD,&Stot); MatSetLocalToGlobalMapping(Stot, mapping); MatSetSizes(Stot,NPb,NPb,NPb_tot,NPb_tot); MatSetType(Stot,MATIS); //or, alternatively //MatCreateIS(PETSC_COMM_WORLD,NPb,NPb,NPb_tot,NPb_tot,mapping,&Stot); for(int i=0; i References: Message-ID: I'm not completelly sure what are you actually trying to do. It seems you know how to actually compute the local Schur complement in a entry-by-entry basis. In that case, your way is semms to be OK. However, you get a error because of a inner implementation detail of MATIS. Do de following: set the local-to-global mapping AFTER setting the matrix type. This way, your code should now work. The other way, is to create the matrix by calling MatCreateIS(), look at the source code of this function, and you will basically see the same steps you are using in your example for setting-up the matrix, however the call to MatSetLocalToGlobalMapping() appears at the very end. Regards, On 3/17/08, Kathrin Burckhardt wrote: > Dear all, > > My problem, once again and more concrete: > > I have given the local Schur matrices Si and I would like to set-up the > global matrix S without assembling the Si on one processor. > > I thought "MATIS" is the suitable matrix type, but I don't succeed in > using it. My code is: > > IS is; > ISCreateGeneral(PETSC_COMM_SELF, NPb, &uBId_global[0], &is); > ISLocalToGlobalMapping mapping; > ISLocalToGlobalMappingCreateIS(is, &mapping); > Mat Stot; > MatCreate(PETSC_COMM_WORLD,&Stot); > MatSetLocalToGlobalMapping(Stot, mapping); > MatSetSizes(Stot,NPb,NPb,NPb_tot,NPb_tot); > MatSetType(Stot,MATIS); > //or, alternatively > //MatCreateIS(PETSC_COMM_WORLD,NPb,NPb,NPb_tot,NPb_tot,mapping,&Stot); > for(int i=0; i for(int j=0; j MatSetValuesLocal(Stot, 1, &i, 1, &j, &S(i+1,j+1), ADD_VALUES); > > And I got the error > > [1]PETSC ERROR: MatSetValuesLocal() line 1471 in > src/mat/interface/matrix.c > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Object is in wrong state! > [1]PETSC ERROR: Object Type not set: Argument # 1! > > Can you tell me what's wrong? > > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From knepley at gmail.com Mon Mar 17 10:13:18 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Mar 2008 10:13:18 -0500 Subject: still Schur complement In-Reply-To: References: Message-ID: On Mon, Mar 17, 2008 at 7:20 AM, Kathrin Burckhardt wrote: > Dear all, > > My problem, once again and more concrete: > > I have given the local Schur matrices Si and I would like to set-up the > global matrix S without assembling the Si on one processor. > > I thought "MATIS" is the suitable matrix type, but I don't succeed in > using it. My code is: > > IS is; > ISCreateGeneral(PETSC_COMM_SELF, NPb, &uBId_global[0], &is); > ISLocalToGlobalMapping mapping; > ISLocalToGlobalMappingCreateIS(is, &mapping); > Mat Stot; > MatCreate(PETSC_COMM_WORLD,&Stot); > MatSetLocalToGlobalMapping(Stot, mapping); > MatSetSizes(Stot,NPb,NPb,NPb_tot,NPb_tot); > MatSetType(Stot,MATIS); > //or, alternatively > //MatCreateIS(PETSC_COMM_WORLD,NPb,NPb,NPb_tot,NPb_tot,mapping,&Stot); > for(int i=0; i for(int j=0; j MatSetValuesLocal(Stot, 1, &i, 1, &j, &S(i+1,j+1), ADD_VALUES); > > And I got the error > > [1]PETSC ERROR: MatSetValuesLocal() line 1471 in > src/mat/interface/matrix.c > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Object is in wrong state! > [1]PETSC ERROR: Object Type not set: Argument # 1! > > Can you tell me what's wrong? It appears the code snipper above should work, however it is usually because something in the actual code was not sent. Also, if the type was not set, it should fail on all processes. I would go in with the debugger and look at Stot to make sure it has a type. You can also call the MatValidType(mat,1) method explicitly in your code to check. However, I have another question. Are you sure that you want MATIS for the Schur complement? This seems strange to me. You usually want unassembled matrices for subdomain problems. What is wrong with just assembling it (if you indeed have the local pieces)? Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From tribur at vision.ee.ethz.ch Mon Mar 17 16:54:27 2008 From: tribur at vision.ee.ethz.ch (Kathrin Burckhardt) Date: Mon, 17 Mar 2008 22:54:27 +0100 (CET) Subject: still Schur complement In-Reply-To: References: Message-ID: Dear Lisandro, > I'm not completelly sure what are you actually trying to do. It seems > you know how to actually compute the local Schur complement in a > entry-by-entry basis. In that case, your way is semms to be OK. What do you mean with entry-by-entry basis? Anyway, I have the local Schur complement explicitely, in form of a matrix (or C++ vector). > However, you get a error because of a inner implementation detail of > MATIS. Do de following: set the local-to-global mapping AFTER setting > the matrix type. This way, your code should now work. Unfortunately, changing the order of the Mat-commands gives me the same error. I guess I found the origin of the problem: I put it in the order below because I thought to avoid [2]PETSC ERROR: Sum of local lengths 3916 does not equal global length 1958 which is due to the fact that in the case of 2 Subdomains and 2 processors, e.g., it is NPb=NPb_tot in the Schur system, instead of NPb=NPb_tot/2. However, I realize just now that the order of the Mat-commands has no influence on this, and that the error mentioned below has probably its origin there. I have chosen MATIS because in the manual page of MatCreateIS() it is written "...m and n are NOT related to the size of the map", but maybe I understand this setence wrong? How can I choose m,n and M,N independently? > On 3/17/08, Kathrin Burckhardt wrote: >> Dear all, >> >> My problem, once again and more concrete: >> >> I have given the local Schur matrices Si and I would like to set-up the >> global matrix S without assembling the Si on one processor. >> >> I thought "MATIS" is the suitable matrix type, but I don't succeed in >> using it. My code is: >> >> IS is; >> ISCreateGeneral(PETSC_COMM_SELF, NPb, &uBId_global[0], &is); >> ISLocalToGlobalMapping mapping; >> ISLocalToGlobalMappingCreateIS(is, &mapping); >> Mat Stot; >> MatCreate(PETSC_COMM_WORLD,&Stot); >> MatSetLocalToGlobalMapping(Stot, mapping); >> MatSetSizes(Stot,NPb,NPb,NPb_tot,NPb_tot); >> MatSetType(Stot,MATIS); >> //or, alternatively >> //MatCreateIS(PETSC_COMM_WORLD,NPb,NPb,NPb_tot,NPb_tot,mapping,&Stot); >> for(int i=0; i> for(int j=0; j> MatSetValuesLocal(Stot, 1, &i, 1, &j, &S(i+1,j+1), ADD_VALUES); >> >> And I got the error >> >> [1]PETSC ERROR: MatSetValuesLocal() line 1471 in >> src/mat/interface/matrix.c >> [1]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [1]PETSC ERROR: Object is in wrong state! >> [1]PETSC ERROR: Object Type not set: Argument # 1! >> >> Can you tell me what's wrong? >> >> >> > > > From rlmackie862 at gmail.com Mon Mar 17 17:28:35 2008 From: rlmackie862 at gmail.com (Randall Mackie) Date: Mon, 17 Mar 2008 15:28:35 -0700 Subject: question on MatMatMult Message-ID: <47DEF093.5040605@gmail.com> If I want to compute H = L^T L where L is a sparse matrix which is an approximation to the laplacian, and hence H is the biharmonic (also sparse), and if I have L as an MPI matrix in PETSc, will MatMatMult work for this (assuming I create the transpose of L first). In other words, does MatMatMult look at the non-zero structure only that would result, or does it think the result is a dense matrix? Thanks, Randy From knepley at gmail.com Mon Mar 17 17:40:12 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Mar 2008 17:40:12 -0500 Subject: question on MatMatMult In-Reply-To: <47DEF093.5040605@gmail.com> References: <47DEF093.5040605@gmail.com> Message-ID: On Mon, Mar 17, 2008 at 5:28 PM, Randall Mackie wrote: > If I want to compute H = L^T L where L is a sparse matrix which is an approximation to the > laplacian, and hence H is the biharmonic (also sparse), and if I have L as an MPI matrix > in PETSc, will MatMatMult work for this (assuming I create the transpose of L first). > > In other words, does MatMatMult look at the non-zero structure only that would result, > or does it think the result is a dense matrix? It builds the structure dynamically, which explains the "fill" argument. This is not a great thing to do unless you have no idea how to form it directly. Matt > Thanks, Randy -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From rlmackie862 at gmail.com Mon Mar 17 18:03:18 2008 From: rlmackie862 at gmail.com (Randall Mackie) Date: Mon, 17 Mar 2008 16:03:18 -0700 Subject: question on MatMatMult In-Reply-To: References: <47DEF093.5040605@gmail.com> Message-ID: <47DEF8B6.5040407@gmail.com> Matthew Knepley wrote: > On Mon, Mar 17, 2008 at 5:28 PM, Randall Mackie wrote: >> If I want to compute H = L^T L where L is a sparse matrix which is an approximation to the >> laplacian, and hence H is the biharmonic (also sparse), and if I have L as an MPI matrix >> in PETSc, will MatMatMult work for this (assuming I create the transpose of L first). >> >> In other words, does MatMatMult look at the non-zero structure only that would result, >> or does it think the result is a dense matrix? > > It builds the structure dynamically, which explains the "fill" > argument. This is not a great > thing to do unless you have no idea how to form it directly. I know how to form the Laplacian (that's easy) but I do not know how to form the biharmonic directly, on a non-uniform grid. If anyone knows how, other than direct multiplication, and can point me in the right direction, that would be most appreciated. Randy > > Matt > >> Thanks, Randy From knepley at gmail.com Mon Mar 17 18:18:47 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 17 Mar 2008 18:18:47 -0500 Subject: question on MatMatMult In-Reply-To: <47DEF8B6.5040407@gmail.com> References: <47DEF093.5040605@gmail.com> <47DEF8B6.5040407@gmail.com> Message-ID: On Mon, Mar 17, 2008 at 6:03 PM, Randall Mackie wrote: > > > Matthew Knepley wrote: > > On Mon, Mar 17, 2008 at 5:28 PM, Randall Mackie wrote: > >> If I want to compute H = L^T L where L is a sparse matrix which is an approximation to the > >> laplacian, and hence H is the biharmonic (also sparse), and if I have L as an MPI matrix > >> in PETSc, will MatMatMult work for this (assuming I create the transpose of L first). > >> > >> In other words, does MatMatMult look at the non-zero structure only that would result, > >> or does it think the result is a dense matrix? > > > > It builds the structure dynamically, which explains the "fill" > > argument. This is not a great > > thing to do unless you have no idea how to form it directly. > > I know how to form the Laplacian (that's easy) but I do not know how > to form the biharmonic directly, on a non-uniform grid. If anyone > knows how, other than direct multiplication, and can point me in > the right direction, that would be most appreciated. I found http://links.jstor.org/sici?sici=0036-1429(199604)33%3A2%3C555%3ATSMFTB%3E2.0.CO%3B2-8 http://citeseer.ist.psu.edu/13843.html http://links.jstor.org/sici?sici=0036-1429(197806)15%3A3%3C556%3AAOTBEB%3E2.0.CO%3B2-E Matt > Randy > > > > > > Matt > > > >> Thanks, Randy > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From rlmackie862 at gmail.com Mon Mar 17 18:23:38 2008 From: rlmackie862 at gmail.com (Randall Mackie) Date: Mon, 17 Mar 2008 16:23:38 -0700 Subject: question on MatMatMult In-Reply-To: References: <47DEF093.5040605@gmail.com> <47DEF8B6.5040407@gmail.com> Message-ID: <47DEFD7A.4010808@gmail.com> Thanks Matt, I'll look into these. Randy Matthew Knepley wrote: > On Mon, Mar 17, 2008 at 6:03 PM, Randall Mackie wrote: >> >> Matthew Knepley wrote: >> > On Mon, Mar 17, 2008 at 5:28 PM, Randall Mackie wrote: >> >> If I want to compute H = L^T L where L is a sparse matrix which is an approximation to the >> >> laplacian, and hence H is the biharmonic (also sparse), and if I have L as an MPI matrix >> >> in PETSc, will MatMatMult work for this (assuming I create the transpose of L first). >> >> >> >> In other words, does MatMatMult look at the non-zero structure only that would result, >> >> or does it think the result is a dense matrix? >> > >> > It builds the structure dynamically, which explains the "fill" >> > argument. This is not a great >> > thing to do unless you have no idea how to form it directly. >> >> I know how to form the Laplacian (that's easy) but I do not know how >> to form the biharmonic directly, on a non-uniform grid. If anyone >> knows how, other than direct multiplication, and can point me in >> the right direction, that would be most appreciated. > > I found > > http://links.jstor.org/sici?sici=0036-1429(199604)33%3A2%3C555%3ATSMFTB%3E2.0.CO%3B2-8 > > http://citeseer.ist.psu.edu/13843.html > > http://links.jstor.org/sici?sici=0036-1429(197806)15%3A3%3C556%3AAOTBEB%3E2.0.CO%3B2-E > > Matt > >> Randy >> >> >> > >> > Matt >> > >> >> Thanks, Randy >> >> > > > From aldo.bonfiglioli at unibas.it Tue Mar 18 05:57:59 2008 From: aldo.bonfiglioli at unibas.it (Aldo Bonfiglioli) Date: Tue, 18 Mar 2008 11:57:59 +0100 Subject: Non repeatability issue In-Reply-To: <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> References: <4715D89B.8070005@unibas.it> <47D6B3E1.5070606@unibas.it> <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> Message-ID: <47DFA037.7030707@unibas.it> > > 1) Have you made runs where you require, say -ksp_rtol 1.e-12 to > eliminate the effects of > not solving the linear systems accurately? I have performed two runs with ksp_rtol = 1.e-12. The relevant plots are enclosed where comparisons are made with PETSc's default for ksp_rtol. In one of these two runs Newton even diverges. It should however be mentioned that at least for some Newton steps, the linear solver does not meet the convergence criterion in 2000 linear iterations (I reduced the default). > I have added the argument -vecscatter_reproduce > that will cause the receives to always be processed in the same order > (though order or > operations in the MPI reductions may still result in slightly > different convergence histories.) > > Hope this helps clear things up, I am not sure "-vecscatter_reproduce " has changed the situation much. Out of 4 subsequent runs, 2 converge while 2 enter a limit cycle I had not seen previously (I mean without the -vecscatter_reproduce option). The initial solution is the same as that shown in the other plot. Aldo -- Dr. Aldo Bonfiglioli Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) Universita' della Basilicata V.le dell'Ateneo lucano, 10 85100 Potenza ITALY tel:+39.0971.205203 fax:+39.0971.205160 -------------- next part -------------- A non-text attachment was scrubbed... Name: low_ksp_rtol.pdf Type: application/pdf Size: 10354 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: vecscatter_reproduce.pdf Type: application/pdf Size: 13056 bytes Desc: not available URL: From knepley at gmail.com Tue Mar 18 08:00:40 2008 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 Mar 2008 08:00:40 -0500 Subject: Non repeatability issue In-Reply-To: <47DFA037.7030707@unibas.it> References: <4715D89B.8070005@unibas.it> <47D6B3E1.5070606@unibas.it> <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> <47DFA037.7030707@unibas.it> Message-ID: To me this looks like 1) You have way too many Newton steps. Newton is quadratically convergent, so if you have 100+ steps, it means you are very very far from the solution when you begin. In this region, Newton is a really bad algorithm and can be very very sensitive to perturbations. I would never expect it to be reproducible. From the well-known fracint demo, it can be chaotic. 2) I would also question whether the system is actually stable. Instability can result in occasional divergence based upon small perturbations. Thus, I am not sure you are asking the correct question with this solve. I don't think reproducible Newton over 100s of steps makes sense theoretically. Maybe Barry thinks differently. Matt On Tue, Mar 18, 2008 at 5:57 AM, Aldo Bonfiglioli wrote: > > > > 1) Have you made runs where you require, say -ksp_rtol 1.e-12 to > > eliminate the effects of > > not solving the linear systems accurately? > > > I have performed two runs with ksp_rtol = 1.e-12. The relevant plots are > enclosed > where comparisons are made with PETSc's default for ksp_rtol. > In one of these two runs Newton even diverges. > It should however be mentioned that > at least for some Newton steps, the linear solver does not meet the > convergence criterion in 2000 linear iterations (I reduced the default). > > > I have added the argument -vecscatter_reproduce > > that will cause the receives to always be processed in the same order > > (though order or > > operations in the MPI reductions may still result in slightly > > different convergence histories.) > > > > Hope this helps clear things up, > > I am not sure "-vecscatter_reproduce " has changed the situation much. > Out of 4 subsequent runs, 2 converge > while 2 enter a limit cycle I had not seen previously > (I mean without the -vecscatter_reproduce option). > The initial solution is the same as that shown in the other plot. > > Aldo > > -- > Dr. Aldo Bonfiglioli > Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) > Universita' della Basilicata > V.le dell'Ateneo lucano, 10 85100 Potenza ITALY > tel:+39.0971.205203 fax:+39.0971.205160 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From bsmith at mcs.anl.gov Tue Mar 18 08:30:40 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 18 Mar 2008 08:30:40 -0500 Subject: Non repeatability issue In-Reply-To: <47DFA037.7030707@unibas.it> References: <4715D89B.8070005@unibas.it> <47D6B3E1.5070606@unibas.it> <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> <47DFA037.7030707@unibas.it> Message-ID: <81BF6B4F-9199-41AA-B571-20186E2DE43E@mcs.anl.gov> 1) Are you sure the -vecscatter_reproduce is working, run with - options_left and see if says the option was not used. 2) did you do the -ksp_rtol 1.e-12 at the same time as the - vecscatter_reproduce? They must be done together. 3) what happens on 1 process? Does it behave exactly the same for two identical runs? 4) there is too much going on here to figure out why you get this behavior. Can you please FIX the continuation parameter and run until Newton converges? Is the convergence in this case reproduceable? What happens if you only change the continuation parameter after you have Newton fully converged for the current continuation parameter? is that reproduceable. 5) in the plots can you indicate when the continuation parameter is changed? You are merging two algorithms (Newton's method and continuation together) you cannot hope to understand that merged algorithm until you understand each of them seperately. Barry On Mar 18, 2008, at 5:57 AM, Aldo Bonfiglioli wrote: >> >> 1) Have you made runs where you require, say -ksp_rtol 1.e-12 >> to eliminate the effects of >> not solving the linear systems accurately? > > > I have performed two runs with ksp_rtol = 1.e-12. The relevant plots > are enclosed > where comparisons are made with PETSc's default for ksp_rtol. > In one of these two runs Newton even diverges. > It should however be mentioned that > at least for some Newton steps, the linear solver does not meet the > convergence criterion in 2000 linear iterations (I reduced the > default). > >> I have added the argument -vecscatter_reproduce >> that will cause the receives to always be processed in the same >> order (though order or >> operations in the MPI reductions may still result in slightly >> different convergence histories.) >> >> Hope this helps clear things up, > > I am not sure "-vecscatter_reproduce " has changed the situation much. > Out of 4 subsequent runs, 2 converge > while 2 enter a limit cycle I had not seen previously > (I mean without the -vecscatter_reproduce option). > The initial solution is the same as that shown in the other plot. > > Aldo > > -- > Dr. Aldo Bonfiglioli > Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) > Universita' della Basilicata > V.le dell'Ateneo lucano, 10 85100 Potenza ITALY > tel:+39.0971.205203 fax:+39.0971.205160 > > From aldo.bonfiglioli at unibas.it Tue Mar 18 08:48:27 2008 From: aldo.bonfiglioli at unibas.it (Aldo Bonfiglioli) Date: Tue, 18 Mar 2008 14:48:27 +0100 Subject: Non repeatability issue In-Reply-To: References: <4715D89B.8070005@unibas.it> <47D6B3E1.5070606@unibas.it> <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> <47DFA037.7030707@unibas.it> Message-ID: <47DFC82B.90904@unibas.it> > > > 1) You have way too many Newton steps. Newton is quadratically >convergent, so if > you have 100+ steps, it means you are very very far from the >solution when you > begin. In this region, Newton is a really bad algorithm and can >be very very > sensitive to perturbations. I would never expect it to be >reproducible. From the > well-known fracint demo, it can be chaotic. > The initial solution from which Newton's algorithm is started is obtained using a fixed point iterative scheme in which the turbulence transport eqn is solved de-coupled from the mean-flow eqns. This de-coupled solution strategy usually stalls (in terms of residual decrease) after a residual drop of a few orders of magnitude. At this stage Newton is invoked. From a qualitative view-point (say at plotting level) the initial solution is close to the fully converged one, but I will try to quantify this and check how mutually far or close are the two. Even for the 2D RANS eqns we have noticed a fairly "long" plateau in terms of residual convergence history before quadratic convergence occurs. This is also in line with the few references on the subject I have in mind. 3D Euler is far more benign in our experience. > 2) I would also question whether the system is actually stable. >Instability can result > in occasional divergence based upon small perturbations. > We know this might occur. At this stage we are still trying to make sure there are no implementation bugs in our code. Aldo -- Dr. Aldo Bonfiglioli Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) Universita' della Basilicata V.le dell'Ateneo lucano, 10 85100 Potenza ITALY tel:+39.0971.205203 fax:+39.0971.205160 From aldo.bonfiglioli at unibas.it Tue Mar 18 11:52:37 2008 From: aldo.bonfiglioli at unibas.it (Aldo Bonfiglioli) Date: Tue, 18 Mar 2008 17:52:37 +0100 Subject: Non repeatability issue In-Reply-To: <81BF6B4F-9199-41AA-B571-20186E2DE43E@mcs.anl.gov> References: <4715D89B.8070005@unibas.it> <47D6B3E1.5070606@unibas.it> <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> <47DFA037.7030707@unibas.it> <81BF6B4F-9199-41AA-B571-20186E2DE43E@mcs.anl.gov> Message-ID: <47DFF355.2000305@unibas.it> > 1) Are you sure the -vecscatter_reproduce is working, run with - > options_left and see if > says the option was not used. I have harwired it into the 2.3.3-p8, following your suggestion. > 2) did you do the -ksp_rtol 1.e-12 at the same time as the - > vecscatter_reproduce? They > must be done together. No, I tested the two separately. I will do as you suggest. > 3) what happens on 1 process? Does it behave exactly the same for two > identical runs? The current testcase is too large to be run on a single processor. Tests performed with smaller datasets (both 2 and 3D) have shown that on 1 proc subsequent runs produce identical output. It should also be mentioned that, on a different grid (somewhat less stretched) the same testcase produces far more repeatible non-linear convergence histories. By "far more repeatible" I mean that the output of subsequent runs are NOT identical, but the non-linear convergence histories are almost superimposed at the "plotting" level, small differences arising only when the residuals are close to machine eps. On this grid, the linear solver (either BJ+ILU(k) or ASM+ILU(k)) also behaves far better. > 4) there is too much going on here to figure out why you get this > behavior. Can you please > FIX the continuation parameter Here I am not sure about the nomenclature: by "continuation parameter" do you mean the strategy by which the pseudo-time derivative term is progressively reduced so as to revert to a true Newton algorithm? My pseudo-time derivative term looks like 1/(CFL) * (Volume/dt) Volume/dt is locally computed based on an explicit stability criterion and CFL is ramped from a starting value (CFL_0, tipically of order 1 or 10) using the ratio between the current and initial residuals of one of the conservation eqns (mass, tipically). In all previous plots, the fixed point iterations were not shown. The fixed point iteration was run only once to generate the initial solution used for all Newton runs. Aldo -- Dr. Aldo Bonfiglioli Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) Universita' della Basilicata V.le dell'Ateneo lucano, 10 85100 Potenza ITALY tel:+39.0971.205203 fax:+39.0971.205160 From bsmith at mcs.anl.gov Tue Mar 18 11:59:07 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 18 Mar 2008 11:59:07 -0500 Subject: Non repeatability issue In-Reply-To: <47DFF355.2000305@unibas.it> References: <4715D89B.8070005@unibas.it> <47D6B3E1.5070606@unibas.it> <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> <47DFA037.7030707@unibas.it> <81BF6B4F-9199-41AA-B571-20186E2DE43E@mcs.anl.gov> <47DFF355.2000305@unibas.it> Message-ID: <49C82E74-937D-4EE5-A640-B06CAE44B4CD@mcs.anl.gov> On Mar 18, 2008, at 11:52 AM, Aldo Bonfiglioli wrote: >> 1) Are you sure the -vecscatter_reproduce is working, run with - >> options_left and see if >> says the option was not used. > > > I have harwired it into the 2.3.3-p8, following your suggestion. > >> 2) did you do the -ksp_rtol 1.e-12 at the same time as the - >> vecscatter_reproduce? They >> must be done together. > > No, I tested the two separately. I will do as you suggest. > >> 3) what happens on 1 process? Does it behave exactly the same for >> two identical runs? > > > The current testcase is too large to be run on a single processor. > Tests performed with smaller datasets (both 2 and 3D) have shown > that on 1 proc > subsequent runs produce identical output. > It should also be mentioned that, on a different grid (somewhat less > stretched) > the same testcase produces far more repeatible non-linear > convergence histories. > By "far more repeatible" I mean that the output of subsequent runs > are NOT > identical, but the non-linear convergence histories are almost > superimposed at the "plotting" level, small differences arising only > when the residuals are close to machine eps. > On this grid, the linear solver (either BJ+ILU(k) or ASM+ILU(k)) > also behaves far better. > >> 4) there is too much going on here to figure out why you get this >> behavior. Can you please >> FIX the continuation parameter > > Here I am not sure about the nomenclature: by "continuation > parameter" do you > mean the strategy by which the pseudo-time derivative term is > progressively reduced so as to revert to a true Newton algorithm? > > My pseudo-time derivative term looks like > 1/(CFL) * (Volume/dt) > > Volume/dt is locally computed based on an explicit stability > criterion and CFL is ramped from a starting value (CFL_0, tipically > of order 1 or 10) > using the ratio between the current and initial residuals of one > of the conservation eqns (mass, tipically). Yes, this is what I mean. When you just fix the CFL and run Newton runs to completion is it stable? Then if you ramp up the CFL much more slowly is it stable and Newton convergence much smoother? > > > In all previous plots, the fixed point iterations were not shown. > The fixed point iteration was run only once to generate the > initial solution used for all Newton runs. That's ok; I don't care about that. What I want to see is the CFL number plotted with the Newton residuals. > > > Aldo > > -- > Dr. Aldo Bonfiglioli > Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) > Universita' della Basilicata > V.le dell'Ateneo lucano, 10 85100 Potenza ITALY > tel:+39.0971.205203 fax:+39.0971.205160 > From mafunk at nmsu.edu Tue Mar 18 17:00:50 2008 From: mafunk at nmsu.edu (Matt Funk) Date: Tue, 18 Mar 2008 16:00:50 -0600 Subject: MG preconditioner issue In-Reply-To: References: <08BD23D9-0CF2-415E-88C0-328DF4D93E0F@columbia.edu> Message-ID: <200803181600.50369.mafunk@nmsu.edu> Hi, i am trying out the MG preconditioner in PETSc and am encountering some issues. In fact when i use the MG preconditioner i get a segmentation violation. This does not happen with other preconditioners. Running Petsc in debug mode with the -info option did not seem to produce any useful data (at least to me) I thought i'd ask if someone encountered this before i use valgrind. I am using 2.3.3-p8. Or does anyone have a hint on where i could start looking? Again, what bothers me is that the other preconditioner do not throw any sort of error. thanks mat From knepley at gmail.com Tue Mar 18 17:30:37 2008 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 Mar 2008 17:30:37 -0500 Subject: MG preconditioner issue In-Reply-To: <200803181600.50369.mafunk@nmsu.edu> References: <08BD23D9-0CF2-415E-88C0-328DF4D93E0F@columbia.edu> <200803181600.50369.mafunk@nmsu.edu> Message-ID: On Tue, Mar 18, 2008 at 5:00 PM, Matt Funk wrote: > Hi, > > i am trying out the MG preconditioner in PETSc and am encountering some > issues. In fact when i use the MG preconditioner i get a segmentation > violation. This does not happen with other preconditioners. If you just do something like -pc_type mg, then you will definitely get an error. PC_MG is just the skeleton for a multigrid preconditioner. MG relies on knowledge of both the operator and domain, which is in general not available to PETSc. If you want something that just works, I suggest configuring with --download-ml or --download-hypre or --download-prometheus and running with the AMG provided in those packages. If you are using a DA, then you can use DMMG which provides the extra information. Matt > Running Petsc in debug mode with the -info option did not seem to produce any > useful data (at least to me) > > I thought i'd ask if someone encountered this before i use valgrind. I am > using 2.3.3-p8. Or does anyone have a hint on where i could start looking? > Again, what bothers me is that the other preconditioner do not throw any sort > of error. > > thanks > mat > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From bsmith at mcs.anl.gov Wed Mar 19 15:04:36 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 19 Mar 2008 15:04:36 -0500 Subject: still Schur complement In-Reply-To: References: Message-ID: Kathrin, The MATIS object (the IS stands for iterative substructuring) is really only good if you want to use the PETSc Neumann-Neumann preconditioner PCNN. This preconditioner is rarely used. Anyways the problem is the lack of clarity of the meaning of the sizes. The size of the IS passed to the MatSetLocalToGlobalMapping() is NOT the same as the local size passed to MatSetSizes(). It is the number of local unknowns plus the number of ghost nodes. The local sizes passed to MatSetSizes() is the size of the part of the vector that belongs to that processor. Barry On Mar 17, 2008, at 4:54 PM, Kathrin Burckhardt wrote: > Dear Lisandro, > >> I'm not completelly sure what are you actually trying to do. It seems >> you know how to actually compute the local Schur complement in a >> entry-by-entry basis. In that case, your way is semms to be OK. > > What do you mean with entry-by-entry basis? Anyway, I have the local > Schur complement explicitely, in form of a matrix (or C++ vector). > > >> However, you get a error because of a inner implementation detail of >> MATIS. Do de following: set the local-to-global mapping AFTER >> setting >> the matrix type. This way, your code should now work. > > Unfortunately, changing the order of the Mat-commands gives me the > same error. I guess I found the origin of the problem: > > I put it in the order below because I thought to avoid > > [2]PETSC ERROR: Sum of local lengths 3916 does not equal global > length 1958 > > which is due to the fact that in the case of 2 Subdomains and 2 > processors, e.g., it is NPb=NPb_tot in the Schur system, instead of > NPb=NPb_tot/2. However, I realize just now that the order of the Mat- > commands has no influence on this, and that the error mentioned > below has probably its origin there. > > I have chosen MATIS because in the manual page of MatCreateIS() it > is written "...m and n are NOT related to the size of the map", but > maybe I understand this setence wrong? How can I choose m,n and M,N > independently? > > > >> On 3/17/08, Kathrin Burckhardt wrote: >>> Dear all, >>> >>> My problem, once again and more concrete: >>> >>> I have given the local Schur matrices Si and I would like to set- >>> up the >>> global matrix S without assembling the Si on one processor. >>> >>> I thought "MATIS" is the suitable matrix type, but I don't succeed >>> in >>> using it. My code is: >>> >>> IS is; >>> ISCreateGeneral(PETSC_COMM_SELF, NPb, &uBId_global[0], &is); >>> ISLocalToGlobalMapping mapping; >>> ISLocalToGlobalMappingCreateIS(is, &mapping); >>> Mat Stot; >>> MatCreate(PETSC_COMM_WORLD,&Stot); >>> MatSetLocalToGlobalMapping(Stot, mapping); >>> MatSetSizes(Stot,NPb,NPb,NPb_tot,NPb_tot); >>> MatSetType(Stot,MATIS); >>> //or, alternatively >>> // >>> MatCreateIS(PETSC_COMM_WORLD,NPb,NPb,NPb_tot,NPb_tot,mapping,&Stot); >>> for(int i=0; i>> for(int j=0; j>> MatSetValuesLocal(Stot, 1, &i, 1, &j, &S(i+1,j+1), >>> ADD_VALUES); >>> >>> And I got the error >>> >>> [1]PETSC ERROR: MatSetValuesLocal() line 1471 in >>> src/mat/interface/matrix.c >>> [1]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [1]PETSC ERROR: Object is in wrong state! >>> [1]PETSC ERROR: Object Type not set: Argument # 1! >>> >>> Can you tell me what's wrong? >>> >>> >>> >> >> >> > From dalcinl at gmail.com Wed Mar 19 18:42:45 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 19 Mar 2008 20:42:45 -0300 Subject: still Schur complement In-Reply-To: References: Message-ID: OK, Have you considered using MATSHELL for this ?? On 3/17/08, Kathrin Burckhardt wrote: > Dear Lisandro, > > > > I'm not completelly sure what are you actually trying to do. It seems > > you know how to actually compute the local Schur complement in a > > entry-by-entry basis. In that case, your way is semms to be OK. > > > What do you mean with entry-by-entry basis? Anyway, I have the local > Schur complement explicitely, in form of a matrix (or C++ vector). > > > > > However, you get a error because of a inner implementation detail of > > MATIS. Do de following: set the local-to-global mapping AFTER setting > > the matrix type. This way, your code should now work. > > > Unfortunately, changing the order of the Mat-commands gives me the same > error. I guess I found the origin of the problem: > > I put it in the order below because I thought to avoid > > [2]PETSC ERROR: Sum of local lengths 3916 does not equal global length 1958 > > which is due to the fact that in the case of 2 Subdomains and 2 > processors, e.g., it is NPb=NPb_tot in the Schur system, instead of > NPb=NPb_tot/2. However, I realize just now that the order of the > Mat-commands has no influence on this, and that the error mentioned > below has probably its origin there. > > I have chosen MATIS because in the manual page of MatCreateIS() it is > written "...m and n are NOT related to the size of the map", but maybe I > understand this setence wrong? How can I choose m,n and M,N > independently? > > > > > > On 3/17/08, Kathrin Burckhardt wrote: > >> Dear all, > >> > >> My problem, once again and more concrete: > >> > >> I have given the local Schur matrices Si and I would like to set-up the > >> global matrix S without assembling the Si on one processor. > >> > >> I thought "MATIS" is the suitable matrix type, but I don't succeed in > >> using it. My code is: > >> > >> IS is; > >> ISCreateGeneral(PETSC_COMM_SELF, NPb, &uBId_global[0], &is); > >> ISLocalToGlobalMapping mapping; > >> ISLocalToGlobalMappingCreateIS(is, &mapping); > >> Mat Stot; > >> MatCreate(PETSC_COMM_WORLD,&Stot); > >> MatSetLocalToGlobalMapping(Stot, mapping); > >> MatSetSizes(Stot,NPb,NPb,NPb_tot,NPb_tot); > >> MatSetType(Stot,MATIS); > >> //or, alternatively > >> //MatCreateIS(PETSC_COMM_WORLD,NPb,NPb,NPb_tot,NPb_tot,mapping,&Stot); > >> for(int i=0; i >> for(int j=0; j >> MatSetValuesLocal(Stot, 1, &i, 1, &j, &S(i+1,j+1), ADD_VALUES); > >> > >> And I got the error > >> > >> [1]PETSC ERROR: MatSetValuesLocal() line 1471 in > >> src/mat/interface/matrix.c > >> [1]PETSC ERROR: --------------------- Error Message > >> ------------------------------------ > >> [1]PETSC ERROR: Object is in wrong state! > >> [1]PETSC ERROR: Object Type not set: Argument # 1! > >> > >> Can you tell me what's wrong? > >> > >> > >> > > > > > > > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From niko.karin at gmail.com Thu Mar 20 13:07:00 2008 From: niko.karin at gmail.com (Nicolas Tardieu) Date: Thu, 20 Mar 2008 14:07:00 -0400 Subject: Block-smoothers with variable block size Message-ID: Dear PETSc experts, I am currently designing a multigrid PC that needs to operate on variable block size matrices. I know PETSc does not support this feature. So, I will do it "by hand". Neverthless I would like to be able to use some smoothers like Jacobi or SOR in their block version. What is the best solution that would work both in sequential and parallel? - define some MATSHELL that handle variable block size -> will I be able to use in some way a block Jacobi or SOR? -> would it be possible to use CSR inside the MATSHELL and manage the variable blocks. - use a AIJ matrix and manage the blocks using some indirection mechanism and reprogram Jacobi or SOR Thank you, Nicolas -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Mar 20 13:17:24 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 20 Mar 2008 13:17:24 -0500 Subject: Block-smoothers with variable block size In-Reply-To: References: Message-ID: <5EDA97BA-3801-42F4-8062-6A6FB8ABAB0D@mcs.anl.gov> Nicolas, Actually we are very close to this. Check out the routine MatRelax_Inode() which is for AIJ matrices. For the block size it uses the inode size. If your matrix has inodes that match the blocks you want then everything is there already just use the PCSOR. If you do not have inodes then you can mimic the MatRelax_Inode() code to make a new routine MatPBRelax_SeqAIJ(). If you do implement this we'd love to add it to PETSc. Please let us know if you have additional questions. Barry On Mar 20, 2008, at 1:07 PM, Nicolas Tardieu wrote: > Dear PETSc experts, > > I am currently designing a multigrid PC that needs to operate on > variable block size matrices. I know PETSc does not support this > feature. > So, I will do it "by hand". Neverthless I would like to be able to > use some smoothers like Jacobi or SOR in their block version. > > What is the best solution that would work both in sequential and > parallel? > > - define some MATSHELL that handle variable block size > -> will I be able to use in some way a block Jacobi or SOR? > -> would it be possible to use CSR inside the MATSHELL and > manage the variable blocks. > > - use a AIJ matrix and manage the blocks using some indirection > mechanism and reprogram Jacobi or SOR > > > > Thank you, > > Nicolas > From niko.karin at gmail.com Thu Mar 20 13:52:01 2008 From: niko.karin at gmail.com (Nicolas Tardieu) Date: Thu, 20 Mar 2008 14:52:01 -0400 Subject: Block-smoothers with variable block size In-Reply-To: <5EDA97BA-3801-42F4-8062-6A6FB8ABAB0D@mcs.anl.gov> References: <5EDA97BA-3801-42F4-8062-6A6FB8ABAB0D@mcs.anl.gov> Message-ID: Dear Barry, Thank you very much for your quick answer. I am searching for MatRelax_Inode in PETSc src directory using grep. But I just can't find it! Please note I am using 2.3.3p6 Could you please tell me where this routine is? Nicoals 2008/3/20, Barry Smith : > > > Nicolas, > > Actually we are very close to this. Check out the routine > MatRelax_Inode() which is for AIJ matrices. > For the block size it uses the inode size. If your matrix has inodes > that match the blocks you want > then everything is there already just use the PCSOR. > > If you do not have inodes then you can mimic the MatRelax_Inode() > code to make a new > routine MatPBRelax_SeqAIJ(). If you do implement this we'd love to add > it to PETSc. > > > Please let us know if you have additional questions. > > > Barry > > > > > On Mar 20, 2008, at 1:07 PM, Nicolas Tardieu wrote: > > > Dear PETSc experts, > > > > I am currently designing a multigrid PC that needs to operate on > > variable block size matrices. I know PETSc does not support this > > feature. > > So, I will do it "by hand". Neverthless I would like to be able to > > use some smoothers like Jacobi or SOR in their block version. > > > > What is the best solution that would work both in sequential and > > parallel? > > > > - define some MATSHELL that handle variable block size > > -> will I be able to use in some way a block Jacobi or SOR? > > -> would it be possible to use CSR inside the MATSHELL and > > manage the variable blocks. > > > > - use a AIJ matrix and manage the blocks using some indirection > > mechanism and reprogram Jacobi or SOR > > > > > > > > Thank you, > > > > Nicolas > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Mar 20 14:59:17 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 20 Mar 2008 14:59:17 -0500 Subject: Block-smoothers with variable block size In-Reply-To: References: <5EDA97BA-3801-42F4-8062-6A6FB8ABAB0D@mcs.anl.gov> Message-ID: <705023EB-D326-47E0-AC7A-30578E9D75D9@mcs.anl.gov> DON"T USE GREP! Use etags # To access the tags in EMACS, type M-x visit-tags-table and specify # the file petsc/TAGS. # 1) To move to where a PETSc function is defined, enter M-. and the # function name. # 2) To search for a string and move to the first occurrence, # use M-x tags-search and the string. # To locate later occurrences, use M-, It may not be in petsc-2.3.3 you may need to use petsc-dev http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html it is in src/mat/impls/aij/seq/inode.c Barry On Mar 20, 2008, at 1:52 PM, Nicolas Tardieu wrote: > Dear Barry, > > Thank you very much for your quick answer. > I am searching for MatRelax_Inode in PETSc src directory using grep. > But I just can't find it! Please note I am using 2.3.3p6 > Could you please tell me where this routine is? > > Nicoals > > 2008/3/20, Barry Smith : > Nicolas, > > Actually we are very close to this. Check out the routine > MatRelax_Inode() which is for AIJ matrices. > For the block size it uses the inode size. If your matrix has inodes > that match the blocks you want > then everything is there already just use the PCSOR. > > If you do not have inodes then you can mimic the MatRelax_Inode() > code to make a new > routine MatPBRelax_SeqAIJ(). If you do implement this we'd love to add > it to PETSc. > > > Please let us know if you have additional questions. > > > Barry > > > > > On Mar 20, 2008, at 1:07 PM, Nicolas Tardieu wrote: > > > Dear PETSc experts, > > > > I am currently designing a multigrid PC that needs to operate on > > variable block size matrices. I know PETSc does not support this > > feature. > > So, I will do it "by hand". Neverthless I would like to be able to > > use some smoothers like Jacobi or SOR in their block version. > > > > What is the best solution that would work both in sequential and > > parallel? > > > > - define some MATSHELL that handle variable block size > > -> will I be able to use in some way a block Jacobi or SOR? > > -> would it be possible to use CSR inside the MATSHELL and > > manage the variable blocks. > > > > - use a AIJ matrix and manage the blocks using some indirection > > mechanism and reprogram Jacobi or SOR > > > > > > > > Thank you, > > > > Nicolas > > > > From niko.karin at gmail.com Fri Mar 21 08:03:43 2008 From: niko.karin at gmail.com (Nicolas Tardieu) Date: Fri, 21 Mar 2008 09:03:43 -0400 Subject: Block-smoothers with variable block size In-Reply-To: <705023EB-D326-47E0-AC7A-30578E9D75D9@mcs.anl.gov> References: <5EDA97BA-3801-42F4-8062-6A6FB8ABAB0D@mcs.anl.gov> <705023EB-D326-47E0-AC7A-30578E9D75D9@mcs.anl.gov> Message-ID: Dear Barry, I have a few questions about inodes : - can the user specify the inodes sizes (it seems to me this is an automatic PETSc feature to gain performance, right?)? - can the inodes have different sizes? Otherwise I plan to define a new matrix type MATSEQVBAIJ (VB for variable block) based on MATSEQAIJ with an additional attribute defining the block sizes. Thanks, Nicolas 2008/3/20, Barry Smith : > > > DON"T USE GREP! Use etags > # To access the tags in EMACS, type M-x visit-tags-table and specify > # the file petsc/TAGS. > # 1) To move to where a PETSc function is defined, enter M-. and the > # function name. > # 2) To search for a string and move to the first occurrence, > # use M-x tags-search and the string. > # To locate later occurrences, use M-, > > It may not be in petsc-2.3.3 you may need to use petsc-dev > http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html > > it is in src/mat/impls/aij/seq/inode.c > > > > Barry > > > > On Mar 20, 2008, at 1:52 PM, Nicolas Tardieu wrote: > > > Dear Barry, > > > > Thank you very much for your quick answer. > > I am searching for MatRelax_Inode in PETSc src directory using grep. > > But I just can't find it! Please note I am using 2.3.3p6 > > Could you please tell me where this routine is? > > > > Nicoals > > > > 2008/3/20, Barry Smith : > > Nicolas, > > > > Actually we are very close to this. Check out the routine > > MatRelax_Inode() which is for AIJ matrices. > > For the block size it uses the inode size. If your matrix has inodes > > that match the blocks you want > > then everything is there already just use the PCSOR. > > > > If you do not have inodes then you can mimic the MatRelax_Inode() > > code to make a new > > routine MatPBRelax_SeqAIJ(). If you do implement this we'd love to add > > it to PETSc. > > > > > > Please let us know if you have additional questions. > > > > > > Barry > > > > > > > > > > On Mar 20, 2008, at 1:07 PM, Nicolas Tardieu wrote: > > > > > Dear PETSc experts, > > > > > > I am currently designing a multigrid PC that needs to operate on > > > variable block size matrices. I know PETSc does not support this > > > feature. > > > So, I will do it "by hand". Neverthless I would like to be able to > > > use some smoothers like Jacobi or SOR in their block version. > > > > > > What is the best solution that would work both in sequential and > > > parallel? > > > > > > - define some MATSHELL that handle variable block size > > > -> will I be able to use in some way a block Jacobi or SOR? > > > -> would it be possible to use CSR inside the MATSHELL and > > > manage the variable blocks. > > > > > > - use a AIJ matrix and manage the blocks using some indirection > > > mechanism and reprogram Jacobi or SOR > > > > > > > > > > > > Thank you, > > > > > > Nicolas > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Mar 21 09:47:32 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 21 Mar 2008 09:47:32 -0500 (CDT) Subject: Block-smoothers with variable block size In-Reply-To: References: <5EDA97BA-3801-42F4-8062-6A6FB8ABAB0D@mcs.anl.gov> <705023EB-D326-47E0-AC7A-30578E9D75D9@mcs.anl.gov> Message-ID: On Fri, 21 Mar 2008, Nicolas Tardieu wrote: > Dear Barry, > > I have a few questions about inodes : > - can the user specify the inodes sizes (it seems to me this is an automatic > PETSc feature to gain performance, right?)? Inode code checks the assembled matrix for consequitive rows with 'same column' values to determine the inode sizes. So this is automatically done - and the user does not specify it. > - can the inodes have different sizes? Yes, you can have the first 3 rows with the same column entries, the next 4 rows with the same column entries, and the next 2 rows etc.. In this case - the inode structure would be [3,4,2] for a matrix of size 9-rows. Satish > > Otherwise I plan to define a new matrix type MATSEQVBAIJ (VB for variable > block) based on MATSEQAIJ with an additional attribute defining the block > sizes. > > Thanks, > > Nicolas > > 2008/3/20, Barry Smith : > > > > > > DON"T USE GREP! Use etags > > # To access the tags in EMACS, type M-x visit-tags-table and specify > > # the file petsc/TAGS. > > # 1) To move to where a PETSc function is defined, enter M-. and the > > # function name. > > # 2) To search for a string and move to the first occurrence, > > # use M-x tags-search and the string. > > # To locate later occurrences, use M-, > > > > It may not be in petsc-2.3.3 you may need to use petsc-dev > > http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html > > > > it is in src/mat/impls/aij/seq/inode.c > > > > > > > > Barry > > > > > > > > On Mar 20, 2008, at 1:52 PM, Nicolas Tardieu wrote: > > > > > Dear Barry, > > > > > > Thank you very much for your quick answer. > > > I am searching for MatRelax_Inode in PETSc src directory using grep. > > > But I just can't find it! Please note I am using 2.3.3p6 > > > Could you please tell me where this routine is? > > > > > > Nicoals > > > > > > 2008/3/20, Barry Smith : > > > Nicolas, > > > > > > Actually we are very close to this. Check out the routine > > > MatRelax_Inode() which is for AIJ matrices. > > > For the block size it uses the inode size. If your matrix has inodes > > > that match the blocks you want > > > then everything is there already just use the PCSOR. > > > > > > If you do not have inodes then you can mimic the MatRelax_Inode() > > > code to make a new > > > routine MatPBRelax_SeqAIJ(). If you do implement this we'd love to add > > > it to PETSc. > > > > > > > > > Please let us know if you have additional questions. > > > > > > > > > Barry > > > > > > > > > > > > > > > On Mar 20, 2008, at 1:07 PM, Nicolas Tardieu wrote: > > > > > > > Dear PETSc experts, > > > > > > > > I am currently designing a multigrid PC that needs to operate on > > > > variable block size matrices. I know PETSc does not support this > > > > feature. > > > > So, I will do it "by hand". Neverthless I would like to be able to > > > > use some smoothers like Jacobi or SOR in their block version. > > > > > > > > What is the best solution that would work both in sequential and > > > > parallel? > > > > > > > > - define some MATSHELL that handle variable block size > > > > -> will I be able to use in some way a block Jacobi or SOR? > > > > -> would it be possible to use CSR inside the MATSHELL and > > > > manage the variable blocks. > > > > > > > > - use a AIJ matrix and manage the blocks using some indirection > > > > mechanism and reprogram Jacobi or SOR > > > > > > > > > > > > > > > > Thank you, > > > > > > > > Nicolas > > > > > > > > > > > > > > > From bsmith at mcs.anl.gov Fri Mar 21 12:13:48 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 21 Mar 2008 12:13:48 -0500 Subject: Block-smoothers with variable block size In-Reply-To: References: <5EDA97BA-3801-42F4-8062-6A6FB8ABAB0D@mcs.anl.gov> <705023EB-D326-47E0-AC7A-30578E9D75D9@mcs.anl.gov> Message-ID: <0CB5111F-7D43-4B33-9A51-F0FB9C34776D@mcs.anl.gov> On Mar 21, 2008, at 8:03 AM, Nicolas Tardieu wrote: > Dear Barry, > > I have a few questions about inodes : > - can the user specify the inodes sizes (it seems to me this is an > automatic PETSc feature to gain performance, right?)? No, they must be inodes, they are not arbitary > > - can the inodes have different sizes? Yes, they are what ever they are. > > > Otherwise I plan to define a new matrix type MATSEQVBAIJ (VB for > variable block) based on MATSEQAIJ with an additional attribute > defining the block sizes. This is overkill, you will end up doing a lot more work then you need to! The locations for storing the diagonal blocks is already in the MATSEQAIJ data structures, if you write code to replace the MatRelax_Inode (that uses that same diagonal space) you are done. Barry > > > Thanks, > > Nicolas > > 2008/3/20, Barry Smith : > DON"T USE GREP! Use etags > # To access the tags in EMACS, type M-x visit-tags-table and specify > # the file petsc/TAGS. > # 1) To move to where a PETSc function is defined, enter M-. and the > # function name. > # 2) To search for a string and move to the first occurrence, > # use M-x tags-search and the string. > # To locate later occurrences, use M-, > > It may not be in petsc-2.3.3 you may need to use petsc-dev > http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html > > it is in src/mat/impls/aij/seq/inode.c > > > > Barry > > > > On Mar 20, 2008, at 1:52 PM, Nicolas Tardieu wrote: > > > Dear Barry, > > > > Thank you very much for your quick answer. > > I am searching for MatRelax_Inode in PETSc src directory using grep. > > But I just can't find it! Please note I am using 2.3.3p6 > > Could you please tell me where this routine is? > > > > Nicoals > > > > 2008/3/20, Barry Smith : > > Nicolas, > > > > Actually we are very close to this. Check out the routine > > MatRelax_Inode() which is for AIJ matrices. > > For the block size it uses the inode size. If your matrix has inodes > > that match the blocks you want > > then everything is there already just use the PCSOR. > > > > If you do not have inodes then you can mimic the > MatRelax_Inode() > > code to make a new > > routine MatPBRelax_SeqAIJ(). If you do implement this we'd love to > add > > it to PETSc. > > > > > > Please let us know if you have additional questions. > > > > > > Barry > > > > > > > > > > On Mar 20, 2008, at 1:07 PM, Nicolas Tardieu wrote: > > > > > Dear PETSc experts, > > > > > > I am currently designing a multigrid PC that needs to operate on > > > variable block size matrices. I know PETSc does not support this > > > feature. > > > So, I will do it "by hand". Neverthless I would like to be able > to > > > use some smoothers like Jacobi or SOR in their block version. > > > > > > What is the best solution that would work both in sequential and > > > parallel? > > > > > > - define some MATSHELL that handle variable block size > > > -> will I be able to use in some way a block Jacobi or SOR? > > > -> would it be possible to use CSR inside the MATSHELL and > > > manage the variable blocks. > > > > > > - use a AIJ matrix and manage the blocks using some indirection > > > mechanism and reprogram Jacobi or SOR > > > > > > > > > > > > Thank you, > > > > > > Nicolas > > > > > > > > > From david.knezevic at balliol.ox.ac.uk Fri Mar 21 10:50:32 2008 From: david.knezevic at balliol.ox.ac.uk (David Knezevic) Date: Fri, 21 Mar 2008 15:50:32 +0000 Subject: Using SUBSET_NONZERO_PATTERN with MatAXPY Message-ID: <47E3D948.8080403@balliol.ox.ac.uk> Hello, I've got a SeqBAIJ matrix A which I am reassembling at each time-step in order to solve a time-dependent PDE. I initially set up the sparsity pattern of A, and then assemble A at each time step by doing operations like the following: ierr = MatZeroEntries(A);CHKERRQ(ierr); ierr = MatAXPY(A,const1,B1,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); ierr = MatAXPY(A,const2,B2,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); This works fine. However, the sparsity patterns of B1, B2 etc should all be subsets of the sparsity pattern of A (which should be retained by MatZeroEntries). But changing DIFFERENT_NONZERO_PATTERN to SUBSET_NONZERO_PATTERN in the above code causes an error. In particular, the code runs fine for the first time step, but on the second time step I get the error: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Invalid argument! [0]PETSC ERROR: Wrong type of object: Parameter # 3! ... [0]PETSC ERROR: MatAXPY() line 34 in src/mat/utils/axpy.c I found a description of the same problem here: http://ganymed.iwr.uni-heidelberg.de/pipermail/dealii/2007/002066.html but I haven't found an answer. Assistance would be most appreciated. Best regards, David From niko.karin at gmail.com Fri Mar 21 15:39:12 2008 From: niko.karin at gmail.com (Nicolas Tardieu) Date: Fri, 21 Mar 2008 16:39:12 -0400 Subject: Block-smoothers with variable block size In-Reply-To: <0CB5111F-7D43-4B33-9A51-F0FB9C34776D@mcs.anl.gov> References: <5EDA97BA-3801-42F4-8062-6A6FB8ABAB0D@mcs.anl.gov> <705023EB-D326-47E0-AC7A-30578E9D75D9@mcs.anl.gov> <0CB5111F-7D43-4B33-9A51-F0FB9C34776D@mcs.anl.gov> Message-ID: Barry, Satish, Thank you very much for these precious informations! Nicolas 2008/3/21, Barry Smith : > > > On Mar 21, 2008, at 8:03 AM, Nicolas Tardieu wrote: > > > Dear Barry, > > > > > I have a few questions about inodes : > > - can the user specify the inodes sizes (it seems to me this is an > > automatic PETSc feature to gain performance, right?)? > > > No, they must be inodes, they are not arbitary > > > > > - can the inodes have different sizes? > > > Yes, they are what ever they are. > > > > > > > Otherwise I plan to define a new matrix type MATSEQVBAIJ (VB for > > variable block) based on MATSEQAIJ with an additional attribute > > defining the block sizes. > > > This is overkill, you will end up doing a lot more work then you > need to! > The locations for storing the diagonal blocks is already in the > MATSEQAIJ data structures, if you > write code to replace the MatRelax_Inode (that uses that same diagonal > space) you are done. > > > Barry > > > > > > > > > Thanks, > > > > Nicolas > > > > 2008/3/20, Barry Smith : > > DON"T USE GREP! Use etags > > # To access the tags in EMACS, type M-x visit-tags-table and specify > > # the file petsc/TAGS. > > # 1) To move to where a PETSc function is defined, enter M-. and the > > # function name. > > # 2) To search for a string and move to the first occurrence, > > # use M-x tags-search and the string. > > # To locate later occurrences, use M-, > > > > It may not be in petsc-2.3.3 you may need to use petsc-dev > > http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html > > > > it is in src/mat/impls/aij/seq/inode.c > > > > > > > > Barry > > > > > > > > On Mar 20, 2008, at 1:52 PM, Nicolas Tardieu wrote: > > > > > Dear Barry, > > > > > > Thank you very much for your quick answer. > > > I am searching for MatRelax_Inode in PETSc src directory using grep. > > > But I just can't find it! Please note I am using 2.3.3p6 > > > Could you please tell me where this routine is? > > > > > > Nicoals > > > > > > 2008/3/20, Barry Smith : > > > Nicolas, > > > > > > Actually we are very close to this. Check out the routine > > > MatRelax_Inode() which is for AIJ matrices. > > > For the block size it uses the inode size. If your matrix has inodes > > > that match the blocks you want > > > then everything is there already just use the PCSOR. > > > > > > If you do not have inodes then you can mimic the > > MatRelax_Inode() > > > code to make a new > > > routine MatPBRelax_SeqAIJ(). If you do implement this we'd love to > > add > > > it to PETSc. > > > > > > > > > Please let us know if you have additional questions. > > > > > > > > > Barry > > > > > > > > > > > > > > > On Mar 20, 2008, at 1:07 PM, Nicolas Tardieu wrote: > > > > > > > Dear PETSc experts, > > > > > > > > I am currently designing a multigrid PC that needs to operate on > > > > variable block size matrices. I know PETSc does not support this > > > > feature. > > > > So, I will do it "by hand". Neverthless I would like to be able > > to > > > > use some smoothers like Jacobi or SOR in their block version. > > > > > > > > What is the best solution that would work both in sequential and > > > > parallel? > > > > > > > > - define some MATSHELL that handle variable block size > > > > -> will I be able to use in some way a block Jacobi or SOR? > > > > -> would it be possible to use CSR inside the MATSHELL and > > > > manage the variable blocks. > > > > > > > > - use a AIJ matrix and manage the blocks using some indirection > > > > mechanism and reprogram Jacobi or SOR > > > > > > > > > > > > > > > > Thank you, > > > > > > > > Nicolas > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Mar 21 20:17:46 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 21 Mar 2008 20:17:46 -0500 Subject: Using SUBSET_NONZERO_PATTERN with MatAXPY In-Reply-To: <47E3D948.8080403@balliol.ox.ac.uk> References: <47E3D948.8080403@balliol.ox.ac.uk> Message-ID: David, Thank you reporting the problem. It is a bug for the BAIJ format. Satish, Could you please make a patch for 2.3.3 In the file src/mat/impls/baij/seq/baij.c in the function MatAXPY_SeqBAIJ() locate PetscErrorCode MatAXPY_SeqBAIJ(Mat Y,PetscScalar a,Mat X,MatStructure str) { Mat_SeqBAIJ *x = (Mat_SeqBAIJ *)X->data,*y = (Mat_SeqBAIJ *)Y- >data; PetscErrorCode ierr; PetscInt i,bs=Y->rmap.bs,j,bs2; PetscBLASInt one=1,bnz = PetscBLASIntCast(x->nz); PetscFunctionBegin; if (str == SAME_NONZERO_PATTERN) { PetscScalar alpha = a; BLASaxpy_(&bnz,&alpha,x->a,&one,y->a,&one); } else if (str == SUBSET_NONZERO_PATTERN) { /* nonzeros of X is a subset of Y's */ if (y->xtoy && y->XtoY != X) { ierr = PetscFree(y->xtoy);CHKERRQ(ierr); ierr = MatDestroy(y->XtoY);CHKERRQ(ierr); } if (!y->xtoy) { /* get xtoy */ ierr = MatAXPYGetxtoy_Private(x->mbs,x->i,x->j,PETSC_NULL, y- >i,y->j,PETSC_NULL, &y->xtoy);CHKERRQ(ierr); y->XtoY = X; ++++++++++++++++++++++++add the next line ierr = PetscObjectReference((PetscObject)X);CHKERRQ(ierr); ++++++++++++++++++++++++++++ recompile in that directory with "make lib shared" Please let us know if you continue to have trouble at petsc-maint at mcs.anl.gov Barry On Mar 21, 2008, at 10:50 AM, David Knezevic wrote: > Hello, > > I've got a SeqBAIJ matrix A which I am reassembling at each time- > step in order to solve a time-dependent PDE. I initially set up the > sparsity pattern of A, and then assemble A at each time step by > doing operations like the following: > > ierr = MatZeroEntries(A);CHKERRQ(ierr); > ierr = MatAXPY(A,const1,B1,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); > ierr = MatAXPY(A,const2,B2,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); > > This works fine. However, the sparsity patterns of B1, B2 etc should > all be subsets of the sparsity pattern of A (which should be > retained by MatZeroEntries). But changing DIFFERENT_NONZERO_PATTERN > to SUBSET_NONZERO_PATTERN in the above code causes an error. In > particular, the code runs fine for the first time step, but on the > second time step I get the error: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Invalid argument! > [0]PETSC ERROR: Wrong type of object: Parameter # 3! > ... > [0]PETSC ERROR: MatAXPY() line 34 in src/mat/utils/axpy.c > > I found a description of the same problem here: > http://ganymed.iwr.uni-heidelberg.de/pipermail/dealii/2007/002066.html > but I haven't found an answer. Assistance would be most appreciated. > > Best regards, > David > > From geenen at gmail.com Sun Mar 23 02:46:43 2008 From: geenen at gmail.com (Thomas Geenen) Date: Sun, 23 Mar 2008 08:46:43 +0100 Subject: ML's multigrid Message-ID: <200803230846.43732.geenen@gmail.com> dear Petsc users, I am using the AMG from ML as a preconditioner to CG. I am looking at the scaling characteristics of different parts of the preconditioner (some rough timings). To this end I have put some timing calls in ml.c. now when i run the code and the solver is called for the first time i can see that indeed the ML routines are called in particular PCSetUp_ML. In this routine a considerable amount of time is spend in the aggregation and interpolation operator construction (as expected). However the second time the solver is called this routine is no longer called. This seems to be inconsistent with the concept of AMG where the coarse grids and interpolation operators have to be constructed again when the matrix has changed (due to changing material properties as a function of time for instance). Destroying the preconditioner after each solve seems a waste of resources especially since in PCSetUp_ML , there are options to reuse datastructures. pc-setupcalled, reuse etc. are the aggregation and interpolation comstruction routines called from a different part in the code? ML_Gen_MGHierarchy_UsingAggregation(ml_object,0,ML_INCREASING,agg_object); seems to be called only from PCSetUp_ML. cheers Thomas From zonexo at gmail.com Sun Mar 23 07:16:18 2008 From: zonexo at gmail.com (Ben Tay) Date: Sun, 23 Mar 2008 20:16:18 +0800 Subject: Linking problems with ifort and visual studio 2003 Message-ID: <47E64A12.4060901@gmail.com> Hi, I've successfully compiled the library of PETSc. The examples also worked (I tested ex1f). However in the IDE of visual studio 2003, I can't seem to link, although compiling the individual files were fine. The errors are: global.obj error LNK2019: unresolved external symbol _PETSCINITIALIZE at 12 referenced in function _GLOBAL_DATA_mp_ALLO_VAR global.obj error LNK2019: unresolved external symbol _MATCREATESEQAIJ at 28 referenced in function _GLOBAL_DATA_mp_ALLO_VAR global.obj error LNK2019: unresolved external symbol _VECCREATESEQ at 16 referenced in function _GLOBAL_DATA_mp_ALLO_VAR global.obj error LNK2019: unresolved external symbol _VECDUPLICATE at 12 referenced in function _GLOBAL_DATA_mp_ALLO_VAR global.obj error LNK2019: unresolved external symbol _KSPCREATE at 12 referenced in function _GLOBAL_DATA_mp_ALLO_VAR global.obj error LNK2019: unresolved external symbol _VECASSEMBLYBEGIN at 8 referenced in function _GLOBAL_DATA_mp_ALLO_VAR global.obj error LNK2019: unresolved external symbol _VECASSEMBLYEND at 8 referenced in function _GLOBAL_DATA_mp_ALLO_VAR bc.obj error LNK2019: unresolved external symbol _MATSETVALUES at 32 referenced in function _BC_mp_BC_BIG_A_S bc_impl.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 set_matrix.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 petsc_sub.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 bc_impl.obj error LNK2019: unresolved external symbol _VECSETVALUE at 20 referenced in function _BC_IMPL_mp_BC_Q_IMPL_S set_matrix.obj error LNK2001: unresolved external symbol _VECSETVALUE at 20 petsc_sub.obj error LNK2001: unresolved external symbol _VECSETVALUE at 20 petsc_sub.obj error LNK2019: unresolved external symbol _MATASSEMBLYBEGIN at 12 referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM petsc_sub.obj error LNK2019: unresolved external symbol _MATASSEMBLYEND at 12 referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM petsc_sub.obj error LNK2019: unresolved external symbol _MATSCALE at 12 referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM petsc_sub.obj error LNK2019: unresolved external symbol _KSPSETOPERATORS at 20 referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM petsc_sub.obj error LNK2019: unresolved external symbol _KSPGETPC at 12 referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM petsc_sub.obj error LNK2019: unresolved external symbol _KSPSETTYPE at 16 referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM petsc_sub.obj error LNK2019: unresolved external symbol _PCSETTYPE at 16 referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM I was using Compaq visual fortran initially and I have converted the *.dsw file to the visual studio 2003 format. Of cos, I have changed the locations to the new PETSc directory. May I know what's happening? Is there anything I need to set or changed in studio 2003? Thanks alot! From balay at mcs.anl.gov Sun Mar 23 09:11:54 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 23 Mar 2008 09:11:54 -0500 (CDT) Subject: Linking problems with ifort and visual studio 2003 In-Reply-To: <47E64A12.4060901@gmail.com> References: <47E64A12.4060901@gmail.com> Message-ID: Ok - you are using VisualStudio 2003 + IntelFortran compilers to build PETSc. A PETSc fortran example [ex1f] compiles from 'make' but not from IDE? Then some setting in the IDE does not match the compile option. Per installation instructions - you should make sure you have *all* compiler options you see from make [when compiling ex1f] are also specified in the IDE settings. > global.obj error LNK2019: unresolved external symbol _PETSCINITIALIZE at 12 > referenced in function _GLOBAL_DATA_mp_ALLO_VAR Looks like this code is compiled by Compaq Fortran compiler - and not intel fortran. Can you verify? Satish On Sun, 23 Mar 2008, Ben Tay wrote: > Hi, > > I've successfully compiled the library of PETSc. The examples also worked (I > tested ex1f). However in the IDE of visual studio 2003, I can't seem to link, > although compiling the individual files were fine. The errors are: > > global.obj error LNK2019: unresolved external symbol _PETSCINITIALIZE at 12 > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > global.obj error LNK2019: unresolved external symbol _MATCREATESEQAIJ at 28 > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > global.obj error LNK2019: unresolved external symbol _VECCREATESEQ at 16 > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > global.obj error LNK2019: unresolved external symbol _VECDUPLICATE at 12 > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > global.obj error LNK2019: unresolved external symbol _KSPCREATE at 12 referenced > in function _GLOBAL_DATA_mp_ALLO_VAR > global.obj error LNK2019: unresolved external symbol _VECASSEMBLYBEGIN at 8 > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > global.obj error LNK2019: unresolved external symbol _VECASSEMBLYEND at 8 > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > bc.obj error LNK2019: unresolved external symbol _MATSETVALUES at 32 referenced > in function _BC_mp_BC_BIG_A_S > bc_impl.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 > set_matrix.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 > petsc_sub.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 > bc_impl.obj error LNK2019: unresolved external symbol _VECSETVALUE at 20 > referenced in function _BC_IMPL_mp_BC_Q_IMPL_S > set_matrix.obj error LNK2001: unresolved external symbol _VECSETVALUE at 20 > petsc_sub.obj error LNK2001: unresolved external symbol _VECSETVALUE at 20 > petsc_sub.obj error LNK2019: unresolved external symbol _MATASSEMBLYBEGIN at 12 > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > petsc_sub.obj error LNK2019: unresolved external symbol _MATASSEMBLYEND at 12 > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > petsc_sub.obj error LNK2019: unresolved external symbol _MATSCALE at 12 > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > petsc_sub.obj error LNK2019: unresolved external symbol _KSPSETOPERATORS at 20 > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > petsc_sub.obj error LNK2019: unresolved external symbol _KSPGETPC at 12 > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > petsc_sub.obj error LNK2019: unresolved external symbol _KSPSETTYPE at 16 > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > petsc_sub.obj error LNK2019: unresolved external symbol _PCSETTYPE at 16 > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > > > I was using Compaq visual fortran initially and I have converted the *.dsw > file to the visual studio 2003 format. Of cos, I have changed the locations to > the new PETSc directory. May I know what's happening? Is there anything I need > to set or changed in studio 2003? > > Thanks alot! > > From bsmith at mcs.anl.gov Sun Mar 23 10:58:49 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 23 Mar 2008 10:58:49 -0500 Subject: ML's multigrid In-Reply-To: <200803230846.43732.geenen@gmail.com> References: <200803230846.43732.geenen@gmail.com> Message-ID: <019E211B-B67E-4749-BE72-CC50D95A573F@mcs.anl.gov> Thomas, It is possible there is an error in our PCSetUp_ML() but more likely there is a misunderstanding. If you call KSPSolve() repeatedly without calling KSPSetOperators() again then it uses the same matrix and preconditioner as for the previous solve (that is KSP does not know that you changed entries in the matrix). If you change the matrix you must call KSPSetOperators() again to get the new preconditioner being built before calling KSPSolve() again. Barry On Mar 23, 2008, at 2:46 AM, Thomas Geenen wrote: > dear Petsc users, > > I am using the AMG from ML as a preconditioner to CG. I am looking > at the > scaling characteristics of different parts of the preconditioner > (some rough > timings). To this end I have put some timing calls in ml.c. > now when i run the code and the solver is called for the first time > i can see > that indeed the ML routines are called in particular PCSetUp_ML. > In this routine a considerable amount of time is spend in the > aggregation and > interpolation operator construction (as expected). However the > second time > the solver is called this routine is no longer called. This seems to > be > inconsistent with the concept of AMG where the coarse grids and > interpolation > operators have to be constructed again when the matrix has changed > (due to > changing material properties as a function of time for instance). > Destroying the preconditioner after each solve seems a waste of > resources > especially since in PCSetUp_ML , there are options to reuse > datastructures. > pc-setupcalled, reuse etc. > > are the aggregation and interpolation comstruction routines called > from a > different part in the code? > ML_Gen_MGHierarchy_UsingAggregation(ml_object, > 0,ML_INCREASING,agg_object); > seems to be called only from PCSetUp_ML. > > cheers > Thomas > From geenen at gmail.com Sun Mar 23 11:08:44 2008 From: geenen at gmail.com (Thomas Geenen) Date: Sun, 23 Mar 2008 17:08:44 +0100 Subject: ML's multigrid In-Reply-To: <019E211B-B67E-4749-BE72-CC50D95A573F@mcs.anl.gov> References: <200803230846.43732.geenen@gmail.com> <019E211B-B67E-4749-BE72-CC50D95A573F@mcs.anl.gov> Message-ID: <200803231708.44609.geenen@gmail.com> thats right. i must have overseen it since i used it before for ILU /cg type combinations and mumps. thanks Thomas On Sunday 23 March 2008 16:58, Barry Smith wrote: > Thomas, > > It is possible there is an error in our PCSetUp_ML() but more > likely > there is a misunderstanding. If you call KSPSolve() repeatedly without > calling KSPSetOperators() again then it uses the same matrix and > preconditioner > as for the previous solve (that is KSP does not know that you changed > entries in the matrix). If you change the matrix you must call > KSPSetOperators() > again to get the new preconditioner being built before calling > KSPSolve() > again. > > Barry > > On Mar 23, 2008, at 2:46 AM, Thomas Geenen wrote: > > dear Petsc users, > > > > I am using the AMG from ML as a preconditioner to CG. I am looking > > at the > > scaling characteristics of different parts of the preconditioner > > (some rough > > timings). To this end I have put some timing calls in ml.c. > > now when i run the code and the solver is called for the first time > > i can see > > that indeed the ML routines are called in particular PCSetUp_ML. > > In this routine a considerable amount of time is spend in the > > aggregation and > > interpolation operator construction (as expected). However the > > second time > > the solver is called this routine is no longer called. This seems to > > be > > inconsistent with the concept of AMG where the coarse grids and > > interpolation > > operators have to be constructed again when the matrix has changed > > (due to > > changing material properties as a function of time for instance). > > Destroying the preconditioner after each solve seems a waste of > > resources > > especially since in PCSetUp_ML , there are options to reuse > > datastructures. > > pc-setupcalled, reuse etc. > > > > are the aggregation and interpolation comstruction routines called > > from a > > different part in the code? > > ML_Gen_MGHierarchy_UsingAggregation(ml_object, > > 0,ML_INCREASING,agg_object); > > seems to be called only from PCSetUp_ML. > > > > cheers > > Thomas From vijay.m at gmail.com Sun Mar 23 14:59:32 2008 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Sun, 23 Mar 2008 14:59:32 -0500 Subject: Creating a new vector based on a already distributed set of Vecs In-Reply-To: <200803230846.43732.geenen@gmail.com> Message-ID: <000601c88d20$6b353c60$383010ac@neutron> Hi all, I am sure the subject line wasn?t too explanatory and so here goes. I have a system which already has a Vec object created and distributed on many processors based on domain decomposition. I could have several such systems, each with a solution vector and a residual vector pertaining to its physics. Now, can I create a global vector which would just be pointers to the already existing vectors but from PETSc's point of view, appears to be a globally valid, (and possibly weirdly) distributed vector. Such an option will save the memory needed for the global vector and eliminates errors as to synchronization of the solution, residuals for the systems. I was reading the documentation and found the PetscMap data structure. But I am not entirely sure if this is what I am looking for. I hope that makes sense. I would appreciate any help you can provide to point me in the right direction. Cheers, Vijay No virus found in this outgoing message. Checked by AVG. Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: 3/22/2008 4:43 PM From knepley at gmail.com Sun Mar 23 15:12:33 2008 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 23 Mar 2008 15:12:33 -0500 Subject: Creating a new vector based on a already distributed set of Vecs In-Reply-To: <000601c88d20$6b353c60$383010ac@neutron> References: <200803230846.43732.geenen@gmail.com> <000601c88d20$6b353c60$383010ac@neutron> Message-ID: 2008/3/23 Vijay S. Mahadevan : > Hi all, > > I am sure the subject line wasn't too explanatory and so here goes. > > I have a system which already has a Vec object created and distributed on > many processors based on domain decomposition. I could have several such > systems, each with a solution vector and a residual vector pertaining to its > physics. > > Now, can I create a global vector which would just be pointers to the > already existing vectors but from PETSc's point of view, appears to be a > globally valid, (and possibly weirdly) distributed vector. Such an option > will save the memory needed for the global vector and eliminates errors as > to synchronization of the solution, residuals for the systems. I do not understand what you want. If you want a vector with the same layout, use VecDuplicate(). If you want the same data, just use the Vec itself. Thanks, Matt > I was reading the documentation and found the PetscMap data structure. But I > am not entirely sure if this is what I am looking for. > > I hope that makes sense. I would appreciate any help you can provide to > point me in the right direction. > > Cheers, > Vijay > > No virus found in this outgoing message. > Checked by AVG. > Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: 3/22/2008 > 4:43 PM > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From vijay.m at gmail.com Sun Mar 23 15:20:59 2008 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Sun, 23 Mar 2008 15:20:59 -0500 Subject: Creating a new vector based on a already distributed set of Vecs In-Reply-To: Message-ID: <000f01c88d23$69daae10$383010ac@neutron> There is more than 1 vector involved, with different data and layout dependent on the physics and mesh. Since the different Vec objects in systems have their own layout, I do not want to modify them or recreate them. I just want to create an interface vector so that I can pass it on to SNESSolve without having the hassle of synchronizing the individual vectors in each system. I am writing a multi-physics code and this global vector concept could be applied to the global nonlinear residual and the global solution vector. It doesn't make sense to recreate an entirely new vector and synchronizing these at every call to the residual. And there lies my motivation. But if you see an alternative way to approach this, I would be glad to hear it ! Sorry if I wasn?t too clear before. -----Original Message----- From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Sunday, March 23, 2008 3:13 PM To: petsc-users at mcs.anl.gov Subject: Re: Creating a new vector based on a already distributed set of Vecs 2008/3/23 Vijay S. Mahadevan : > Hi all, > > I am sure the subject line wasn't too explanatory and so here goes. > > I have a system which already has a Vec object created and distributed on > many processors based on domain decomposition. I could have several such > systems, each with a solution vector and a residual vector pertaining to its > physics. > > Now, can I create a global vector which would just be pointers to the > already existing vectors but from PETSc's point of view, appears to be a > globally valid, (and possibly weirdly) distributed vector. Such an option > will save the memory needed for the global vector and eliminates errors as > to synchronization of the solution, residuals for the systems. I do not understand what you want. If you want a vector with the same layout, use VecDuplicate(). If you want the same data, just use the Vec itself. Thanks, Matt > I was reading the documentation and found the PetscMap data structure. But I > am not entirely sure if this is what I am looking for. > > I hope that makes sense. I would appreciate any help you can provide to > point me in the right direction. > > Cheers, > Vijay > > No virus found in this outgoing message. > Checked by AVG. > Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: 3/22/2008 > 4:43 PM > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener No virus found in this incoming message. Checked by AVG. Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: 3/22/2008 4:43 PM No virus found in this outgoing message. Checked by AVG. Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: 3/22/2008 4:43 PM From knepley at gmail.com Sun Mar 23 15:33:02 2008 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 23 Mar 2008 15:33:02 -0500 Subject: Creating a new vector based on a already distributed set of Vecs In-Reply-To: <000f01c88d23$69daae10$383010ac@neutron> References: <000f01c88d23$69daae10$383010ac@neutron> Message-ID: 2008/3/23 Vijay S. Mahadevan : > There is more than 1 vector involved, with different data and layout > dependent on the physics and mesh. Since the different Vec objects in > systems have their own layout, I do not want to modify them or recreate > them. I just want to create an interface vector so that I can pass it on to > SNESSolve without having the hassle of synchronizing the individual vectors > in each system. > > I am writing a multi-physics code and this global vector concept could be > applied to the global nonlinear residual and the global solution vector. It > doesn't make sense to recreate an entirely new vector and synchronizing > these at every call to the residual. And there lies my motivation. But if > you see an alternative way to approach this, I would be glad to hear it ! You would like a single Vec which is the concatenation of a set of distributed Vecs. I think this is needless economizing. If you are solving systems, the memory involved in the Mat, PC, and KSP will swamp one vector. I would get the code working first. Then if you really need this memory savings, you can extract the pointer from the global vector and use it to create several smaller vectors. However, as I said, I think it is unnecessary. Matt > Sorry if I wasn't too clear before. > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] > On Behalf Of Matthew Knepley > Sent: Sunday, March 23, 2008 3:13 PM > To: petsc-users at mcs.anl.gov > Subject: Re: Creating a new vector based on a already distributed set of > Vecs > > 2008/3/23 Vijay S. Mahadevan : > > Hi all, > > > > I am sure the subject line wasn't too explanatory and so here goes. > > > > I have a system which already has a Vec object created and distributed on > > many processors based on domain decomposition. I could have several such > > systems, each with a solution vector and a residual vector pertaining to > its > > physics. > > > > Now, can I create a global vector which would just be pointers to the > > already existing vectors but from PETSc's point of view, appears to be a > > globally valid, (and possibly weirdly) distributed vector. Such an option > > will save the memory needed for the global vector and eliminates errors > as > > to synchronization of the solution, residuals for the systems. > > I do not understand what you want. If you want a vector with the same > layout, > use VecDuplicate(). If you want the same data, just use the Vec itself. > > Thanks, > > Matt > > > I was reading the documentation and found the PetscMap data structure. > But I > > am not entirely sure if this is what I am looking for. > > > > I hope that makes sense. I would appreciate any help you can provide to > > point me in the right direction. > > > > Cheers, > > Vijay > > > > No virus found in this outgoing message. > > Checked by AVG. > > Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: > 3/22/2008 > > 4:43 PM > > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > No virus found in this incoming message. > > > Checked by AVG. > Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: 3/22/2008 > 4:43 PM > > > No virus found in this outgoing message. > Checked by AVG. > Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: 3/22/2008 > 4:43 PM > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From vijay.m at gmail.com Sun Mar 23 16:03:18 2008 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Sun, 23 Mar 2008 16:03:18 -0500 Subject: Creating a new vector based on a already distributed set of Vecs In-Reply-To: Message-ID: <001301c88d29$53564f40$383010ac@neutron> Matt, > You would like a single Vec which is the concatenation of a set of > distributed Vecs. Yes. But I do not intend to allocate memory separately for the single Vec but rather it just acts like an interface to the original, individual vector. I am not worried about optimization yet and hence this is not entirely because of memory concerns. I am trying to use JFNK technique to solve a coupled problem. Since each physics knows its mesh and the optimal layout, it is best to let the physics object create and distribute the solution and residual vector. A global coupler would then only have to call the residuals on the individual physics to assemble the total residual. If I do not have a pointer to the physics residuals for the global residual, which is passed to SNESSolve, I then would have to take out the individual components from the global residual belonging to a single physics and then call its residual function. And that has to be done for every single residual call from SNES. That IMO is a far bigger computational issue that can be resolved by creating this interface vector. If you still think this is not the way to go, that is fine. Thanks for the help. Vijay -----Original Message----- From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Sunday, March 23, 2008 3:33 PM To: petsc-users at mcs.anl.gov Subject: Re: Creating a new vector based on a already distributed set of Vecs 2008/3/23 Vijay S. Mahadevan : > There is more than 1 vector involved, with different data and layout > dependent on the physics and mesh. Since the different Vec objects in > systems have their own layout, I do not want to modify them or recreate > them. I just want to create an interface vector so that I can pass it on to > SNESSolve without having the hassle of synchronizing the individual vectors > in each system. > > I am writing a multi-physics code and this global vector concept could be > applied to the global nonlinear residual and the global solution vector. It > doesn't make sense to recreate an entirely new vector and synchronizing > these at every call to the residual. And there lies my motivation. But if > you see an alternative way to approach this, I would be glad to hear it ! You would like a single Vec which is the concatenation of a set of distributed Vecs. I think this is needless economizing. If you are solving systems, the memory involved in the Mat, PC, and KSP will swamp one vector. I would get the code working first. Then if you really need this memory savings, you can extract the pointer from the global vector and use it to create several smaller vectors. However, as I said, I think it is unnecessary. Matt > Sorry if I wasn't too clear before. > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] > On Behalf Of Matthew Knepley > Sent: Sunday, March 23, 2008 3:13 PM > To: petsc-users at mcs.anl.gov > Subject: Re: Creating a new vector based on a already distributed set of > Vecs > > 2008/3/23 Vijay S. Mahadevan : > > Hi all, > > > > I am sure the subject line wasn't too explanatory and so here goes. > > > > I have a system which already has a Vec object created and distributed on > > many processors based on domain decomposition. I could have several such > > systems, each with a solution vector and a residual vector pertaining to > its > > physics. > > > > Now, can I create a global vector which would just be pointers to the > > already existing vectors but from PETSc's point of view, appears to be a > > globally valid, (and possibly weirdly) distributed vector. Such an option > > will save the memory needed for the global vector and eliminates errors > as > > to synchronization of the solution, residuals for the systems. > > I do not understand what you want. If you want a vector with the same > layout, > use VecDuplicate(). If you want the same data, just use the Vec itself. > > Thanks, > > Matt > > > I was reading the documentation and found the PetscMap data structure. > But I > > am not entirely sure if this is what I am looking for. > > > > I hope that makes sense. I would appreciate any help you can provide to > > point me in the right direction. > > > > Cheers, > > Vijay > > > > No virus found in this outgoing message. > > Checked by AVG. > > Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: > 3/22/2008 > > 4:43 PM > > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > No virus found in this incoming message. > > > Checked by AVG. > Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: 3/22/2008 > 4:43 PM > > > No virus found in this outgoing message. > Checked by AVG. > Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: 3/22/2008 > 4:43 PM > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener No virus found in this incoming message. Checked by AVG. Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: 3/22/2008 4:43 PM No virus found in this outgoing message. Checked by AVG. Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: 3/22/2008 4:43 PM From bsmith at mcs.anl.gov Sun Mar 23 17:03:01 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 23 Mar 2008 17:03:01 -0500 Subject: Creating a new vector based on a already distributed set of Vecs In-Reply-To: <000601c88d20$6b353c60$383010ac@neutron> References: <000601c88d20$6b353c60$383010ac@neutron> Message-ID: On Mar 23, 2008, at 2:59 PM, Vijay S. Mahadevan wrote: > Hi all, > > I am sure the subject line wasn?t too explanatory and so here goes. > > I have a system which already has a Vec object created and > distributed on > many processors based on domain decomposition. I could have several > such > systems, each with a solution vector and a residual vector > pertaining to its > physics. Is this really an accurate description of the situation? In parallel computing (and certainly in the PETSc paradigm) for a single physics there are really TWO distinct vector layouts. The layout that the Newton and linear solver sees (without holes in the vectors or ghost points in the vectors), what we call the Global vectors and then what the "physics" on each process sees, which is a part of the global vector plus the ghost points that are needed to perform the local part of the computation, what we call the local (or ghosted) vectors. At the beginning of the function evaluations (or matrix-free multiply) values are moved from the global representation to the local representation, local "physics" is computed and then (depending on the discretization, for finite differences usually no, for finite element usually yes) computed values are moved back to the process where they belong from ghost locations. In PETSc the movement of the ghost values is handled with VecScatter or DAGlobalToLocal/DALocalToGlobal. With multiple "physics" (say two) there are possibly three levels: the global representation of the unified physics, the global representation of the individual physics and the local representation of the individual physics (note that "individual" physics local computations actually depend, in general, on other individual physics either as given parameters or boundary values ). If you are doing linear or nonlinear solves on the individual physics then you really need all three representations, if you only have an outer Newton (or Krylov) solver and compute the physics by computing a bunch of individuals physics solves then you really don't need the global representation of the individual physics, you could scatter directly from the global representation of the unified physics to the local (ghosted) representation of the individual physics. Based on your question I am guessing you have the first case, that is you already work with global representation of the individual physics. If this is correct then a global representation of the multiphysics conceptually is simply a concatenation of the global representation of the individual physics: to be more concrete say I have global representations of physics one which is an MPI vector u where u(part-zero) lives on process 0 while u(part-one) lives on process 1. Meanwhile physics two has T(part-zero) on the first process and T(part- one) lives on process 1. (Since these are global representations of u and T there are no ghost points inside the u and T). Now the global representation of the unified physics would be P(part- zero) = [u(part-zero) T(part-zero)] on process zero and P(part-one) = [u(part-one) T(part-one)]. Your question is then: if you have a vector u(part-*) and T(part-*) can you avoid copying them in P(part*) (the global SNES solver works on the global unified vector) and instead wrap up the two vectors u and T to make them look like a single vector for SNES? The answer is yes you can, BUT (as Matt points out) doing this is ONLY a tiny optimization (in time and memory savings) over simply copying the two parts into P when needed and copying them out of P when neccessary). In the entire running of your two physics I would be amazed if these little copies (which involve no parallel communication and will be totally swamped by the parallel communication used in updating the ghost points) made up more than a tiny percentage of the time (if they do then the two "physics" are each trivial). I highly recommend starting with the three distinct levels of representation and doing the needed copies; your code will be clean and MUCH easier to debug and keep track of. Once you are running your cool multiphysics app on your real problem ,if profiling indicates the extra copies are killing you you can come back to us and we can help with optimizations. Barry We are ourselves struggling with approaches to making doing multi- physics very straightforward with PETSc; any feed back from users is helpful . > > > Now, can I create a global vector which would just be pointers to the > already existing vectors but from PETSc's point of view, appears to > be a > globally valid, (and possibly weirdly) distributed vector. Such an > option > will save the memory needed for the global vector and eliminates > errors as > to synchronization of the solution, residuals for the systems. > > I was reading the documentation and found the PetscMap data > structure. But I > am not entirely sure if this is what I am looking for. > > I hope that makes sense. I would appreciate any help you can provide > to > point me in the right direction. > > Cheers, > Vijay > > No virus found in this outgoing message. > Checked by AVG. > Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: > 3/22/2008 > 4:43 PM > > From vijay.m at gmail.com Sun Mar 23 18:21:15 2008 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Sun, 23 Mar 2008 18:21:15 -0500 Subject: Creating a new vector based on a already distributed set of Vecs In-Reply-To: Message-ID: <002601c88d3c$98c91bd0$383010ac@neutron> Barry, First, thanks for the detailed answer. That cleared few of my questions already. > Based on your question I am guessing you have the first case, that is > you already > work with global representation of the individual physics. If this is > correct then a global representation > of the multiphysics conceptually is simply a concatenation of the > global representation of the > individual physics Yes. This is my situation currently. And I am making now the transition from single (case 1: global individual physics and ghosted individual physics vectors) to multi-physics (case 2: global unified physics, global individual physics and ghosted individual physics vectors). > I highly > recommend starting with the three distinct levels of representation > and doing the needed copies; > your code will be clean and MUCH easier to debug and keep track of. Fair enough. Thanks for analyzing the possibilities here. > Once you are running your > cool multiphysics app on your real problem ,if profiling indicates the > extra copies are killing you > you can come back to us and we can help with optimizations. I will keep that in mind Barry :-) Again, Matt and Barry, thanks for the help ! > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] > On Behalf Of Barry Smith > Sent: Sunday, March 23, 2008 5:03 PM > To: petsc-users at mcs.anl.gov > Subject: Re: Creating a new vector based on a already distributed set of > Vecs > > > On Mar 23, 2008, at 2:59 PM, Vijay S. Mahadevan wrote: > > Hi all, > > > > I am sure the subject line wasn?t too explanatory and so here goes. > > > > I have a system which already has a Vec object created and > > distributed on > > many processors based on domain decomposition. I could have several > > such > > systems, each with a solution vector and a residual vector > > pertaining to its > > physics. > > Is this really an accurate description of the situation? In > parallel computing > (and certainly in the PETSc paradigm) for a single physics there are > really TWO distinct > vector layouts. The layout that the Newton and linear solver sees > (without holes in the vectors > or ghost points in the vectors), what we call the Global vectors and > then what the > "physics" on each process sees, which is a part of the global vector > plus the ghost > points that are needed to perform the local part of the computation, > what we > call the local (or ghosted) vectors. At the beginning of the function > evaluations (or matrix-free multiply) > values are moved from the global representation to the local > representation, > local "physics" is computed and then (depending on the discretization, > for finite differences > usually no, for finite element usually yes) computed values are moved > back to the > process where they belong from ghost locations. In PETSc the movement > of the ghost values is handled > with VecScatter or DAGlobalToLocal/DALocalToGlobal. > > With multiple "physics" (say two) there are possibly three levels: > the global representation > of the unified physics, the global representation of the individual > physics and the > local representation of the individual physics (note that "individual" > physics local computations > actually depend, in general, on other individual physics either as > given parameters or > boundary values ). If you are doing linear or nonlinear solves on the > individual physics > then you really need all three representations, if you only have an > outer Newton (or Krylov) > solver and compute the physics by computing a bunch of individuals > physics solves then > you really don't need the global representation of the individual > physics, you could > scatter directly from the global representation of the unified physics > to the local (ghosted) > representation of the individual physics. > > Based on your question I am guessing you have the first case, that is > you already > work with global representation of the individual physics. If this is > correct then a global representation > of the multiphysics conceptually is simply a concatenation of the > global representation of the > individual physics: to be more concrete say I have global > representations of physics one > which is an MPI vector u where u(part-zero) lives on process 0 while > u(part-one) lives on process 1. > Meanwhile physics two has T(part-zero) on the first process and T(part- > one) lives on process 1. > (Since these are global representations of u and T there are no ghost > points inside the u and T). > Now the global representation of the unified physics would be P(part- > zero) = [u(part-zero) T(part-zero)] on process > zero and P(part-one) = [u(part-one) T(part-one)]. > > Your question is then: if you have a vector u(part-*) and T(part-*) > can you avoid copying them > in P(part*) (the global SNES solver works on the global unified > vector) and instead wrap up the > two vectors u and T to make them look like a single vector for SNES? > > The answer is yes you can, BUT (as Matt points out) doing this is ONLY > a tiny optimization (in time and memory > savings) over simply copying the two parts into P when needed and > copying them out of P when neccessary). > In the entire running of your two physics I would be amazed if these > little copies (which involve no > parallel communication and will be totally swamped by the parallel > communication used in updating the > ghost points) made up more than > a tiny percentage of the time (if they do then the two "physics" are > each trivial). I highly > recommend starting with the three distinct levels of representation > and doing the needed copies; > your code will be clean and MUCH easier to debug and keep track of. > Once you are running your > cool multiphysics app on your real problem ,if profiling indicates the > extra copies are killing you > you can come back to us and we can help with optimizations. > > > Barry > > > We are ourselves struggling with approaches to making doing multi- > physics very straightforward with PETSc; > any feed back from users is helpful . > > > > > > > > > Now, can I create a global vector which would just be pointers to the > > already existing vectors but from PETSc's point of view, appears to > > be a > > globally valid, (and possibly weirdly) distributed vector. Such an > > option > > will save the memory needed for the global vector and eliminates > > errors as > > to synchronization of the solution, residuals for the systems. > > > > I was reading the documentation and found the PetscMap data > > structure. But I > > am not entirely sure if this is what I am looking for. > > > > I hope that makes sense. I would appreciate any help you can provide > > to > > point me in the right direction. > > > > Cheers, > > Vijay > > No virus found in this outgoing message. Checked by AVG. Version: 7.5.519 / Virus Database: 269.21.8/1339 - Release Date: 3/22/2008 4:43 PM From zonexo at gmail.com Sun Mar 23 21:23:40 2008 From: zonexo at gmail.com (Ben Tay) Date: Mon, 24 Mar 2008 10:23:40 +0800 Subject: Linking problems with ifort and visual studio 2003 In-Reply-To: References: <47E64A12.4060901@gmail.com> Message-ID: <47E710AC.50302@gmail.com> Hi Satish, I tried to change the compiler options to be as similar to that in the cygwin environment. Still can't manage to get it to work. The errors I copy n paste is directly from the IDE of studio 2003, so I'm sure it can't be from Compaq. Btw, here's the options: For compiling (no prob) /nologo /Zi /O3 /fpp /include:"Debug/" /include:"E:\cygwin\codes\petsc-2.3.3-p10" /include:"E:\cygwin\codes\petsc-2.3.3-p10\include" /include:"E:\cygwin\codes\petsc-2.3.3-p10\bmake\win32_ifort_nodebug" /include:"E:\cygwin\codes\petsc-2.3.3-p10\include\mpiuni" /include:"Debug/" /extend_source:132 /real_size:64 /Qsave /iface:cvf /module:"Debug/" /object:"Debug/" /traceback /check:bounds /c Cygwin: For compiling (no prob) /cygdrive/e/cygwin/codes/petsc-2.3.3-p10/bin/win32fe/win32fe ifort --nodetect -c -MT -O3 -QxW -fpp -I/cygdrive/e/cygwin/codes/petsc-2.3.3-p10 -I/cygdrive/e/cy gwin/codes/petsc-2.3.3-p10/bmake/win32_ifort_nodebug -I/cygdrive/e/cygwin/codes/ petsc-2.3.3-p10/include -I/cygdrive/e/cygwin/codes/petsc-2.3.3-p10/include/mpiun i -o ex1f.o ex1f.F Visualstudio 2003: For linker /OUT:"Debug/ns2d_c.exe" /INCREMENTAL /NOLOGO /LIBPATH:"E:\cygwin\codes\petsc-2.3.3-p10\lib\win32_ifort_nodebug" /LIBPATH:"E:\cygwin\codes\petsc-2.3.3-p10\externalpackages\fblaslapack\win32_ifort_nodebug" /LIBPATH:"E:\cygwin\codes\petsc-2.3.3-p10\include\mpiuni" /LIBPATH:"C:\Program Files\Tecplot\Tec360\bin\\" /NODEFAULTLIB:"LIBCMT" /DEBUG /PDB:"Debug/ns2d_c.pdb" /SUBSYSTEM:CONSOLE tecio.lib ws2_32.lib kernel32.lib user32.lib gdi32.lib advapi32.lib libpetscts.lib libpetscsnes.lib libpetscksp.lib libpetscdm.lib libpetscmat.lib libpetscvec.lib libpetsc.lib libfblas.lib libflapack.lib libmpiuni.lib Cygwin: For linker /cygdrive/e/cygwin/codes/petsc-2.3.3-p10/bin/win32fe/win32fe ifort --nodetect -M T -O3 -QxW -fpp -o ex1f ex1f.o -L/cygdrive/e/cygwin/codes/petsc-2.3.3-p10/lib/ win32_ifort_nodebug -L/cygdrive/e/cygwin/codes/petsc-2.3.3-p10/lib/win32_ifort_n odebug -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc -L/cygdrive/e/cygwi n/codes/petsc-2.3.3-p10/lib/win32_ifort_nodebug -L/cygdrive/e/cygwin/codes/petsc -2.3.3-p10/lib/win32_ifort_nodebug -lmpiuni -L/cygdrive/e/cygwin/codes/petsc-2.3 .3-p10/externalpackages/fblaslapack/win32_ifort_nodebug -L/cygdrive/e/cygwin/cod es/petsc-2.3.3-p10/externalpackages/fblaslapack/win32_ifort_nodebug -lflapack -L /cygdrive/e/cygwin/codes/petsc-2.3.3-p10/externalpackages/fblaslapack/win32_ifor t_nodebug -L/cygdrive/e/cygwin/codes/petsc-2.3.3-p10/externalpackages/fblaslapac k/win32_ifort_nodebug -lfblas Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2 _32.lib Did I missed out anything? I also tried to add /nodetect /MT /QxW /fpp but visualstudio 2003 says unrecognized options. Thank you very much Satish Balay wrote: > Ok - you are using VisualStudio 2003 + IntelFortran compilers to build > PETSc. A PETSc fortran example [ex1f] compiles from 'make' but not > from IDE? > > Then some setting in the IDE does not match the compile option. Per > installation instructions - you should make sure you have *all* > compiler options you see from make [when compiling ex1f] are also > specified in the IDE settings. > > >> global.obj error LNK2019: unresolved external symbol _PETSCINITIALIZE at 12 >> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >> > > Looks like this code is compiled by Compaq Fortran compiler - and not > intel fortran. Can you verify? > > Satish > > On Sun, 23 Mar 2008, Ben Tay wrote: > > >> Hi, >> >> I've successfully compiled the library of PETSc. The examples also worked (I >> tested ex1f). However in the IDE of visual studio 2003, I can't seem to link, >> although compiling the individual files were fine. The errors are: >> >> global.obj error LNK2019: unresolved external symbol _PETSCINITIALIZE at 12 >> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >> global.obj error LNK2019: unresolved external symbol _MATCREATESEQAIJ at 28 >> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >> global.obj error LNK2019: unresolved external symbol _VECCREATESEQ at 16 >> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >> global.obj error LNK2019: unresolved external symbol _VECDUPLICATE at 12 >> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >> global.obj error LNK2019: unresolved external symbol _KSPCREATE at 12 referenced >> in function _GLOBAL_DATA_mp_ALLO_VAR >> global.obj error LNK2019: unresolved external symbol _VECASSEMBLYBEGIN at 8 >> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >> global.obj error LNK2019: unresolved external symbol _VECASSEMBLYEND at 8 >> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >> bc.obj error LNK2019: unresolved external symbol _MATSETVALUES at 32 referenced >> in function _BC_mp_BC_BIG_A_S >> bc_impl.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 >> set_matrix.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 >> petsc_sub.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 >> bc_impl.obj error LNK2019: unresolved external symbol _VECSETVALUE at 20 >> referenced in function _BC_IMPL_mp_BC_Q_IMPL_S >> set_matrix.obj error LNK2001: unresolved external symbol _VECSETVALUE at 20 >> petsc_sub.obj error LNK2001: unresolved external symbol _VECSETVALUE at 20 >> petsc_sub.obj error LNK2019: unresolved external symbol _MATASSEMBLYBEGIN at 12 >> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >> petsc_sub.obj error LNK2019: unresolved external symbol _MATASSEMBLYEND at 12 >> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >> petsc_sub.obj error LNK2019: unresolved external symbol _MATSCALE at 12 >> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >> petsc_sub.obj error LNK2019: unresolved external symbol _KSPSETOPERATORS at 20 >> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >> petsc_sub.obj error LNK2019: unresolved external symbol _KSPGETPC at 12 >> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >> petsc_sub.obj error LNK2019: unresolved external symbol _KSPSETTYPE at 16 >> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >> petsc_sub.obj error LNK2019: unresolved external symbol _PCSETTYPE at 16 >> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >> >> >> I was using Compaq visual fortran initially and I have converted the *.dsw >> file to the visual studio 2003 format. Of cos, I have changed the locations to >> the new PETSc directory. May I know what's happening? Is there anything I need >> to set or changed in studio 2003? >> >> Thanks alot! >> >> >> > > > From balay at mcs.anl.gov Sun Mar 23 21:31:20 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 23 Mar 2008 21:31:20 -0500 (CDT) Subject: Linking problems with ifort and visual studio 2003 In-Reply-To: <47E710AC.50302@gmail.com> References: <47E64A12.4060901@gmail.com> <47E710AC.50302@gmail.com> Message-ID: On Mon, 24 Mar 2008, Ben Tay wrote: > Hi Satish, > > I tried to change the compiler options to be as similar to that in the cygwin > environment. Still can't manage to get it to work. The errors I copy n paste > is directly from the IDE of studio 2003, so I'm sure it can't be from Compaq. > Btw, here's the options: > > > > For compiling (no prob) > > /nologo /Zi /O3 /fpp /include:"Debug/" > /include:"E:\cygwin\codes\petsc-2.3.3-p10" > /include:"E:\cygwin\codes\petsc-2.3.3-p10\include" > /include:"E:\cygwin\codes\petsc-2.3.3-p10\bmake\win32_ifort_nodebug" > /include:"E:\cygwin\codes\petsc-2.3.3-p10\include\mpiuni" /include:"Debug/" > /extend_source:132 /real_size:64 /Qsave /iface:cvf /module:"Debug/" ^^^^^^^^^^^^^^^^^ Notice these options are not in the PETSc build. The option '/iface:cvf' is the cause of this incompatibility. Satish > /object:"Debug/" /traceback /check:bounds /c > > Cygwin: > > For compiling (no prob) > > /cygdrive/e/cygwin/codes/petsc-2.3.3-p10/bin/win32fe/win32fe ifort --nodetect > -c > -MT -O3 -QxW -fpp -I/cygdrive/e/cygwin/codes/petsc-2.3.3-p10 > -I/cygdrive/e/cy > gwin/codes/petsc-2.3.3-p10/bmake/win32_ifort_nodebug > -I/cygdrive/e/cygwin/codes/ > petsc-2.3.3-p10/include > -I/cygdrive/e/cygwin/codes/petsc-2.3.3-p10/include/mpiun > i -o ex1f.o ex1f.F > > Visualstudio 2003: > > For linker > > /OUT:"Debug/ns2d_c.exe" /INCREMENTAL /NOLOGO > /LIBPATH:"E:\cygwin\codes\petsc-2.3.3-p10\lib\win32_ifort_nodebug" > /LIBPATH:"E:\cygwin\codes\petsc-2.3.3-p10\externalpackages\fblaslapack\win32_ifort_nodebug" > /LIBPATH:"E:\cygwin\codes\petsc-2.3.3-p10\include\mpiuni" /LIBPATH:"C:\Program > Files\Tecplot\Tec360\bin\\" /NODEFAULTLIB:"LIBCMT" /DEBUG > /PDB:"Debug/ns2d_c.pdb" /SUBSYSTEM:CONSOLE tecio.lib ws2_32.lib kernel32.lib > user32.lib gdi32.lib advapi32.lib libpetscts.lib libpetscsnes.lib > libpetscksp.lib libpetscdm.lib libpetscmat.lib libpetscvec.lib libpetsc.lib > libfblas.lib libflapack.lib libmpiuni.lib > > Cygwin: > > For linker > > /cygdrive/e/cygwin/codes/petsc-2.3.3-p10/bin/win32fe/win32fe ifort --nodetect > -M > T -O3 -QxW -fpp -o ex1f ex1f.o > -L/cygdrive/e/cygwin/codes/petsc-2.3.3-p10/lib/ > win32_ifort_nodebug > -L/cygdrive/e/cygwin/codes/petsc-2.3.3-p10/lib/win32_ifort_n > odebug -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc > -L/cygdrive/e/cygwi > n/codes/petsc-2.3.3-p10/lib/win32_ifort_nodebug > -L/cygdrive/e/cygwin/codes/petsc > -2.3.3-p10/lib/win32_ifort_nodebug -lmpiuni > -L/cygdrive/e/cygwin/codes/petsc-2.3 > .3-p10/externalpackages/fblaslapack/win32_ifort_nodebug > -L/cygdrive/e/cygwin/cod > es/petsc-2.3.3-p10/externalpackages/fblaslapack/win32_ifort_nodebug -lflapack > -L > /cygdrive/e/cygwin/codes/petsc-2.3.3-p10/externalpackages/fblaslapack/win32_ifor > t_nodebug > -L/cygdrive/e/cygwin/codes/petsc-2.3.3-p10/externalpackages/fblaslapac > k/win32_ifort_nodebug -lfblas Gdi32.lib User32.lib Advapi32.lib Kernel32.lib > Ws2 > _32.lib > > Did I missed out anything? I also tried to add /nodetect /MT /QxW /fpp but > visualstudio 2003 says unrecognized options. > > Thank you very much > > Satish Balay wrote: > > Ok - you are using VisualStudio 2003 + IntelFortran compilers to build > > PETSc. A PETSc fortran example [ex1f] compiles from 'make' but not > > from IDE? > > > > Then some setting in the IDE does not match the compile option. Per > > installation instructions - you should make sure you have *all* > > compiler options you see from make [when compiling ex1f] are also > > specified in the IDE settings. > > > > > > > global.obj error LNK2019: unresolved external symbol _PETSCINITIALIZE at 12 > > > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > > > > > > > Looks like this code is compiled by Compaq Fortran compiler - and not > > intel fortran. Can you verify? > > > > Satish > > > > On Sun, 23 Mar 2008, Ben Tay wrote: > > > > > > > Hi, > > > > > > I've successfully compiled the library of PETSc. The examples also worked > > > (I > > > tested ex1f). However in the IDE of visual studio 2003, I can't seem to > > > link, > > > although compiling the individual files were fine. The errors are: > > > > > > global.obj error LNK2019: unresolved external symbol _PETSCINITIALIZE at 12 > > > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > > > global.obj error LNK2019: unresolved external symbol _MATCREATESEQAIJ at 28 > > > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > > > global.obj error LNK2019: unresolved external symbol _VECCREATESEQ at 16 > > > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > > > global.obj error LNK2019: unresolved external symbol _VECDUPLICATE at 12 > > > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > > > global.obj error LNK2019: unresolved external symbol _KSPCREATE at 12 > > > referenced > > > in function _GLOBAL_DATA_mp_ALLO_VAR > > > global.obj error LNK2019: unresolved external symbol _VECASSEMBLYBEGIN at 8 > > > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > > > global.obj error LNK2019: unresolved external symbol _VECASSEMBLYEND at 8 > > > referenced in function _GLOBAL_DATA_mp_ALLO_VAR > > > bc.obj error LNK2019: unresolved external symbol _MATSETVALUES at 32 > > > referenced > > > in function _BC_mp_BC_BIG_A_S > > > bc_impl.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 > > > set_matrix.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 > > > petsc_sub.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 > > > bc_impl.obj error LNK2019: unresolved external symbol _VECSETVALUE at 20 > > > referenced in function _BC_IMPL_mp_BC_Q_IMPL_S > > > set_matrix.obj error LNK2001: unresolved external symbol _VECSETVALUE at 20 > > > petsc_sub.obj error LNK2001: unresolved external symbol _VECSETVALUE at 20 > > > petsc_sub.obj error LNK2019: unresolved external symbol > > > _MATASSEMBLYBEGIN at 12 > > > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > > > petsc_sub.obj error LNK2019: unresolved external symbol > > > _MATASSEMBLYEND at 12 > > > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > > > petsc_sub.obj error LNK2019: unresolved external symbol _MATSCALE at 12 > > > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > > > petsc_sub.obj error LNK2019: unresolved external symbol > > > _KSPSETOPERATORS at 20 > > > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > > > petsc_sub.obj error LNK2019: unresolved external symbol _KSPGETPC at 12 > > > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > > > petsc_sub.obj error LNK2019: unresolved external symbol _KSPSETTYPE at 16 > > > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > > > petsc_sub.obj error LNK2019: unresolved external symbol _PCSETTYPE at 16 > > > referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM > > > > > > > > > I was using Compaq visual fortran initially and I have converted the *.dsw > > > file to the visual studio 2003 format. Of cos, I have changed the > > > locations to > > > the new PETSc directory. May I know what's happening? Is there anything I > > > need > > > to set or changed in studio 2003? > > > > > > Thanks alot! > > > > > > > > > > > > > > > > > From zonexo at gmail.com Mon Mar 24 00:58:06 2008 From: zonexo at gmail.com (Ben Tay) Date: Mon, 24 Mar 2008 13:58:06 +0800 Subject: Linking problems with ifort and visual studio 2003 In-Reply-To: References: <47E64A12.4060901@gmail.com> <47E710AC.50302@gmail.com> Message-ID: <47E742EE.8060903@gmail.com> Oh thanks Satish. It finally worked! Satish Balay wrote: > > On Mon, 24 Mar 2008, Ben Tay wrote: > > >> Hi Satish, >> >> I tried to change the compiler options to be as similar to that in the cygwin >> environment. Still can't manage to get it to work. The errors I copy n paste >> is directly from the IDE of studio 2003, so I'm sure it can't be from Compaq. >> Btw, here's the options: >> >> >> >> For compiling (no prob) >> >> /nologo /Zi /O3 /fpp /include:"Debug/" >> /include:"E:\cygwin\codes\petsc-2.3.3-p10" >> /include:"E:\cygwin\codes\petsc-2.3.3-p10\include" >> /include:"E:\cygwin\codes\petsc-2.3.3-p10\bmake\win32_ifort_nodebug" >> /include:"E:\cygwin\codes\petsc-2.3.3-p10\include\mpiuni" /include:"Debug/" >> /extend_source:132 /real_size:64 /Qsave /iface:cvf /module:"Debug/" >> > ^^^^^^^^^^^^^^^^^ > > Notice these options are not in the PETSc build. The option > '/iface:cvf' is the cause of this incompatibility. > > Satish > > > >> /object:"Debug/" /traceback /check:bounds /c >> >> Cygwin: >> >> For compiling (no prob) >> >> /cygdrive/e/cygwin/codes/petsc-2.3.3-p10/bin/win32fe/win32fe ifort --nodetect >> -c >> -MT -O3 -QxW -fpp -I/cygdrive/e/cygwin/codes/petsc-2.3.3-p10 >> -I/cygdrive/e/cy >> gwin/codes/petsc-2.3.3-p10/bmake/win32_ifort_nodebug >> -I/cygdrive/e/cygwin/codes/ >> petsc-2.3.3-p10/include >> -I/cygdrive/e/cygwin/codes/petsc-2.3.3-p10/include/mpiun >> i -o ex1f.o ex1f.F >> >> Visualstudio 2003: >> >> For linker >> >> /OUT:"Debug/ns2d_c.exe" /INCREMENTAL /NOLOGO >> /LIBPATH:"E:\cygwin\codes\petsc-2.3.3-p10\lib\win32_ifort_nodebug" >> /LIBPATH:"E:\cygwin\codes\petsc-2.3.3-p10\externalpackages\fblaslapack\win32_ifort_nodebug" >> /LIBPATH:"E:\cygwin\codes\petsc-2.3.3-p10\include\mpiuni" /LIBPATH:"C:\Program >> Files\Tecplot\Tec360\bin\\" /NODEFAULTLIB:"LIBCMT" /DEBUG >> /PDB:"Debug/ns2d_c.pdb" /SUBSYSTEM:CONSOLE tecio.lib ws2_32.lib kernel32.lib >> user32.lib gdi32.lib advapi32.lib libpetscts.lib libpetscsnes.lib >> libpetscksp.lib libpetscdm.lib libpetscmat.lib libpetscvec.lib libpetsc.lib >> libfblas.lib libflapack.lib libmpiuni.lib >> >> Cygwin: >> >> For linker >> >> /cygdrive/e/cygwin/codes/petsc-2.3.3-p10/bin/win32fe/win32fe ifort --nodetect >> -M >> T -O3 -QxW -fpp -o ex1f ex1f.o >> -L/cygdrive/e/cygwin/codes/petsc-2.3.3-p10/lib/ >> win32_ifort_nodebug >> -L/cygdrive/e/cygwin/codes/petsc-2.3.3-p10/lib/win32_ifort_n >> odebug -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc >> -L/cygdrive/e/cygwi >> n/codes/petsc-2.3.3-p10/lib/win32_ifort_nodebug >> -L/cygdrive/e/cygwin/codes/petsc >> -2.3.3-p10/lib/win32_ifort_nodebug -lmpiuni >> -L/cygdrive/e/cygwin/codes/petsc-2.3 >> .3-p10/externalpackages/fblaslapack/win32_ifort_nodebug >> -L/cygdrive/e/cygwin/cod >> es/petsc-2.3.3-p10/externalpackages/fblaslapack/win32_ifort_nodebug -lflapack >> -L >> /cygdrive/e/cygwin/codes/petsc-2.3.3-p10/externalpackages/fblaslapack/win32_ifor >> t_nodebug >> -L/cygdrive/e/cygwin/codes/petsc-2.3.3-p10/externalpackages/fblaslapac >> k/win32_ifort_nodebug -lfblas Gdi32.lib User32.lib Advapi32.lib Kernel32.lib >> Ws2 >> _32.lib >> >> Did I missed out anything? I also tried to add /nodetect /MT /QxW /fpp but >> visualstudio 2003 says unrecognized options. >> >> Thank you very much >> >> Satish Balay wrote: >> >>> Ok - you are using VisualStudio 2003 + IntelFortran compilers to build >>> PETSc. A PETSc fortran example [ex1f] compiles from 'make' but not >>> from IDE? >>> >>> Then some setting in the IDE does not match the compile option. Per >>> installation instructions - you should make sure you have *all* >>> compiler options you see from make [when compiling ex1f] are also >>> specified in the IDE settings. >>> >>> >>> >>>> global.obj error LNK2019: unresolved external symbol _PETSCINITIALIZE at 12 >>>> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >>>> >>>> >>> Looks like this code is compiled by Compaq Fortran compiler - and not >>> intel fortran. Can you verify? >>> >>> Satish >>> >>> On Sun, 23 Mar 2008, Ben Tay wrote: >>> >>> >>> >>>> Hi, >>>> >>>> I've successfully compiled the library of PETSc. The examples also worked >>>> (I >>>> tested ex1f). However in the IDE of visual studio 2003, I can't seem to >>>> link, >>>> although compiling the individual files were fine. The errors are: >>>> >>>> global.obj error LNK2019: unresolved external symbol _PETSCINITIALIZE at 12 >>>> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >>>> global.obj error LNK2019: unresolved external symbol _MATCREATESEQAIJ at 28 >>>> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >>>> global.obj error LNK2019: unresolved external symbol _VECCREATESEQ at 16 >>>> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >>>> global.obj error LNK2019: unresolved external symbol _VECDUPLICATE at 12 >>>> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >>>> global.obj error LNK2019: unresolved external symbol _KSPCREATE at 12 >>>> referenced >>>> in function _GLOBAL_DATA_mp_ALLO_VAR >>>> global.obj error LNK2019: unresolved external symbol _VECASSEMBLYBEGIN at 8 >>>> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >>>> global.obj error LNK2019: unresolved external symbol _VECASSEMBLYEND at 8 >>>> referenced in function _GLOBAL_DATA_mp_ALLO_VAR >>>> bc.obj error LNK2019: unresolved external symbol _MATSETVALUES at 32 >>>> referenced >>>> in function _BC_mp_BC_BIG_A_S >>>> bc_impl.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 >>>> set_matrix.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 >>>> petsc_sub.obj error LNK2001: unresolved external symbol _MATSETVALUES at 32 >>>> bc_impl.obj error LNK2019: unresolved external symbol _VECSETVALUE at 20 >>>> referenced in function _BC_IMPL_mp_BC_Q_IMPL_S >>>> set_matrix.obj error LNK2001: unresolved external symbol _VECSETVALUE at 20 >>>> petsc_sub.obj error LNK2001: unresolved external symbol _VECSETVALUE at 20 >>>> petsc_sub.obj error LNK2019: unresolved external symbol >>>> _MATASSEMBLYBEGIN at 12 >>>> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >>>> petsc_sub.obj error LNK2019: unresolved external symbol >>>> _MATASSEMBLYEND at 12 >>>> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >>>> petsc_sub.obj error LNK2019: unresolved external symbol _MATSCALE at 12 >>>> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >>>> petsc_sub.obj error LNK2019: unresolved external symbol >>>> _KSPSETOPERATORS at 20 >>>> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >>>> petsc_sub.obj error LNK2019: unresolved external symbol _KSPGETPC at 12 >>>> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >>>> petsc_sub.obj error LNK2019: unresolved external symbol _KSPSETTYPE at 16 >>>> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >>>> petsc_sub.obj error LNK2019: unresolved external symbol _PCSETTYPE at 16 >>>> referenced in function _PETSC_SUB_mp_PETSC_SOLVER_MOM >>>> >>>> >>>> I was using Compaq visual fortran initially and I have converted the *.dsw >>>> file to the visual studio 2003 format. Of cos, I have changed the >>>> locations to >>>> the new PETSc directory. May I know what's happening? Is there anything I >>>> need >>>> to set or changed in studio 2003? >>>> >>>> Thanks alot! >>>> >>>> >>>> >>>> >>> >>> >> > > > From dalcinl at gmail.com Mon Mar 24 09:05:42 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 24 Mar 2008 11:05:42 -0300 Subject: ML's multigrid In-Reply-To: <200803231708.44609.geenen@gmail.com> References: <200803230846.43732.geenen@gmail.com> <019E211B-B67E-4749-BE72-CC50D95A573F@mcs.anl.gov> <200803231708.44609.geenen@gmail.com> Message-ID: Thomas, please note that what Barry described, in fact should apply to ALL solvers and preconditioners. That is, if you sucessively call KSPSolve several times (without calling KSPSetOperator), then the setup phases are done only in the first call. If you detect some situation where this seems to not be true, let us know about it. On 3/23/08, Thomas Geenen wrote: > thats right. > i must have overseen it since i used it before for ILU /cg type combinations > and mumps. > > thanks > > Thomas > > > On Sunday 23 March 2008 16:58, Barry Smith wrote: > > Thomas, > > > > It is possible there is an error in our PCSetUp_ML() but more > > likely > > there is a misunderstanding. If you call KSPSolve() repeatedly without > > calling KSPSetOperators() again then it uses the same matrix and > > preconditioner > > as for the previous solve (that is KSP does not know that you changed > > entries in the matrix). If you change the matrix you must call > > KSPSetOperators() > > again to get the new preconditioner being built before calling > > KSPSolve() > > again. > > > > Barry > > > > On Mar 23, 2008, at 2:46 AM, Thomas Geenen wrote: > > > dear Petsc users, > > > > > > I am using the AMG from ML as a preconditioner to CG. I am looking > > > at the > > > scaling characteristics of different parts of the preconditioner > > > (some rough > > > timings). To this end I have put some timing calls in ml.c. > > > now when i run the code and the solver is called for the first time > > > i can see > > > that indeed the ML routines are called in particular PCSetUp_ML. > > > In this routine a considerable amount of time is spend in the > > > aggregation and > > > interpolation operator construction (as expected). However the > > > second time > > > the solver is called this routine is no longer called. This seems to > > > be > > > inconsistent with the concept of AMG where the coarse grids and > > > interpolation > > > operators have to be constructed again when the matrix has changed > > > (due to > > > changing material properties as a function of time for instance). > > > Destroying the preconditioner after each solve seems a waste of > > > resources > > > especially since in PCSetUp_ML , there are options to reuse > > > datastructures. > > > pc-setupcalled, reuse etc. > > > > > > are the aggregation and interpolation comstruction routines called > > > from a > > > different part in the code? > > > ML_Gen_MGHierarchy_UsingAggregation(ml_object, > > > 0,ML_INCREASING,agg_object); > > > seems to be called only from PCSetUp_ML. > > > > > > cheers > > > Thomas > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From jaime.suarez at iesl.forth.gr Mon Mar 24 10:32:03 2008 From: jaime.suarez at iesl.forth.gr (Jaime Suarez) Date: Mon, 24 Mar 2008 17:32:03 +0200 Subject: Getting values of a parallel vector Message-ID: <47E7C973.40709@iesl.forth.gr> Good morning, My question is about getting the values of a parallel vector in the PETSc subroutines using Fortran 90. As far as I know, I can invoke: CALL VecGetArrayF90 in order to get the full array that is being considered in each processor. However, I cannot employ this tool since I need to have also the values that are stored in other processors. Is there a straight way to get the value of the n-th element of a parallel vector, no matter where this element is stored? I am aware of the difficulty of this task, since it would involve transfer of information between different processors. Thanking you in advance, Jaime Suarez Theoretical and Computational Chemistry in Crete (TCCC), Institute of Electronic Structure and Laser (IESL), FOundation for Research and Technology-Hellas (FORTH) From balay at mcs.anl.gov Mon Mar 24 10:59:09 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 24 Mar 2008 10:59:09 -0500 (CDT) Subject: Getting values of a parallel vector In-Reply-To: <47E7C973.40709@iesl.forth.gr> References: <47E7C973.40709@iesl.forth.gr> Message-ID: On Mon, 24 Mar 2008, Jaime Suarez wrote: > Good morning, > > My question is about getting the values of a parallel vector in the > PETSc subroutines using Fortran 90. > > As far as I know, I can invoke: > CALL VecGetArrayF90 > in order to get the full array that is being considered in each > processor. However, I cannot employ this tool since I need to have also > the values that are stored in other processors. Is there a straight way > to get the value of the n-th element of a parallel vector, no matter > where this element is stored? > > I am aware of the difficulty of this task, since it would involve > transfer of information between different processors. > > Thanking you in advance, Check the manpage for VecScatterCreate(). You'll have to create the VecScatter object corresponding to the off-proc values you need - and then scatter them. After the scatter, the values will be available locally [Now you can use VecGetArrayF90 to access the values] Satish From zonexo at gmail.com Mon Mar 24 20:22:39 2008 From: zonexo at gmail.com (Ben Tay) Date: Tue, 25 Mar 2008 09:22:39 +0800 Subject: Can't run ./configure at installation Message-ID: <47E853DF.1090405@gmail.com> Hi, I'm trying to install a new version of PETSc into windows using cygwin. I met with the following error at installation: $ ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe f90' --download-f-blas-lapack=1 LIBS="-L'/cygdrive/c/Program Files/Microsoft Visual Studio/DF98/LIB'" 'import site' failed; use -v for traceback Traceback (most recent call last): File "./config/configure.py", line 2, in import os ImportError: No module named os May I know why is this happening? Thank you very much. Regards. From knepley at gmail.com Mon Mar 24 20:26:50 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 24 Mar 2008 20:26:50 -0500 Subject: Can't run ./configure at installation In-Reply-To: <47E853DF.1090405@gmail.com> References: <47E853DF.1090405@gmail.com> Message-ID: Python is fundamentally broken. I recommend a reinstall. That 'import site' error indicates that Python cannot load its builtin library. Matt On Mon, Mar 24, 2008 at 8:22 PM, Ben Tay wrote: > Hi, > > I'm trying to install a new version of PETSc into windows using cygwin. > I met with the following error at installation: > > $ ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe f90' > --download-f-blas-lapack=1 LIBS="-L'/cygdrive/c/Program Files/Microsoft > Visual Studio/DF98/LIB'" > > > 'import site' failed; use -v for traceback > Traceback (most recent call last): > File "./config/configure.py", line 2, in > import os > ImportError: No module named os > > May I know why is this happening? > > Thank you very much. > > Regards. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From zonexo at gmail.com Mon Mar 24 20:57:06 2008 From: zonexo at gmail.com (Ben Tay) Date: Tue, 25 Mar 2008 09:57:06 +0800 Subject: Can't run ./configure at installation In-Reply-To: References: <47E853DF.1090405@gmail.com> Message-ID: <47E85BF2.3020209@gmail.com> Hmm, Tried to reinstall Python but it still can't work. Any other idea? Thanks again! Matthew Knepley wrote: > Python is fundamentally broken. I recommend a reinstall. That > 'import site' error indicates that Python cannot load its builtin library. > > Matt > > On Mon, Mar 24, 2008 at 8:22 PM, Ben Tay wrote: > >> Hi, >> >> I'm trying to install a new version of PETSc into windows using cygwin. >> I met with the following error at installation: >> >> $ ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe f90' >> --download-f-blas-lapack=1 LIBS="-L'/cygdrive/c/Program Files/Microsoft >> Visual Studio/DF98/LIB'" >> >> >> 'import site' failed; use -v for traceback >> Traceback (most recent call last): >> File "./config/configure.py", line 2, in >> import os >> ImportError: No module named os >> >> May I know why is this happening? >> >> Thank you very much. >> >> Regards. >> >> >> > > > > From knepley at gmail.com Mon Mar 24 21:12:15 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 24 Mar 2008 21:12:15 -0500 Subject: Can't run ./configure at installation In-Reply-To: <47E85BF2.3020209@gmail.com> References: <47E853DF.1090405@gmail.com> <47E85BF2.3020209@gmail.com> Message-ID: On Mon, Mar 24, 2008 at 8:57 PM, Ben Tay wrote: > Hmm, > > Tried to reinstall Python but it still can't work. Any other idea? In order to install you must get a working copy of Python. I suspect you have not fully uninstalled that copy. I would reinstall cygwin completely, since it appears that you have a corrupt copy. Matt > Thanks again! > > Matthew Knepley wrote: > > Python is fundamentally broken. I recommend a reinstall. That > > 'import site' error indicates that Python cannot load its builtin library. > > > > Matt > > > > On Mon, Mar 24, 2008 at 8:22 PM, Ben Tay wrote: > > > >> Hi, > >> > >> I'm trying to install a new version of PETSc into windows using cygwin. > >> I met with the following error at installation: > >> > >> $ ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe f90' > >> --download-f-blas-lapack=1 LIBS="-L'/cygdrive/c/Program Files/Microsoft > >> Visual Studio/DF98/LIB'" > >> > >> > >> 'import site' failed; use -v for traceback > >> Traceback (most recent call last): > >> File "./config/configure.py", line 2, in > >> import os > >> ImportError: No module named os > >> > >> May I know why is this happening? > >> > >> Thank you very much. > >> > >> Regards. > >> > >> > >> > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From bsmith at mcs.anl.gov Mon Mar 24 21:29:17 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 24 Mar 2008 21:29:17 -0500 Subject: Getting values of a parallel vector In-Reply-To: References: <47E7C973.40709@iesl.forth.gr> Message-ID: <27D7AD1F-E3EA-4047-A02C-4E125BDD4B7C@mcs.anl.gov> If you want all the values available on all the processes you can use VecScatterCreateToAll() to create your scatter context; but this is an extreme situation and usually does not make sense. Barry On Mar 24, 2008, at 10:59 AM, Satish Balay wrote: > On Mon, 24 Mar 2008, Jaime Suarez wrote: > >> Good morning, >> >> My question is about getting the values of a parallel vector in the >> PETSc subroutines using Fortran 90. >> >> As far as I know, I can invoke: >> CALL VecGetArrayF90 >> in order to get the full array that is being considered in each >> processor. However, I cannot employ this tool since I need to have >> also >> the values that are stored in other processors. Is there a straight >> way >> to get the value of the n-th element of a parallel vector, no matter >> where this element is stored? >> >> I am aware of the difficulty of this task, since it would involve >> transfer of information between different processors. >> >> Thanking you in advance, > > Check the manpage for VecScatterCreate(). You'll have to create > the VecScatter object corresponding to the off-proc values you > need - and then scatter them. > > After the scatter, the values will be available locally [Now you can > use VecGetArrayF90 to access the values] > > Satish > From mafunk at nmsu.edu Tue Mar 25 18:15:23 2008 From: mafunk at nmsu.edu (Matt Funk) Date: Tue, 25 Mar 2008 17:15:23 -0600 Subject: petsc commerical use In-Reply-To: References: <08BD23D9-0CF2-415E-88C0-328DF4D93E0F@columbia.edu> <200803181600.50369.mafunk@nmsu.edu> Message-ID: <200803251715.23821.mafunk@nmsu.edu> Sorry to bug the list with this. i was just wondering how the possible use of PETSC in a commerical environment is/should be handled? I read the copyright notification on the website as well as the FAQs, but i didn't see much to that question. If i am too blind to see it, maybe someone can point me to where i can read about it? thanks mat From knepley at gmail.com Tue Mar 25 18:17:29 2008 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Mar 2008 18:17:29 -0500 Subject: petsc commerical use In-Reply-To: <200803251715.23821.mafunk@nmsu.edu> References: <08BD23D9-0CF2-415E-88C0-328DF4D93E0F@columbia.edu> <200803181600.50369.mafunk@nmsu.edu> <200803251715.23821.mafunk@nmsu.edu> Message-ID: On Tue, Mar 25, 2008 at 6:15 PM, Matt Funk wrote: > Sorry to bug the list with this. > > i was just wondering how the possible use of PETSC in a commerical environment > is/should be handled? I read the copyright notification on the website as > well as the FAQs, but i didn't see much to that question. > > If i am too blind to see it, maybe someone can point me to where i can read > about it? I think the license is clear. You can use it for whatever commercial purpose you want. Its already in a few commercial codes. Thanks, Matt > thanks > mat -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From vijay.m at gmail.com Tue Mar 25 20:54:23 2008 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Tue, 25 Mar 2008 20:54:23 -0500 Subject: SNESSetApplicationContext usage In-Reply-To: Message-ID: <00ea01c88ee4$520caff0$383010ac@neutron> Hi, I am trying to use the SNESSetApplicationContext function to set an object as a user application context. This is being done after the SNESSolve was already called once on the SNES object, using a NULL application context. For some reason, after the SNESSetApplicationContext call, the application context on SNES is not being set. I am wondering if maybe I need to set some flag to tell the SNES object to include this new context in further residual and Jacobian calls. I saw the code for the SNESSetApplicationContext function and it doesn't look like it but may be I am missing something ? Thanks for the help. Vijay No virus found in this outgoing message. Checked by AVG. Version: 7.5.519 / Virus Database: 269.22.0/1342 - Release Date: 3/25/2008 10:26 AM From knepley at gmail.com Tue Mar 25 21:15:52 2008 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Mar 2008 21:15:52 -0500 Subject: SNESSetApplicationContext usage In-Reply-To: <00ea01c88ee4$520caff0$383010ac@neutron> References: <00ea01c88ee4$520caff0$383010ac@neutron> Message-ID: On Tue, Mar 25, 2008 at 8:54 PM, Vijay S. Mahadevan wrote: > Hi, > > I am trying to use the SNESSetApplicationContext function to set an object > as a user application context. This is being done after the SNESSolve was > already called once on the SNES object, using a NULL application context. > > For some reason, after the SNESSetApplicationContext call, the application > context on SNES is not being set. I am wondering if maybe I need to set some > flag to tell the SNES object to include this new context in further residual > and Jacobian calls. I saw the code for the SNESSetApplicationContext > function and it doesn't look like it but may be I am missing something ? There is a misunderstanding. The application context is separate from both the Function context and the Jacobian context. If you want to change these, you must call SetFunction/Jacobian() again. Matt > Thanks for the help. > > Vijay > > No virus found in this outgoing message. > Checked by AVG. > Version: 7.5.519 / Virus Database: 269.22.0/1342 - Release Date: 3/25/2008 > 10:26 AM > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From vijay.m at gmail.com Tue Mar 25 21:31:45 2008 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Tue, 25 Mar 2008 21:31:45 -0500 Subject: SNESSetApplicationContext usage In-Reply-To: Message-ID: <00eb01c88ee9$89bbce90$383010ac@neutron> Matt, > There is a misunderstanding. The application context is separate from both > the Function context and the Jacobian context. If the object completely belongs to the user, can I use this function to pass on pointers to my user objects ? Or is there some advanced usage that would conflict with this procedure ? Alternately I did try using SetFunction/Jacobian() already followed by SetFromOptions and it does pass the correct user context as expected. The glitch in this path is that even when I have -snes_mf, my Jacobian function is being called with a MFFD Mat type. And hence, I get an error there while trying to manipulate it. I should note that the first time I use my SNES object, the Jacobian is never called. It happens only after I set the Jacobian again. Please do let me know if the first method or the second one would be a good choice and the workaround needed to get it working. Thanks. Vijay > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] > On Behalf Of Matthew Knepley > Sent: Tuesday, March 25, 2008 9:16 PM > To: petsc-users at mcs.anl.gov > Subject: Re: SNESSetApplicationContext usage > > On Tue, Mar 25, 2008 at 8:54 PM, Vijay S. Mahadevan > wrote: > > Hi, > > > > I am trying to use the SNESSetApplicationContext function to set an > object > > as a user application context. This is being done after the SNESSolve > was > > already called once on the SNES object, using a NULL application > context. > > > > For some reason, after the SNESSetApplicationContext call, the > application > > context on SNES is not being set. I am wondering if maybe I need to set > some > > flag to tell the SNES object to include this new context in further > residual > > and Jacobian calls. I saw the code for the SNESSetApplicationContext > > function and it doesn't look like it but may be I am missing something > ? > > There is a misunderstanding. The application context is separate from both > the > Function context and the Jacobian context. If you want to change these, > you > must call SetFunction/Jacobian() again. > > Matt > > > Thanks for the help. > > > > Vijay > > > > No virus found in this outgoing message. > > Checked by AVG. > > Version: 7.5.519 / Virus Database: 269.22.0/1342 - Release Date: > 3/25/2008 > > 10:26 AM > > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > No virus found in this incoming message. > Checked by AVG. > Version: 7.5.519 / Virus Database: 269.22.0/1342 - Release Date: 3/25/2008 > 10:26 AM > No virus found in this outgoing message. Checked by AVG. Version: 7.5.519 / Virus Database: 269.22.0/1342 - Release Date: 3/25/2008 10:26 AM From dave at vpac.org Tue Mar 25 21:39:30 2008 From: dave at vpac.org (Dave Lee) Date: Wed, 26 Mar 2008 13:39:30 +1100 (EST) Subject: snes querie In-Reply-To: <1096837561.44301206498908140.JavaMail.root@zimbra.vpac.org> Message-ID: <1806855440.44541206499170652.JavaMail.root@zimbra.vpac.org> Hello PETSc, I'm trying to solve a non-linear problem using the SNES routines, and i've been finding that for every even time step the solution converges as I'd expect, however for every odd time step, the solution diverges. I've run an additional test problem for which the solution diverges on the second time step only (ie: converges for every successive time step), so I'm pretty sure its not a odd/even logic problem in my code. The output (for the initial guess and first two time steps) when I run with the flags -info -snes_monitor is as follows: [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 9.34921e-14 is less than relative tolerance 1e-05 times initial right hand side norm 48.7135 at iteration 1 [0] SNESSolve_LS(): iter=0, linear solve iterations=1 [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 112.285 near zero implies inconsistent rhs [0] SNESLineSearchCubic(): Using full step [0] SNESSolve_LS(): fnorm=3.0170114380870206e+01, gnorm=1.0142108262591121e-12, ynorm=4.8713530783010192e+01, lssucceed=1 1 SNES Function norm 1.014210826259e-12 [0] SNESDefaultConverged(): Converged due to function norm 1.01421e-12 < 3.01701e-07 (relative tolerance) Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE [0] SNESSolve_LS(): iter=0, linear solve iterations=8 [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 0.925505 near zero implies inconsistent rhs [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430941e-02 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430942e-03 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430942e-04 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430944e-05 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430946e-06 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430949e-07 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430952e-08 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430954e-09 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430954e-10 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430954e-11 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430954e-12 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430954e-13 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430956e-14 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430958e-15 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430960e-16 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430961e-17 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430961e-18 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430962e-19 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430962e-20 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430964e-21 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430967e-22 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430969e-23 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430971e-24 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430971e-25 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430974e-26 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430976e-27 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430979e-28 [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, lambda=2.6904093002430979e-29 [0] SNESLineSearchCubic(): Unable to find good step length! 29 [0] SNESLineSearchCubic(): fnorm=9.6575221913187848e-01, gnorm=2.0934668664952425e+00, ynorm=5.1772194669712386e+00, lambda=2.6904093002430979e-29, initial slope=-9.3267734652255951e-01 [0] SNESSolve_LS(): fnorm=9.6575221913187848e-01, gnorm=2.0934668664952425e+00, ynorm=5.1772194669712386e+00, lssucceed=0 1 SNES Function norm 2.093466866495e+00 [0] SNESLSCheckLocalMin_Private(): || J^T F|| 107.04 near zero implies found a local minimum Nonlinear solve did not converge due to DIVERGED_LS_FAILURE 0 SNES Function norm 2.185496756061e+00 [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 2.29435e-06 is less than relative tolerance 1e-05 times initial right hand side norm 3.74384 at iteration 8 [0] SNESSolve_LS(): iter=0, linear solve iterations=8 [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 119.216 near zero implies inconsistent rhs [0] SNESLineSearchCubic(): Using full step [0] SNESSolve_LS(): fnorm=2.1854967560609060e+00, gnorm=1.1219829200815006e+00, ynorm=6.0664603876861261e+00, lssucceed=1 1 SNES Function norm 1.121982920082e+00 [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 1.49325e-06 is less than relative tolerance 1e-05 times initial right hand side norm 1.54195 at iteration 7 [0] SNESSolve_LS(): iter=1, linear solve iterations=7 [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.58793 near zero implies inconsistent rhs [0] SNESLineSearchCubic(): Using full step [0] SNESSolve_LS(): fnorm=1.1219829200815006e+00, gnorm=2.6934556136855723e-02, ynorm=1.5405251272601992e+00, lssucceed=1 2 SNES Function norm 2.693455613686e-02 [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 3.31729e-08 is less than relative tolerance 1e-05 times initial right hand side norm 0.0334934 at iteration 7 [0] SNESSolve_LS(): iter=2, linear solve iterations=7 [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.36601 near zero implies inconsistent rhs [0] SNESLineSearchCubic(): Using full step [0] SNESSolve_LS(): fnorm=2.6934556136855723e-02, gnorm=6.6982884106362914e-04, ynorm=3.3460040092452174e-02, lssucceed=1 3 SNES Function norm 6.698288410636e-04 [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 7.22802e-10 is less than relative tolerance 1e-05 times initial right hand side norm 0.000832349 at iteration 7 [0] SNESSolve_LS(): iter=3, linear solve iterations=7 [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.30347 near zero implies inconsistent rhs [0] SNESLineSearchCubic(): Using full step [0] SNESSolve_LS(): fnorm=6.6982884106362914e-04, gnorm=1.7317637296224453e-05, ynorm=8.3135074665742882e-04, lssucceed=1 4 SNES Function norm 1.731763729622e-05 [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 1.77439e-11 is less than relative tolerance 1e-05 times initial right hand side norm 2.04787e-05 at iteration 7 [0] SNESSolve_LS(): iter=4, linear solve iterations=7 [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.14134 near zero implies inconsistent rhs [0] SNESLineSearchCubic(): Using full step [0] SNESSolve_LS(): fnorm=1.7317637296224453e-05, gnorm=4.5421644145512881e-07, ynorm=2.0457661052256775e-05, lssucceed=1 5 SNES Function norm 4.542164414551e-07 [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 4.11491e-13 is less than relative tolerance 1e-05 times initial right hand side norm 5.14482e-07 at iteration 7 [0] SNESSolve_LS(): iter=5, linear solve iterations=7 [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.24632 near zero implies inconsistent rhs [0] SNESLineSearchCubic(): Using full step [0] SNESSolve_LS(): fnorm=4.5421644145512881e-07, gnorm=1.1954313416752086e-08, ynorm=5.1403324849206651e-07, lssucceed=1 6 SNES Function norm 1.195431341675e-08 [0] SNESDefaultConverged(): Converged due to function norm 1.19543e-08 < 2.1855e-08 (relative tolerance) Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE I'm not sure what to make of the statement: SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 112.285 near zero implies inconsistent rhs which arises for both the converged time steps and the unresolved time steps - should I be concerned about this? Also at the point where the SNESComputeFunction call is repeatedly made (ls.c:645) during the unresolved time steps, the system is not actually being solved, as the condition on lambda is not being met, so the SNESLineSearchCubic function just keeps calling the residual assembly function. Any ideas what all of this means? cheers, dave. From vijay.m at gmail.com Tue Mar 25 21:57:20 2008 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Tue, 25 Mar 2008 21:57:20 -0500 Subject: SNESSetApplicationContext usage In-Reply-To: Message-ID: <00ec01c88eed$1d5d0d50$383010ac@neutron> Matt, The first method from my previous email works fine. If I use SNES(Set/Get)ApplicationContext, I can store and retrieve user objects from the SNES object. And this is what I wanted. Thanks for the help and clarifying my mistake. Vijay > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] > On Behalf Of Matthew Knepley > Sent: Tuesday, March 25, 2008 9:16 PM > To: petsc-users at mcs.anl.gov > Subject: Re: SNESSetApplicationContext usage > > On Tue, Mar 25, 2008 at 8:54 PM, Vijay S. Mahadevan > wrote: > > Hi, > > > > I am trying to use the SNESSetApplicationContext function to set an > object > > as a user application context. This is being done after the SNESSolve > was > > already called once on the SNES object, using a NULL application > context. > > > > For some reason, after the SNESSetApplicationContext call, the > application > > context on SNES is not being set. I am wondering if maybe I need to set > some > > flag to tell the SNES object to include this new context in further > residual > > and Jacobian calls. I saw the code for the SNESSetApplicationContext > > function and it doesn't look like it but may be I am missing something > ? > > There is a misunderstanding. The application context is separate from both > the > Function context and the Jacobian context. If you want to change these, > you > must call SetFunction/Jacobian() again. > > Matt > > > Thanks for the help. > > > > Vijay > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > No virus found in this outgoing message. Checked by AVG. Version: 7.5.519 / Virus Database: 269.22.0/1342 - Release Date: 3/25/2008 10:26 AM From zonexo at gmail.com Wed Mar 26 01:35:48 2008 From: zonexo at gmail.com (Ben Tay) Date: Wed, 26 Mar 2008 14:35:48 +0800 Subject: Can't run ./configure at installation In-Reply-To: References: <47E853DF.1090405@gmail.com> <47E85BF2.3020209@gmail.com> Message-ID: <47E9EEC4.1090106@gmail.com> Hi, I guess it's because I've another copy of Python for windows installed by Tecplot. Uninstalled it and reinstall my cygwin. However I got this msg: ================================================================================ = Configuring PETSc to compile on your system ================================================================================ = ================================================================================ = *** Incomplete cygwin install detected **************************************** * *** Please rerun cygwin-setup and install module make [and its dependencies]*** * ================================================================================ = I run "make" and it seems to be. What do you mean by dependencies? I just select "gnu make" in the developer's installation of cygwin (and python 8000+k). That's what I've been using and it worked last time. I wonder what other additions must I install? Thank you. Matthew Knepley wrote: > On Mon, Mar 24, 2008 at 8:57 PM, Ben Tay wrote: > >> Hmm, >> >> Tried to reinstall Python but it still can't work. Any other idea? >> > > In order to install you must get a working copy of Python. I suspect you have > not fully uninstalled that copy. I would reinstall cygwin completely, since it > appears that you have a corrupt copy. > > Matt > > >> Thanks again! >> >> Matthew Knepley wrote: >> > Python is fundamentally broken. I recommend a reinstall. That >> > 'import site' error indicates that Python cannot load its builtin library. >> > >> > Matt >> > >> > On Mon, Mar 24, 2008 at 8:22 PM, Ben Tay wrote: >> > >> >> Hi, >> >> >> >> I'm trying to install a new version of PETSc into windows using cygwin. >> >> I met with the following error at installation: >> >> >> >> $ ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe f90' >> >> --download-f-blas-lapack=1 LIBS="-L'/cygdrive/c/Program Files/Microsoft >> >> Visual Studio/DF98/LIB'" >> >> >> >> >> >> 'import site' failed; use -v for traceback >> >> Traceback (most recent call last): >> >> File "./config/configure.py", line 2, in >> >> import os >> >> ImportError: No module named os >> >> >> >> May I know why is this happening? >> >> >> >> Thank you very much. >> >> >> >> Regards. >> >> >> >> >> >> >> > >> > >> > >> > >> >> >> > > > > From bsmith at mcs.anl.gov Wed Mar 26 05:54:18 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 26 Mar 2008 06:54:18 -0400 Subject: snes querie In-Reply-To: <1806855440.44541206499170652.JavaMail.root@zimbra.vpac.org> References: <1806855440.44541206499170652.JavaMail.root@zimbra.vpac.org> Message-ID: <2D634C67-FEA1-4F7D-AFF8-3DEA8219817B@mcs.anl.gov> The number one reason Newton's method does not converge is due to an error in a supplied Jacobian (how is your Jacobian computed?). You can *first run with -snes_type test and it will attempt to determine if your Jacobian is correct * if that works then run with -snes_mf_operator, what happens then, does the Newton converge? The number two reason Newton's method does not converge is due to an error in the function evaluation. This is harder for us to help track down because it is pure user code. What I suggest doing is trying to run on the smallest size problem possible (use a tiny grid), what happens? The number three reason Newton's method does not converge is due to too poor an initial guess. If doing time-stepping you can try cutting the time- step back until you get Newton converged. Good luck, Barry On Mar 25, 2008, at 10:39 PM, Dave Lee wrote: > Hello PETSc, > > I'm trying to solve a non-linear problem using the SNES routines, > and i've been > finding that for every even time step the solution converges as I'd > expect, however > for every odd time step, the solution diverges. I've run an > additional test problem > for which the solution diverges on the second time step only (ie: > converges for every > successive time step), so I'm pretty sure its not a odd/even logic > problem in my > code. The output (for the initial guess and first two time steps) > when I run with > the flags -info -snes_monitor is as follows: > > > [0] KSPDefaultConverged(): Linear solver has converged. Residual > norm 9.34921e-14 is less than relative tolerance 1e-05 times initial > right hand side norm 48.7135 at iteration 1 > [0] SNESSolve_LS(): iter=0, linear solve iterations=1 > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 112.285 > near zero implies inconsistent rhs > [0] SNESLineSearchCubic(): Using full step > [0] SNESSolve_LS(): fnorm=3.0170114380870206e+01, > gnorm=1.0142108262591121e-12, ynorm=4.8713530783010192e+01, > lssucceed=1 > 1 SNES Function norm 1.014210826259e-12 > [0] SNESDefaultConverged(): Converged due to function norm > 1.01421e-12 < 3.01701e-07 (relative tolerance) > Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE > [0] SNESSolve_LS(): iter=0, linear solve iterations=8 > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 0.925505 > near zero implies inconsistent rhs > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430941e-02 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430942e-03 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430942e-04 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430944e-05 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430946e-06 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430949e-07 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430952e-08 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430954e-09 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430954e-10 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430954e-11 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430954e-12 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430954e-13 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430956e-14 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430958e-15 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430960e-16 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430961e-17 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430961e-18 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430962e-19 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430962e-20 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430964e-21 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430967e-22 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430969e-23 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430971e-24 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430971e-25 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430974e-26 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430976e-27 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430979e-28 > [0] SNESLineSearchCubic(): Cubic step no good, shrinking lambda, > lambda=2.6904093002430979e-29 > [0] SNESLineSearchCubic(): Unable to find good step length! 29 > [0] SNESLineSearchCubic(): fnorm=9.6575221913187848e-01, > gnorm=2.0934668664952425e+00, ynorm=5.1772194669712386e+00, > lambda=2.6904093002430979e-29, initial slope=-9.3267734652255951e-01 > [0] SNESSolve_LS(): fnorm=9.6575221913187848e-01, > gnorm=2.0934668664952425e+00, ynorm=5.1772194669712386e+00, > lssucceed=0 > 1 SNES Function norm 2.093466866495e+00 > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 107.04 near zero > implies found a local minimum > Nonlinear solve did not converge due to DIVERGED_LS_FAILURE > 0 SNES Function norm 2.185496756061e+00 > [0] KSPDefaultConverged(): Linear solver has converged. Residual > norm 2.29435e-06 is less than relative tolerance 1e-05 times initial > right hand side norm 3.74384 at iteration 8 > [0] SNESSolve_LS(): iter=0, linear solve iterations=8 > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 119.216 > near zero implies inconsistent rhs > [0] SNESLineSearchCubic(): Using full step > [0] SNESSolve_LS(): fnorm=2.1854967560609060e+00, > gnorm=1.1219829200815006e+00, ynorm=6.0664603876861261e+00, > lssucceed=1 > 1 SNES Function norm 1.121982920082e+00 > [0] KSPDefaultConverged(): Linear solver has converged. Residual > norm 1.49325e-06 is less than relative tolerance 1e-05 times initial > right hand side norm 1.54195 at iteration 7 > [0] SNESSolve_LS(): iter=1, linear solve iterations=7 > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.58793 > near zero implies inconsistent rhs > [0] SNESLineSearchCubic(): Using full step > [0] SNESSolve_LS(): fnorm=1.1219829200815006e+00, > gnorm=2.6934556136855723e-02, ynorm=1.5405251272601992e+00, > lssucceed=1 > 2 SNES Function norm 2.693455613686e-02 > [0] KSPDefaultConverged(): Linear solver has converged. Residual > norm 3.31729e-08 is less than relative tolerance 1e-05 times initial > right hand side norm 0.0334934 at iteration 7 > [0] SNESSolve_LS(): iter=2, linear solve iterations=7 > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.36601 > near zero implies inconsistent rhs > [0] SNESLineSearchCubic(): Using full step > [0] SNESSolve_LS(): fnorm=2.6934556136855723e-02, > gnorm=6.6982884106362914e-04, ynorm=3.3460040092452174e-02, > lssucceed=1 > 3 SNES Function norm 6.698288410636e-04 > [0] KSPDefaultConverged(): Linear solver has converged. Residual > norm 7.22802e-10 is less than relative tolerance 1e-05 times initial > right hand side norm 0.000832349 at iteration 7 > [0] SNESSolve_LS(): iter=3, linear solve iterations=7 > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.30347 > near zero implies inconsistent rhs > [0] SNESLineSearchCubic(): Using full step > [0] SNESSolve_LS(): fnorm=6.6982884106362914e-04, > gnorm=1.7317637296224453e-05, ynorm=8.3135074665742882e-04, > lssucceed=1 > 4 SNES Function norm 1.731763729622e-05 > [0] KSPDefaultConverged(): Linear solver has converged. Residual > norm 1.77439e-11 is less than relative tolerance 1e-05 times initial > right hand side norm 2.04787e-05 at iteration 7 > [0] SNESSolve_LS(): iter=4, linear solve iterations=7 > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.14134 > near zero implies inconsistent rhs > [0] SNESLineSearchCubic(): Using full step > [0] SNESSolve_LS(): fnorm=1.7317637296224453e-05, > gnorm=4.5421644145512881e-07, ynorm=2.0457661052256775e-05, > lssucceed=1 > 5 SNES Function norm 4.542164414551e-07 > [0] KSPDefaultConverged(): Linear solver has converged. Residual > norm 4.11491e-13 is less than relative tolerance 1e-05 times initial > right hand side norm 5.14482e-07 at iteration 7 > [0] SNESSolve_LS(): iter=5, linear solve iterations=7 > [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1.24632 > near zero implies inconsistent rhs > [0] SNESLineSearchCubic(): Using full step > [0] SNESSolve_LS(): fnorm=4.5421644145512881e-07, > gnorm=1.1954313416752086e-08, ynorm=5.1403324849206651e-07, > lssucceed=1 > 6 SNES Function norm 1.195431341675e-08 > [0] SNESDefaultConverged(): Converged due to function norm > 1.19543e-08 < 2.1855e-08 (relative tolerance) > Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE > > > I'm not sure what to make of the statement: > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 112.285 near > zero implies inconsistent rhs > which arises for both the converged time steps and the unresolved > time steps - should I be concerned > about this? > > Also at the point where the SNESComputeFunction call is repeatedly > made (ls.c:645) during the unresolved > time steps, the system is not actually being solved, as the > condition on lambda is not being met, so > the SNESLineSearchCubic function just keeps calling the residual > assembly function. > > Any ideas what all of this means? > > cheers, dave. > From balay at mcs.anl.gov Wed Mar 26 08:01:54 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 26 Mar 2008 08:01:54 -0500 (CDT) Subject: Can't run ./configure at installation In-Reply-To: <47E9EEC4.1090106@gmail.com> References: <47E853DF.1090405@gmail.com> <47E85BF2.3020209@gmail.com> <47E9EEC4.1090106@gmail.com> Message-ID: what do you have for: which make which diff Satish On Wed, 26 Mar 2008, Ben Tay wrote: > Hi, > > I guess it's because I've another copy of Python for windows installed by > Tecplot. Uninstalled it and reinstall my cygwin. However I got this msg: > > ================================================================================ > = > Configuring PETSc to compile on your system > > ================================================================================ > = > ================================================================================ > = > *** Incomplete cygwin install detected > **************************************** > * > *** Please rerun cygwin-setup and install module make [and its > dependencies]*** > * > ================================================================================ > = > > I run "make" and it seems to be. What do you mean by dependencies? I just > select "gnu make" in the developer's installation of cygwin (and python > 8000+k). That's what I've been using and it worked last time. I wonder what > other additions must I install? > > Thank you. > > > > > Matthew Knepley wrote: > > On Mon, Mar 24, 2008 at 8:57 PM, Ben Tay wrote: > > > > > Hmm, > > > > > > Tried to reinstall Python but it still can't work. Any other idea? > > > > > > > In order to install you must get a working copy of Python. I suspect you > > have > > not fully uninstalled that copy. I would reinstall cygwin completely, since > > it > > appears that you have a corrupt copy. > > > > Matt > > > > > > > Thanks again! > > > > > > Matthew Knepley wrote: > > > > Python is fundamentally broken. I recommend a reinstall. That > > > > 'import site' error indicates that Python cannot load its builtin > > > library. > > > > > > > > Matt > > > > > > > > On Mon, Mar 24, 2008 at 8:22 PM, Ben Tay wrote: > > > > > > > >> Hi, > > > >> > > > >> I'm trying to install a new version of PETSc into windows using > > > cygwin. > > > >> I met with the following error at installation: > > > >> > > > >> $ ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe > > > f90' > > > >> --download-f-blas-lapack=1 LIBS="-L'/cygdrive/c/Program > > > Files/Microsoft > > > >> Visual Studio/DF98/LIB'" > > > >> > > > >> > > > >> 'import site' failed; use -v for traceback > > > >> Traceback (most recent call last): > > > >> File "./config/configure.py", line 2, in > > > >> import os > > > >> ImportError: No module named os > > > >> > > > >> May I know why is this happening? > > > >> > > > >> Thank you very much. > > > >> > > > >> Regards. > > > >> > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From mafunk at nmsu.edu Wed Mar 26 15:31:01 2008 From: mafunk at nmsu.edu (Matt Funk) Date: Wed, 26 Mar 2008 14:31:01 -0600 Subject: matrix memory allocation In-Reply-To: References: <08BD23D9-0CF2-415E-88C0-328DF4D93E0F@columbia.edu> <200803181600.50369.mafunk@nmsu.edu> Message-ID: <200803261431.01327.mafunk@nmsu.edu> Hi, in order to create a sparse MPIMatrix (with preallocated memory) it is necessary to give it the number of nonzero entries per row for the local and off-diagonal submatrix. The way i do things right now is that i need to allocate the entire matrix. However, my domain is decomposed into boxes where one or more boxes reside on the current processor. So, i was wondering if it is possible to allocate the matrix memory per box. Right now what i do is this: 1) iterate over the whole domain (every single box) and extract the info i need from each box for memory allocation. 2) call MatCreateMPIAIJ to create the matrix/allocate memory. 3) iterate over the whole domain again to get the info to insert the values into the matrix. Is it possible to instead do something like: 1) get info for values and memory allocation needed per box 2) allocate memory for x number of rows in global matrix and insert values for them (via MatSetValues) 3) start at 1) till every box in the domain is covered. The problem in 2) is that i can insert the values, but there is no memory preallocated for it, so it would be slow. If i could allocate it, i think it would solve my problem. I hope i am making sense. By the way, i cannot use the DA construct because i have overlapping values. thanks mat From vijay.m at gmail.com Wed Mar 26 16:14:05 2008 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Wed, 26 Mar 2008 16:14:05 -0500 Subject: problem with -snes_mf_operator option Message-ID: Hi all, I have a nonlinear solver test code that can use a user specified Jacobian function directly, set using SNESSetJacobian. Now if I use -snes_mf option, the solution scheme depends only on the residual and the convergence is equivalent to the case when using the real Jacobian matrix itself. But if I try the -snes_mf_operator option, with the Jacobian=NULL to start with and the preconditioner being correctly allocated the memory needed, I end up with a NULL argument exception. I also alternately set the preconditioner to Identity matrix in order to avoid any problems from user code to trickling in, but to no avail. Here's the call stack. [0]PETSC ERROR: Null argument, when expecting valid pointer! [0]PETSC ERROR: Null Object: Parameter # 1! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 7, Fri Oct 26 14:21:35 CDT 2007 HG revision: 2e223033ba960114833e1f9713ab393ec78c056f [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: bin/test-dbg on a linux-gnu named grove by vijaysm Wed Mar 26 16:06:45 2008 [0]PETSC ERROR: Libraries linked from /state/partition1/local/petsc-2.3.3-p7/lib/linux-gnu-c-debug [0]PETSC ERROR: Configure run at Fri Nov 9 15:02:46 2007 [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort --with-cxx=g++ --download-f-blas-lapack=1 --download-mpich=1 --with-shared=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c [0]PETSC ERROR: MatMult_MFFD() line 294 in src/mat/impls/mffd/mffd.c [0]PETSC ERROR: MatMult() line 1632 in src/mat/interface/matrix.c [0]PETSC ERROR: PCApplyBAorAB() line 584 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: GMREScycle() line 159 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve() line 379 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: SNES_KSPSolve() line 2578 in src/snes/interface/snes.c [0]PETSC ERROR: SNESSolve_LS() line 184 in src/snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve() line 1871 in src/snes/interface/snes.c >From what I can understand from the PETSc code (snes.c, mffd.c), the current_u for MFFD context is NULL here. And that is the reason for the final exception. But I do not understand how this can be since it works perfectly fine with -snes_mf option with no changes in the code. Please do let me know if you think I should specifically be looking for something to tackle this issue. Any help will be appreciated. Thanks, Vijay From balay at mcs.anl.gov Wed Mar 26 16:45:26 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 26 Mar 2008 16:45:26 -0500 (CDT) Subject: problem with -snes_mf_operator option In-Reply-To: References: Message-ID: Is this usage from fortran? There was a bug in fortran interface with 'MatStructure' that caused similar problems from fortran. [the fix was updated bfort]. If this is indeed the case for you - I'll respin 2.3.3 tarball with the the updated bfort. Satish On Wed, 26 Mar 2008, Vijay S. Mahadevan wrote: > Hi all, > > I have a nonlinear solver test code that can use a user specified > Jacobian function directly, set using SNESSetJacobian. Now if I use > -snes_mf option, the solution scheme depends only on the residual and > the convergence is equivalent to the case when using the real Jacobian > matrix itself. But if I try the -snes_mf_operator option, with the > Jacobian=NULL to start with and the preconditioner being correctly > allocated the memory needed, I end up with a NULL argument exception. > I also alternately set the preconditioner to Identity matrix in order > to avoid any problems from user code to trickling in, but to no avail. > > Here's the call stack. > > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 7, Fri Oct 26 > 14:21:35 CDT 2007 HG revision: > 2e223033ba960114833e1f9713ab393ec78c056f > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: bin/test-dbg on a linux-gnu named grove by vijaysm Wed > Mar 26 16:06:45 2008 > [0]PETSC ERROR: Libraries linked from > /state/partition1/local/petsc-2.3.3-p7/lib/linux-gnu-c-debug > [0]PETSC ERROR: Configure run at Fri Nov 9 15:02:46 2007 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort > --with-cxx=g++ --download-f-blas-lapack=1 --download-mpich=1 > --with-shared=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c > [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c > [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c > [0]PETSC ERROR: MatMult_MFFD() line 294 in src/mat/impls/mffd/mffd.c > [0]PETSC ERROR: MatMult() line 1632 in src/mat/interface/matrix.c > [0]PETSC ERROR: PCApplyBAorAB() line 584 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: GMREScycle() line 159 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve() line 379 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: SNES_KSPSolve() line 2578 in src/snes/interface/snes.c > [0]PETSC ERROR: SNESSolve_LS() line 184 in src/snes/impls/ls/ls.c > [0]PETSC ERROR: SNESSolve() line 1871 in src/snes/interface/snes.c > > From what I can understand from the PETSc code (snes.c, mffd.c), the > current_u for MFFD context is NULL here. And that is the reason for > the final exception. But I do not understand how this can be since it > works perfectly fine with -snes_mf option with no changes in the code. > > Please do let me know if you think I should specifically be looking > for something to tackle this issue. Any help will be appreciated. > > Thanks, > Vijay > > From vijay.m at gmail.com Wed Mar 26 16:51:11 2008 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Wed, 26 Mar 2008 16:51:11 -0500 Subject: problem with -snes_mf_operator option In-Reply-To: References: Message-ID: Satish, No. I am using C++. I currently have my MatStructure flag set to DIFFERENT_NONZERO_PATTERN since I am using the identity matrix for the preconditioner in order to eliminate any issues from user code. I'd be interested in knowing what the fix was and can test if that works in my case. Do let me know if you have any ideas. Vijay On Wed, Mar 26, 2008 at 4:45 PM, Satish Balay wrote: > Is this usage from fortran? > > There was a bug in fortran interface with 'MatStructure' that caused > similar problems from fortran. [the fix was updated bfort]. > > If this is indeed the case for you - I'll respin 2.3.3 tarball with > the the updated bfort. > > Satish > > > > On Wed, 26 Mar 2008, Vijay S. Mahadevan wrote: > > > Hi all, > > > > I have a nonlinear solver test code that can use a user specified > > Jacobian function directly, set using SNESSetJacobian. Now if I use > > -snes_mf option, the solution scheme depends only on the residual and > > the convergence is equivalent to the case when using the real Jacobian > > matrix itself. But if I try the -snes_mf_operator option, with the > > Jacobian=NULL to start with and the preconditioner being correctly > > allocated the memory needed, I end up with a NULL argument exception. > > I also alternately set the preconditioner to Identity matrix in order > > to avoid any problems from user code to trickling in, but to no avail. > > > > Here's the call stack. > > > > [0]PETSC ERROR: Null argument, when expecting valid pointer! > > [0]PETSC ERROR: Null Object: Parameter # 1! > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 7, Fri Oct 26 > > 14:21:35 CDT 2007 HG revision: > > 2e223033ba960114833e1f9713ab393ec78c056f > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: bin/test-dbg on a linux-gnu named grove by vijaysm Wed > > Mar 26 16:06:45 2008 > > [0]PETSC ERROR: Libraries linked from > > /state/partition1/local/petsc-2.3.3-p7/lib/linux-gnu-c-debug > > [0]PETSC ERROR: Configure run at Fri Nov 9 15:02:46 2007 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort > > --with-cxx=g++ --download-f-blas-lapack=1 --download-mpich=1 > > --with-shared=1 > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c > > [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c > > [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c > > [0]PETSC ERROR: MatMult_MFFD() line 294 in src/mat/impls/mffd/mffd.c > > [0]PETSC ERROR: MatMult() line 1632 in src/mat/interface/matrix.c > > [0]PETSC ERROR: PCApplyBAorAB() line 584 in src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: GMREScycle() line 159 in src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: KSPSolve() line 379 in src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: SNES_KSPSolve() line 2578 in src/snes/interface/snes.c > > [0]PETSC ERROR: SNESSolve_LS() line 184 in src/snes/impls/ls/ls.c > > [0]PETSC ERROR: SNESSolve() line 1871 in src/snes/interface/snes.c > > > > From what I can understand from the PETSc code (snes.c, mffd.c), the > > current_u for MFFD context is NULL here. And that is the reason for > > the final exception. But I do not understand how this can be since it > > works perfectly fine with -snes_mf option with no changes in the code. > > > > Please do let me know if you think I should specifically be looking > > for something to tackle this issue. Any help will be appreciated. > > > > Thanks, > > Vijay > > > > > > From mafunk at nmsu.edu Wed Mar 26 17:01:43 2008 From: mafunk at nmsu.edu (Matt Funk) Date: Wed, 26 Mar 2008 16:01:43 -0600 Subject: matrix memory allocation In-Reply-To: <200803261431.01327.mafunk@nmsu.edu> References: <08BD23D9-0CF2-415E-88C0-328DF4D93E0F@columbia.edu> <200803261431.01327.mafunk@nmsu.edu> Message-ID: <200803261601.43931.mafunk@nmsu.edu> Maybe i should clarify a little more. I guess what i am looking for is something that lets me split up the memory allocation within a given processor. The example you guys often show (for example on MatCreateMPIAIJ manual page) is where the local process owns 3 rows. But consider for example if the local process owns 1000 rows. Right now (as i can see it) i have two options. Do not allocate memory at all and use MatSetValues and allocate memory per row. This is very slow as you guys state on your manual pages. The other option is to allocate all 1000 rows at a time which means extra overhead. However, what i would like to be able to (as an example), make 2 calls for an allocation of 200 rows and a second call for an allocation of the remaining 800 rows. I do not know if this is possible as i don't know whether all the local stuff is stored in on continuous array? Anyway, i hope this clarifies my question a little more. thank mat On Wednesday 26 March 2008 14:31, Matt Funk wrote: > Hi, > > in order to create a sparse MPIMatrix (with preallocated memory) it is > necessary to give it the number of nonzero entries per row for the local > and off-diagonal submatrix. > > The way i do things right now is that i need to allocate the entire matrix. > However, my domain is decomposed into boxes where one or more boxes reside > on the current processor. > > So, i was wondering if it is possible to allocate the matrix memory per > box. Right now what i do is this: > 1) iterate over the whole domain (every single box) and extract the info i > need from each box for memory allocation. > 2) call MatCreateMPIAIJ to create the matrix/allocate memory. > 3) iterate over the whole domain again to get the info to insert the values > into the matrix. > > Is it possible to instead do something like: > 1) get info for values and memory allocation needed per box > 2) allocate memory for x number of rows in global matrix and insert values > for them (via MatSetValues) > 3) start at 1) till every box in the domain is covered. > > The problem in 2) is that i can insert the values, but there is no memory > preallocated for it, so it would be slow. If i could allocate it, i think > it would solve my problem. > > I hope i am making sense. By the way, i cannot use the DA construct because > i have overlapping values. > > thanks > mat From knepley at gmail.com Wed Mar 26 18:01:36 2008 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 26 Mar 2008 18:01:36 -0500 Subject: problem with -snes_mf_operator option In-Reply-To: References: Message-ID: I am not sure what is going on with your code, but I have just tested an example and it works. I recommend you try cd src/snes/examples/tutorials make ex5 ./ex5 -snes_monitor -snes_view ./ex5 -snes_monitor -snes_view -snes_mf ./ex5 -snes_monitor -snes_view -snes_mf_operator Thanks, Matt On Wed, Mar 26, 2008 at 4:14 PM, Vijay S. Mahadevan wrote: > Hi all, > > I have a nonlinear solver test code that can use a user specified > Jacobian function directly, set using SNESSetJacobian. Now if I use > -snes_mf option, the solution scheme depends only on the residual and > the convergence is equivalent to the case when using the real Jacobian > matrix itself. But if I try the -snes_mf_operator option, with the > Jacobian=NULL to start with and the preconditioner being correctly > allocated the memory needed, I end up with a NULL argument exception. > I also alternately set the preconditioner to Identity matrix in order > to avoid any problems from user code to trickling in, but to no avail. > > Here's the call stack. > > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 7, Fri Oct 26 > 14:21:35 CDT 2007 HG revision: > 2e223033ba960114833e1f9713ab393ec78c056f > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: bin/test-dbg on a linux-gnu named grove by vijaysm Wed > Mar 26 16:06:45 2008 > [0]PETSC ERROR: Libraries linked from > /state/partition1/local/petsc-2.3.3-p7/lib/linux-gnu-c-debug > [0]PETSC ERROR: Configure run at Fri Nov 9 15:02:46 2007 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort > --with-cxx=g++ --download-f-blas-lapack=1 --download-mpich=1 > --with-shared=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c > [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c > [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c > [0]PETSC ERROR: MatMult_MFFD() line 294 in src/mat/impls/mffd/mffd.c > [0]PETSC ERROR: MatMult() line 1632 in src/mat/interface/matrix.c > [0]PETSC ERROR: PCApplyBAorAB() line 584 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: GMREScycle() line 159 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve() line 379 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: SNES_KSPSolve() line 2578 in src/snes/interface/snes.c > [0]PETSC ERROR: SNESSolve_LS() line 184 in src/snes/impls/ls/ls.c > [0]PETSC ERROR: SNESSolve() line 1871 in src/snes/interface/snes.c > > From what I can understand from the PETSc code (snes.c, mffd.c), the > current_u for MFFD context is NULL here. And that is the reason for > the final exception. But I do not understand how this can be since it > works perfectly fine with -snes_mf option with no changes in the code. > > Please do let me know if you think I should specifically be looking > for something to tackle this issue. Any help will be appreciated. > > Thanks, > Vijay > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From knepley at gmail.com Wed Mar 26 18:07:06 2008 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 26 Mar 2008 18:07:06 -0500 Subject: matrix memory allocation In-Reply-To: <200803261601.43931.mafunk@nmsu.edu> References: <08BD23D9-0CF2-415E-88C0-328DF4D93E0F@columbia.edu> <200803261431.01327.mafunk@nmsu.edu> <200803261601.43931.mafunk@nmsu.edu> Message-ID: This is not possible with the current implementation. I am not sure why you would want to do it. It seems to me to save nothing. Do you have some sort of performance problem? Matt On Wed, Mar 26, 2008 at 5:01 PM, Matt Funk wrote: > Maybe i should clarify a little more. > I guess what i am looking for is something that lets me split up the memory > allocation within a given processor. > > The example you guys often show (for example on MatCreateMPIAIJ manual page) > is where the local process owns 3 rows. > > But consider for example if the local process owns 1000 rows. Right now (as i > can see it) i have two options. Do not allocate memory at all and use > MatSetValues and allocate memory per row. This is very slow as you guys state > on your manual pages. > The other option is to allocate all 1000 rows at a time which means extra > overhead. > > However, what i would like to be able to (as an example), make 2 calls for an > allocation of 200 rows and a second call for an allocation of the remaining > 800 rows. > > I do not know if this is possible as i don't know whether all the local stuff > is stored in on continuous array? > > Anyway, i hope this clarifies my question a little more. > > thank > mat > > > > > On Wednesday 26 March 2008 14:31, Matt Funk wrote: > > Hi, > > > > in order to create a sparse MPIMatrix (with preallocated memory) it is > > necessary to give it the number of nonzero entries per row for the local > > and off-diagonal submatrix. > > > > The way i do things right now is that i need to allocate the entire matrix. > > However, my domain is decomposed into boxes where one or more boxes reside > > on the current processor. > > > > So, i was wondering if it is possible to allocate the matrix memory per > > box. Right now what i do is this: > > 1) iterate over the whole domain (every single box) and extract the info i > > need from each box for memory allocation. > > 2) call MatCreateMPIAIJ to create the matrix/allocate memory. > > 3) iterate over the whole domain again to get the info to insert the values > > into the matrix. > > > > Is it possible to instead do something like: > > 1) get info for values and memory allocation needed per box > > 2) allocate memory for x number of rows in global matrix and insert values > > for them (via MatSetValues) > > 3) start at 1) till every box in the domain is covered. > > > > The problem in 2) is that i can insert the values, but there is no memory > > preallocated for it, so it would be slow. If i could allocate it, i think > > it would solve my problem. > > > > I hope i am making sense. By the way, i cannot use the DA construct because > > i have overlapping values. > > > > thanks > > mat > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From mafunk at nmsu.edu Wed Mar 26 18:30:14 2008 From: mafunk at nmsu.edu (Matt Funk) Date: Wed, 26 Mar 2008 17:30:14 -0600 Subject: matrix memory allocation In-Reply-To: References: <08BD23D9-0CF2-415E-88C0-328DF4D93E0F@columbia.edu> <200803261601.43931.mafunk@nmsu.edu> Message-ID: <200803261730.14903.mafunk@nmsu.edu> Well, it is of course possible that i am looking at this wrong. as to your question what it would save: - case 1) : allocating all local rows on the local process at one time: in order to determine the information for the nonzero entries for the local on and off diagonal matrices i have to iterate over my entire domain and do logic for each node to determine whether my surrounding nodes (for the Laplacian) are on the local processor or not and/or whether they are even in the domain. Because i have to do the same thing again once i get to the point where i insert values into the matrix this is really redundant, but necessary to determine the space to allocate. i would like to eliminate this. -case 2) : do not preallocate anything very slow, and not really an option. so, what it would gain me would be the best compromise between memory allocation performance and logic overhead. I.e. the logic while iterating through the domain layout it executed only once instead of twice. I hope it makes sense... but i guess it is not possible anyway. too bad ... thanks though mat On Wednesday 26 March 2008 17:07, Matthew Knepley wrote: > This is not possible with the current implementation. I am not sure why you > would want to do it. It seems to me to save nothing. Do you have some sort > of performance problem? > > Matt > > On Wed, Mar 26, 2008 at 5:01 PM, Matt Funk wrote: > > Maybe i should clarify a little more. > > I guess what i am looking for is something that lets me split up the > > memory allocation within a given processor. > > > > The example you guys often show (for example on MatCreateMPIAIJ manual > > page) is where the local process owns 3 rows. > > > > But consider for example if the local process owns 1000 rows. Right now > > (as i can see it) i have two options. Do not allocate memory at all and > > use MatSetValues and allocate memory per row. This is very slow as you > > guys state on your manual pages. > > The other option is to allocate all 1000 rows at a time which means > > extra overhead. > > > > However, what i would like to be able to (as an example), make 2 calls > > for an allocation of 200 rows and a second call for an allocation of the > > remaining 800 rows. > > > > I do not know if this is possible as i don't know whether all the local > > stuff is stored in on continuous array? > > > > Anyway, i hope this clarifies my question a little more. > > > > thank > > mat > > > > On Wednesday 26 March 2008 14:31, Matt Funk wrote: > > > Hi, > > > > > > in order to create a sparse MPIMatrix (with preallocated memory) it is > > > necessary to give it the number of nonzero entries per row for the > > > local and off-diagonal submatrix. > > > > > > The way i do things right now is that i need to allocate the entire > > > matrix. However, my domain is decomposed into boxes where one or more > > > boxes reside on the current processor. > > > > > > So, i was wondering if it is possible to allocate the matrix memory > > > per box. Right now what i do is this: > > > 1) iterate over the whole domain (every single box) and extract the > > > info i need from each box for memory allocation. > > > 2) call MatCreateMPIAIJ to create the matrix/allocate memory. > > > 3) iterate over the whole domain again to get the info to insert the > > > values into the matrix. > > > > > > Is it possible to instead do something like: > > > 1) get info for values and memory allocation needed per box > > > 2) allocate memory for x number of rows in global matrix and insert > > > values for them (via MatSetValues) > > > 3) start at 1) till every box in the domain is covered. > > > > > > The problem in 2) is that i can insert the values, but there is no > > > memory preallocated for it, so it would be slow. If i could allocate > > > it, i think it would solve my problem. > > > > > > I hope i am making sense. By the way, i cannot use the DA construct > > > because i have overlapping values. > > > > > > thanks > > > mat From knepley at gmail.com Wed Mar 26 18:38:15 2008 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 26 Mar 2008 18:38:15 -0500 Subject: matrix memory allocation In-Reply-To: <200803261730.14903.mafunk@nmsu.edu> References: <08BD23D9-0CF2-415E-88C0-328DF4D93E0F@columbia.edu> <200803261601.43931.mafunk@nmsu.edu> <200803261730.14903.mafunk@nmsu.edu> Message-ID: On Wed, Mar 26, 2008 at 6:30 PM, Matt Funk wrote: > Well, > > it is of course possible that i am looking at this wrong. > > as to your question what it would save: > > - case 1) : allocating all local rows on the local process at one time: > in order to determine the information for the nonzero entries for the local on > and off diagonal matrices i have to iterate over my entire domain and do > logic for each node to determine whether my surrounding nodes (for the > Laplacian) are on the local processor or not and/or whether they are even in > the domain. Because i have to do the same thing again once i get to the point > where i insert values into the matrix this is really redundant, but necessary > to determine the space to allocate. > i would like to eliminate this. This is what we do. If you measure it, you will find it is a completely negligible fraction of the runtime. I see this as premature (and unnecessary) optimization. If I am wrong here, please send the numbers. Thanks, Matt > -case 2) : do not preallocate anything > very slow, and not really an option. > > so, what it would gain me would be the best compromise between memory > allocation performance and logic overhead. I.e. the logic while iterating > through the domain layout it executed only once instead of twice. > > I hope it makes sense... > > but i guess it is not possible anyway. too bad ... > > thanks though > mat > > > > > > > On Wednesday 26 March 2008 17:07, Matthew Knepley wrote: > > This is not possible with the current implementation. I am not sure why you > > would want to do it. It seems to me to save nothing. Do you have some sort > > of performance problem? > > > > Matt > > > > On Wed, Mar 26, 2008 at 5:01 PM, Matt Funk wrote: > > > Maybe i should clarify a little more. > > > I guess what i am looking for is something that lets me split up the > > > memory allocation within a given processor. > > > > > > The example you guys often show (for example on MatCreateMPIAIJ manual > > > page) is where the local process owns 3 rows. > > > > > > But consider for example if the local process owns 1000 rows. Right now > > > (as i can see it) i have two options. Do not allocate memory at all and > > > use MatSetValues and allocate memory per row. This is very slow as you > > > guys state on your manual pages. > > > The other option is to allocate all 1000 rows at a time which means > > > extra overhead. > > > > > > However, what i would like to be able to (as an example), make 2 calls > > > for an allocation of 200 rows and a second call for an allocation of the > > > remaining 800 rows. > > > > > > I do not know if this is possible as i don't know whether all the local > > > stuff is stored in on continuous array? > > > > > > Anyway, i hope this clarifies my question a little more. > > > > > > thank > > > mat > > > > > > On Wednesday 26 March 2008 14:31, Matt Funk wrote: > > > > Hi, > > > > > > > > in order to create a sparse MPIMatrix (with preallocated memory) it is > > > > necessary to give it the number of nonzero entries per row for the > > > > local and off-diagonal submatrix. > > > > > > > > The way i do things right now is that i need to allocate the entire > > > > matrix. However, my domain is decomposed into boxes where one or more > > > > boxes reside on the current processor. > > > > > > > > So, i was wondering if it is possible to allocate the matrix memory > > > > per box. Right now what i do is this: > > > > 1) iterate over the whole domain (every single box) and extract the > > > > info i need from each box for memory allocation. > > > > 2) call MatCreateMPIAIJ to create the matrix/allocate memory. > > > > 3) iterate over the whole domain again to get the info to insert the > > > > values into the matrix. > > > > > > > > Is it possible to instead do something like: > > > > 1) get info for values and memory allocation needed per box > > > > 2) allocate memory for x number of rows in global matrix and insert > > > > values for them (via MatSetValues) > > > > 3) start at 1) till every box in the domain is covered. > > > > > > > > The problem in 2) is that i can insert the values, but there is no > > > > memory preallocated for it, so it would be slow. If i could allocate > > > > it, i think it would solve my problem. > > > > > > > > I hope i am making sense. By the way, i cannot use the DA construct > > > > because i have overlapping values. > > > > > > > > thanks > > > > mat > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From vijay.m at gmail.com Wed Mar 26 19:10:52 2008 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Wed, 26 Mar 2008 19:10:52 -0500 Subject: problem with -snes_mf_operator option In-Reply-To: References: Message-ID: Matt, Yes, I tested ex5 and it works correctly. But I am not suggesting any errors in PETSc code here. I am kind of puzzled that my unchanged code calling PETSc's SNESSolve works while using -snes_mf option (no allocation for Jac or Prec) and with no option (allocation for both Jac and Prec). And since -snes_mf_operator exists in between, (no allocation for Jac but allocate Prec), atleast in philosophy, I expected it to work without any faults. This is also the reason why I set the Prec = Identity rather than the true Jacobian to eliminate user errors. Anyway I still have not figured out why the NULL reference occurs when PETSc allocates MFFD object in my current code. I was just looking for suggestions here to help me find the error. Vijay On Wed, Mar 26, 2008 at 6:01 PM, Matthew Knepley wrote: > I am not sure what is going on with your code, but I have just tested an example > and it works. I recommend you try > > cd src/snes/examples/tutorials > make ex5 > ./ex5 -snes_monitor -snes_view > ./ex5 -snes_monitor -snes_view -snes_mf > ./ex5 -snes_monitor -snes_view -snes_mf_operator > > Thanks, > > Matt > > > > On Wed, Mar 26, 2008 at 4:14 PM, Vijay S. Mahadevan wrote: > > Hi all, > > > > I have a nonlinear solver test code that can use a user specified > > Jacobian function directly, set using SNESSetJacobian. Now if I use > > -snes_mf option, the solution scheme depends only on the residual and > > the convergence is equivalent to the case when using the real Jacobian > > matrix itself. But if I try the -snes_mf_operator option, with the > > Jacobian=NULL to start with and the preconditioner being correctly > > allocated the memory needed, I end up with a NULL argument exception. > > I also alternately set the preconditioner to Identity matrix in order > > to avoid any problems from user code to trickling in, but to no avail. > > > > Here's the call stack. > > > > [0]PETSC ERROR: Null argument, when expecting valid pointer! > > [0]PETSC ERROR: Null Object: Parameter # 1! > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 7, Fri Oct 26 > > 14:21:35 CDT 2007 HG revision: > > 2e223033ba960114833e1f9713ab393ec78c056f > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: bin/test-dbg on a linux-gnu named grove by vijaysm Wed > > Mar 26 16:06:45 2008 > > [0]PETSC ERROR: Libraries linked from > > /state/partition1/local/petsc-2.3.3-p7/lib/linux-gnu-c-debug > > [0]PETSC ERROR: Configure run at Fri Nov 9 15:02:46 2007 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort > > --with-cxx=g++ --download-f-blas-lapack=1 --download-mpich=1 > > --with-shared=1 > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c > > [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c > > [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c > > [0]PETSC ERROR: MatMult_MFFD() line 294 in src/mat/impls/mffd/mffd.c > > [0]PETSC ERROR: MatMult() line 1632 in src/mat/interface/matrix.c > > [0]PETSC ERROR: PCApplyBAorAB() line 584 in src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: GMREScycle() line 159 in src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: KSPSolve() line 379 in src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: SNES_KSPSolve() line 2578 in src/snes/interface/snes.c > > [0]PETSC ERROR: SNESSolve_LS() line 184 in src/snes/impls/ls/ls.c > > [0]PETSC ERROR: SNESSolve() line 1871 in src/snes/interface/snes.c > > > > From what I can understand from the PETSc code (snes.c, mffd.c), the > > current_u for MFFD context is NULL here. And that is the reason for > > the final exception. But I do not understand how this can be since it > > works perfectly fine with -snes_mf option with no changes in the code. > > > > Please do let me know if you think I should specifically be looking > > for something to tackle this issue. Any help will be appreciated. > > > > Thanks, > > Vijay > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > From knepley at gmail.com Wed Mar 26 19:21:17 2008 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 26 Mar 2008 19:21:17 -0500 Subject: problem with -snes_mf_operator option In-Reply-To: References: Message-ID: My suggestion would be to successively change the working code (ex5) until you get what you want. Matt On Wed, Mar 26, 2008 at 7:10 PM, Vijay S. Mahadevan wrote: > Matt, > > Yes, I tested ex5 and it works correctly. But I am not suggesting any > errors in PETSc code here. I am kind of puzzled that my unchanged code > calling PETSc's SNESSolve works while using -snes_mf option (no > allocation for Jac or Prec) and with no option (allocation for both > Jac and Prec). And since -snes_mf_operator exists in between, (no > allocation for Jac but allocate Prec), atleast in philosophy, I > expected it to work without any faults. This is also the reason why I > set the Prec = Identity rather than the true Jacobian to eliminate > user errors. > > Anyway I still have not figured out why the NULL reference occurs when > PETSc allocates MFFD object in my current code. I was just looking for > suggestions here to help me find the error. > > Vijay > > > > On Wed, Mar 26, 2008 at 6:01 PM, Matthew Knepley wrote: > > I am not sure what is going on with your code, but I have just tested an example > > and it works. I recommend you try > > > > cd src/snes/examples/tutorials > > make ex5 > > ./ex5 -snes_monitor -snes_view > > ./ex5 -snes_monitor -snes_view -snes_mf > > ./ex5 -snes_monitor -snes_view -snes_mf_operator > > > > Thanks, > > > > Matt > > > > > > > > On Wed, Mar 26, 2008 at 4:14 PM, Vijay S. Mahadevan wrote: > > > Hi all, > > > > > > I have a nonlinear solver test code that can use a user specified > > > Jacobian function directly, set using SNESSetJacobian. Now if I use > > > -snes_mf option, the solution scheme depends only on the residual and > > > the convergence is equivalent to the case when using the real Jacobian > > > matrix itself. But if I try the -snes_mf_operator option, with the > > > Jacobian=NULL to start with and the preconditioner being correctly > > > allocated the memory needed, I end up with a NULL argument exception. > > > I also alternately set the preconditioner to Identity matrix in order > > > to avoid any problems from user code to trickling in, but to no avail. > > > > > > Here's the call stack. > > > > > > [0]PETSC ERROR: Null argument, when expecting valid pointer! > > > [0]PETSC ERROR: Null Object: Parameter # 1! > > > [0]PETSC ERROR: > > > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 7, Fri Oct 26 > > > 14:21:35 CDT 2007 HG revision: > > > 2e223033ba960114833e1f9713ab393ec78c056f > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > [0]PETSC ERROR: > > > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: bin/test-dbg on a linux-gnu named grove by vijaysm Wed > > > Mar 26 16:06:45 2008 > > > [0]PETSC ERROR: Libraries linked from > > > /state/partition1/local/petsc-2.3.3-p7/lib/linux-gnu-c-debug > > > [0]PETSC ERROR: Configure run at Fri Nov 9 15:02:46 2007 > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort > > > --with-cxx=g++ --download-f-blas-lapack=1 --download-mpich=1 > > > --with-shared=1 > > > [0]PETSC ERROR: > > > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c > > > [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c > > > [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c > > > [0]PETSC ERROR: MatMult_MFFD() line 294 in src/mat/impls/mffd/mffd.c > > > [0]PETSC ERROR: MatMult() line 1632 in src/mat/interface/matrix.c > > > [0]PETSC ERROR: PCApplyBAorAB() line 584 in src/ksp/pc/interface/precon.c > > > [0]PETSC ERROR: GMREScycle() line 159 in src/ksp/ksp/impls/gmres/gmres.c > > > [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c > > > [0]PETSC ERROR: KSPSolve() line 379 in src/ksp/ksp/interface/itfunc.c > > > [0]PETSC ERROR: SNES_KSPSolve() line 2578 in src/snes/interface/snes.c > > > [0]PETSC ERROR: SNESSolve_LS() line 184 in src/snes/impls/ls/ls.c > > > [0]PETSC ERROR: SNESSolve() line 1871 in src/snes/interface/snes.c > > > > > > From what I can understand from the PETSc code (snes.c, mffd.c), the > > > current_u for MFFD context is NULL here. And that is the reason for > > > the final exception. But I do not understand how this can be since it > > > works perfectly fine with -snes_mf option with no changes in the code. > > > > > > Please do let me know if you think I should specifically be looking > > > for something to tackle this issue. Any help will be appreciated. > > > > > > Thanks, > > > Vijay > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From bsmith at mcs.anl.gov Thu Mar 27 06:55:55 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 27 Mar 2008 06:55:55 -0500 Subject: problem with -snes_mf_operator option In-Reply-To: References: Message-ID: <19196A27-94D9-48AD-A5F5-3EB5D22DBE36@mcs.anl.gov> When you are providing the function with SNESetJacobian() and the A matrix and the B matrix are DIFFERENT they must BOTH have MatAssemblyBegin/End() called on them. For example in snes/examples/tutorials/ex1.c we use if (*jac != *B){ ierr = MatAssemblyBegin(*jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(*jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); } Note: in your original code both matrices are the same but with the -snes_mf_operator option the A matrix is changed to a MFFD matrix so is now different then the B matrix. Please let us know if this resolves the problem, Barry I have added this information to the manual page for SNESSetJacobian() On Mar 26, 2008, at 4:14 PM, Vijay S. Mahadevan wrote: > Hi all, > > I have a nonlinear solver test code that can use a user specified > Jacobian function directly, set using SNESSetJacobian. Now if I use > -snes_mf option, the solution scheme depends only on the residual and > the convergence is equivalent to the case when using the real Jacobian > matrix itself. But if I try the -snes_mf_operator option, with the > Jacobian=NULL to start with and the preconditioner being correctly > allocated the memory needed, I end up with a NULL argument exception. > I also alternately set the preconditioner to Identity matrix in order > to avoid any problems from user code to trickling in, but to no avail. > > Here's the call stack. > > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 7, Fri Oct 26 > 14:21:35 CDT 2007 HG revision: > 2e223033ba960114833e1f9713ab393ec78c056f > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: bin/test-dbg on a linux-gnu named grove by vijaysm Wed > Mar 26 16:06:45 2008 > [0]PETSC ERROR: Libraries linked from > /state/partition1/local/petsc-2.3.3-p7/lib/linux-gnu-c-debug > [0]PETSC ERROR: Configure run at Fri Nov 9 15:02:46 2007 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort > --with-cxx=g++ --download-f-blas-lapack=1 --download-mpich=1 > --with-shared=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/ > gcomm.c > [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c > [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c > [0]PETSC ERROR: MatMult_MFFD() line 294 in src/mat/impls/mffd/mffd.c > [0]PETSC ERROR: MatMult() line 1632 in src/mat/interface/matrix.c > [0]PETSC ERROR: PCApplyBAorAB() line 584 in src/ksp/pc/interface/ > precon.c > [0]PETSC ERROR: GMREScycle() line 159 in src/ksp/ksp/impls/gmres/ > gmres.c > [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/ > gmres.c > [0]PETSC ERROR: KSPSolve() line 379 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: SNES_KSPSolve() line 2578 in src/snes/interface/snes.c > [0]PETSC ERROR: SNESSolve_LS() line 184 in src/snes/impls/ls/ls.c > [0]PETSC ERROR: SNESSolve() line 1871 in src/snes/interface/snes.c > > From what I can understand from the PETSc code (snes.c, mffd.c), the > current_u for MFFD context is NULL here. And that is the reason for > the final exception. But I do not understand how this can be since it > works perfectly fine with -snes_mf option with no changes in the code. > > Please do let me know if you think I should specifically be looking > for something to tackle this issue. Any help will be appreciated. > > Thanks, > Vijay > From vijay.m at gmail.com Thu Mar 27 11:18:37 2008 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Thu, 27 Mar 2008 11:18:37 -0500 Subject: problem with -snes_mf_operator option In-Reply-To: <19196A27-94D9-48AD-A5F5-3EB5D22DBE36@mcs.anl.gov> Message-ID: <002d01c89026$40a325d0$713010ac@neutron> Barry, Perfect. Yes, that did the trick. I spent hours trying to figure out what was going on and never thought of the following, since the Jacobian MFFD matrix for -snes_mf_operator is created and handled by SNESSetup and SNESSolve. But now I understand that for -snes_mf option, the PETSc wrapper function for FormJacobian is called instead of the user function, and hence, it makes a difference in -snes_mf_operator alone. Thanks a lot for the help Barry. Also, appreciate updating the manual pages. Cheers, Vijay > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] > On Behalf Of Barry Smith > Sent: Thursday, March 27, 2008 6:56 AM > To: petsc-users at mcs.anl.gov > Subject: Re: problem with -snes_mf_operator option > > > When you are providing the function with SNESetJacobian() and the > A matrix and the B matrix are DIFFERENT they must BOTH have > MatAssemblyBegin/End() > called on them. For example in snes/examples/tutorials/ex1.c we > use > if (*jac != *B){ > ierr = MatAssemblyBegin(*jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(*jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > } > > Note: in your original code both matrices are the same but with the > -snes_mf_operator option the A matrix is changed to a MFFD matrix so is > now different then the B matrix. > > Please let us know if this resolves the problem, > > Barry > > I have added this information to the manual page for SNESSetJacobian() > > > > On Mar 26, 2008, at 4:14 PM, Vijay S. Mahadevan wrote: > > Hi all, > > > > I have a nonlinear solver test code that can use a user specified > > Jacobian function directly, set using SNESSetJacobian. Now if I use > > -snes_mf option, the solution scheme depends only on the residual and > > the convergence is equivalent to the case when using the real Jacobian > > matrix itself. But if I try the -snes_mf_operator option, with the > > Jacobian=NULL to start with and the preconditioner being correctly > > allocated the memory needed, I end up with a NULL argument exception. > > I also alternately set the preconditioner to Identity matrix in order > > to avoid any problems from user code to trickling in, but to no avail. > > > > Here's the call stack. > > > > [0]PETSC ERROR: Null argument, when expecting valid pointer! > > [0]PETSC ERROR: Null Object: Parameter # 1! > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 7, Fri Oct 26 > > 14:21:35 CDT 2007 HG revision: > > 2e223033ba960114833e1f9713ab393ec78c056f > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: bin/test-dbg on a linux-gnu named grove by vijaysm Wed > > Mar 26 16:06:45 2008 > > [0]PETSC ERROR: Libraries linked from > > /state/partition1/local/petsc-2.3.3-p7/lib/linux-gnu-c-debug > > [0]PETSC ERROR: Configure run at Fri Nov 9 15:02:46 2007 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort > > --with-cxx=g++ --download-f-blas-lapack=1 --download-mpich=1 > > --with-shared=1 > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/ > > gcomm.c > > [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c > > [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c > > [0]PETSC ERROR: MatMult_MFFD() line 294 in src/mat/impls/mffd/mffd.c > > [0]PETSC ERROR: MatMult() line 1632 in src/mat/interface/matrix.c > > [0]PETSC ERROR: PCApplyBAorAB() line 584 in src/ksp/pc/interface/ > > precon.c > > [0]PETSC ERROR: GMREScycle() line 159 in src/ksp/ksp/impls/gmres/ > > gmres.c > > [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/ > > gmres.c > > [0]PETSC ERROR: KSPSolve() line 379 in src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: SNES_KSPSolve() line 2578 in src/snes/interface/snes.c > > [0]PETSC ERROR: SNESSolve_LS() line 184 in src/snes/impls/ls/ls.c > > [0]PETSC ERROR: SNESSolve() line 1871 in src/snes/interface/snes.c > > > > From what I can understand from the PETSc code (snes.c, mffd.c), the > > current_u for MFFD context is NULL here. And that is the reason for > > the final exception. But I do not understand how this can be since it > > works perfectly fine with -snes_mf option with no changes in the code. > > > > Please do let me know if you think I should specifically be looking > > for something to tackle this issue. Any help will be appreciated. > > > > Thanks, > > Vijay > > > > No virus found in this incoming message. > Checked by AVG. > Version: 7.5.519 / Virus Database: 269.22.1/1346 - Release Date: 3/27/2008 > 10:03 AM > No virus found in this outgoing message. Checked by AVG. Version: 7.5.519 / Virus Database: 269.22.1/1346 - Release Date: 3/27/2008 10:03 AM From grs2103 at columbia.edu Thu Mar 27 18:03:01 2008 From: grs2103 at columbia.edu (Gideon Simpson) Date: Thu, 27 Mar 2008 19:03:01 -0400 Subject: ml library Message-ID: Any suggestions on getting the ML library? The new trilinos website doesn't seem to provide access to the older builds. -gideon From knepley at gmail.com Thu Mar 27 18:27:15 2008 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 27 Mar 2008 18:27:15 -0500 Subject: ml library In-Reply-To: References: Message-ID: --download-ml :) Matt On Thu, Mar 27, 2008 at 6:03 PM, Gideon Simpson wrote: > Any suggestions on getting the ML library? The new trilinos website > doesn't seem to provide access to the older builds. > > -gideon > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From bsmith at mcs.anl.gov Sat Mar 29 21:20:34 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 29 Mar 2008 21:20:34 -0500 Subject: matrix memory allocation In-Reply-To: <200803261431.01327.mafunk@nmsu.edu> References: <08BD23D9-0CF2-415E-88C0-328DF4D93E0F@columbia.edu> <200803181600.50369.mafunk@nmsu.edu> <200803261431.01327.mafunk@nmsu.edu> Message-ID: <21D7A4E8-6DE6-4657-BFCB-72A0CA56D049@mcs.anl.gov> Matt, I do not see any specific dis-advantage in your looping over the boxes and determining the preallocation and then looping again to set the values? Surely there is no performance issue? Having support for this in PETSc would be a very dramatic change, perhaps if we understand better the problem perhaps we can come up with an alternative solution to your problem. Barry On Mar 26, 2008, at 3:31 PM, Matt Funk wrote: > Hi, > > in order to create a sparse MPIMatrix (with preallocated memory) it is > necessary to give it the number of nonzero entries per row for the > local and > off-diagonal submatrix. > > The way i do things right now is that i need to allocate the entire > matrix. > However, my domain is decomposed into boxes where one or more boxes > reside on > the current processor. > > So, i was wondering if it is possible to allocate the matrix memory > per box. > Right now what i do is this: > 1) iterate over the whole domain (every single box) and extract the > info i > need from each box for memory allocation. > 2) call MatCreateMPIAIJ to create the matrix/allocate memory. > 3) iterate over the whole domain again to get the info to insert the > values > into the matrix. > > Is it possible to instead do something like: > 1) get info for values and memory allocation needed per box > 2) allocate memory for x number of rows in global matrix and insert > values for > them (via MatSetValues) > 3) start at 1) till every box in the domain is covered. > > The problem in 2) is that i can insert the values, but there is no > memory > preallocated for it, so it would be slow. If i could allocate it, i > think it > would solve my problem. > > I hope i am making sense. By the way, i cannot use the DA construct > because i > have overlapping values. > > thanks > mat > From tim.kroeger at cevis.uni-bremen.de Mon Mar 31 04:08:54 2008 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Mon, 31 Mar 2008 11:08:54 +0200 (CEST) Subject: PETSc installation problem Message-ID: Dear PETSc team, When trying to install PETSc-2.3.3-p11, I get the following output from ./config/configure.py: Traceback (most recent call last): File "./config/configure.py", line 298, in petsc_configure([]) File "./config/configure.py", line 210, in petsc_configure import cPickle ImportError: /usr/lib/python2.5/lib-dynload/cPickle.so: undefined symbol: PyUnicodeUCS4_DecodeRawUnicodeEscape Is there something wrong with my python installation, or what else did I do wrong? Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim.kroeger at mevis.de, tim.kroeger at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, 28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From tim.kroeger at cevis.uni-bremen.de Mon Mar 31 04:26:49 2008 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Mon, 31 Mar 2008 11:26:49 +0200 (CEST) Subject: PETSc installation problem In-Reply-To: References: Message-ID: Dear PETSc team, I found the problem myself now -- it was a wrong setting in LD_LIBRARY_PATH. Sorry for the posting. Tim On Mon, 31 Mar 2008, Tim Kroeger wrote: > Dear PETSc team, > > When trying to install PETSc-2.3.3-p11, I get the following output from > ./config/configure.py: > > Traceback (most recent call last): > File "./config/configure.py", line 298, in > petsc_configure([]) > File "./config/configure.py", line 210, in petsc_configure > import cPickle > ImportError: /usr/lib/python2.5/lib-dynload/cPickle.so: undefined symbol: > PyUnicodeUCS4_DecodeRawUnicodeEscape > > Is there something wrong with my python installation, or what else did I do > wrong? > > Best Regards, > > Tim > > -- > Dr. Tim Kroeger Phone +49-421-218-7710 > tim.kroeger at mevis.de, tim.kroeger at cevis.uni-bremen.de Fax +49-421-218-4236 > > MeVis Research GmbH, Universitaetsallee 29, 28359 Bremen, Germany > > Amtsgericht Bremen HRB 16222 > Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen > -- Dr. Tim Kroeger Phone +49-421-218-7710 tim.kroeger at mevis.de, tim.kroeger at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, 28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From aldo.bonfiglioli at unibas.it Mon Mar 31 10:11:04 2008 From: aldo.bonfiglioli at unibas.it (Aldo Bonfiglioli) Date: Mon, 31 Mar 2008 17:11:04 +0200 Subject: Non repeatability issue In-Reply-To: <81BF6B4F-9199-41AA-B571-20186E2DE43E@mcs.anl.gov> References: <4715D89B.8070005@unibas.it> <47D6B3E1.5070606@unibas.it> <9D646B5A-2D5C-4492-82AF-8E732AF848BD@mcs.anl.gov> <47DFA037.7030707@unibas.it> <81BF6B4F-9199-41AA-B571-20186E2DE43E@mcs.anl.gov> Message-ID: <47F0FF08.9000800@unibas.it> Barry, Matt I am back on the Non repeatability issue with answers to your questions. > 2) did you do the -ksp_rtol 1.e-12 at the same time as the - > vecscatter_reproduce? They > must be done together. The enclosed plot (res_vs_step) shows the mass residual history versus the Newton step counter. For these same runs, the continuation parameter (CFL) shows similar jumps being based upon the SER approach, see plot cfl_vs_its.pdf > When you just fix the CFL and run Newton runs to completion > is it stable? I have restarted the code from an almost fully converged solution using infinite CFL and let it run for 30 Newton steps. The behaviour is much more "reasonable" and the solution remains within the steady state (see plot restarted....) > Then if you ramp up the CFL much more slowly is it stable and Newton > convergence much smoother? I have not tried yet. I know there exist smoother strategies than SER to rump the continuation parameter (I know this http://www.cs.kuleuven.ac.be/publicaties/rapporten/tw/TW304.ps.gz for instance) Aldo -- Dr. Aldo Bonfiglioli Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) Universita' della Basilicata V.le dell'Ateneo lucano, 10 85100 Potenza ITALY tel:+39.0971.205203 fax:+39.0971.205160 -------------- next part -------------- A non-text attachment was scrubbed... Name: res_vs_step.pdf Type: application/pdf Size: 14002 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cfl_vs_its.pdf Type: application/pdf Size: 13982 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: restarted_from_converged_solution.pdf Type: application/pdf Size: 3386 bytes Desc: not available URL: From jinzishuai at yahoo.com Mon Mar 31 16:04:02 2008 From: jinzishuai at yahoo.com (Shi Jin) Date: Mon, 31 Mar 2008 14:04:02 -0700 (PDT) Subject: Mannually specify a diagonal matrix as a preconditioner? Message-ID: <953520.15427.qm@web36206.mail.mud.yahoo.com> Hi there, I am trying to solve a mass matrix linear system by KSPSolve. Right now, I am passing the mass matrix itself (let's call it M) to KSPSetOperators() as the Pmat argument. In order to speed up the convergence, I have constructed the lumped mass matrix (named lumpedM). For a linear finite element, this is simply a diagonal matrix with entries equal to the sum of the row on M. It is a common practice to replace M with lumpedM to have faster convergence without losing the order of accuracy. What I want to do is to still solve the M matrix but use lumpedM to precondition it. This way hopefully the number of iterations would be greatly reduced. In Petsc code, I tried ierr = KSPSetOperators( solMP, M, lumpedM, SAME_PRECONDITIONER); However, instead of giving faster convergence, it actually takes more iterations to convergence than the regular one. Therefore, I wonder if setting lumpedM as Pmat is the correct way to do it. Could you please advice? I think right now lumpedM is taken as the input to compute the preconditioning matrix, using whatever method is specified by -pc_type . What I really want to do is to simply set lumpedM as the precondition matrix, without spending time to compute anything. Thank you very much. Shi -- Shi Jin, PhD ____________________________________________________________________________________ Like movies? Here's a limited-time offer: Blockbuster Total Access for one month at no cost. http://tc.deals.yahoo.com/tc/blockbuster/text4.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jinzishuai at yahoo.com Mon Mar 31 16:11:32 2008 From: jinzishuai at yahoo.com (Shi Jin) Date: Mon, 31 Mar 2008 14:11:32 -0700 (PDT) Subject: Create a pure diagonal matrix Message-ID: <107887.63179.qm@web36204.mail.mud.yahoo.com> Hi, I have a related question to the one above: what is the best way to create a purely diagonal matrix? I am using ierr = MatCreateMPIAIJ(comm,localM,localN,PETSC_DETERMINE,PETSC_DETERMINE,1, PETSC_NULL, 0, PETSC_NULL, &lumpedMP);CHKERRQ(ierr); But I would think there might be a even better (faster/less memory) way? I thought about the BDiag Matrix. I tried ierr=MatCreateMPIBDiag(comm,localM,globalSize,globalSize,PETSC_NULL,1, PETSC_NULL,PETSC_NULL,&lumpedM);CHKERRQ(ierr); but not sure if it is done correctly. I am a little suspicious about those pointers I set as NULL. If the above two are both correct, is there a better one? Thank you very much. -- Shi Jin, PhD ____________________________________________________________________________________ No Cost - Get a month of Blockbuster Total Access now. Sweet deal for Yahoo! users and friends. http://tc.deals.yahoo.com/tc/blockbuster/text1.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 31 16:26:37 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 31 Mar 2008 16:26:37 -0500 Subject: Mannually specify a diagonal matrix as a preconditioner? In-Reply-To: <953520.15427.qm@web36206.mail.mud.yahoo.com> References: <953520.15427.qm@web36206.mail.mud.yahoo.com> Message-ID: You do not need another matrix to do lumping. Just give the mass matrix in both arguments, and then use -pc_type jacobi -pc_jacobi_rowsum Matt On Mon, Mar 31, 2008 at 4:04 PM, Shi Jin wrote: > > Hi there, > > I am trying to solve a mass matrix linear system by KSPSolve. Right now, I > am passing the mass matrix itself (let's call it M) to KSPSetOperators() as > the Pmat argument. In order to speed up the convergence, I have constructed > the lumped mass matrix (named lumpedM). For a linear finite element, this is > simply a diagonal matrix with entries equal to the sum of the row on M. It > is a common practice to replace M with lumpedM to have faster convergence > without losing the order of accuracy. > > What I want to do is to still solve the M matrix but use lumpedM to > precondition it. This way hopefully the number of iterations would be > greatly reduced. In Petsc code, I tried > > ierr = KSPSetOperators( solMP, M, lumpedM, SAME_PRECONDITIONER); > > However, instead of giving faster convergence, it actually takes more > iterations to convergence than the regular one. Therefore, I wonder if > setting lumpedM as Pmat is the correct way to do it. Could you please > advice? I think right now lumpedM is taken as the input to compute the > preconditioning matrix, using whatever method is specified by -pc_type . > What I really want to do is to simply set lumpedM as the precondition > matrix, without spending time to compute anything. > Thank you very much. > > Shi > > > > > > -- > Shi Jin, PhD > > > ________________________________ > Like movies? Here's a limited-time offer: Blockbuster Total Access for one > month at no cost. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From knepley at gmail.com Mon Mar 31 16:30:50 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 31 Mar 2008 16:30:50 -0500 Subject: Create a pure diagonal matrix In-Reply-To: <107887.63179.qm@web36204.mail.mud.yahoo.com> References: <107887.63179.qm@web36204.mail.mud.yahoo.com> Message-ID: Using a combination of Jacobi and KSP scaling, you almost never need a diagonal matrix in solves. However, if you really do, I would just use MatDiagonScale(): http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatDiagonalScale.html#MatDiagonalScale and if you really need to, you can wrap that operation in a MatShell. Matt On Mon, Mar 31, 2008 at 4:11 PM, Shi Jin wrote: > > Hi, > > I have a related question to the one above: what is the best way to create a > purely diagonal matrix? > I am using > ierr = MatCreateMPIAIJ(comm,localM,localN,PETSC_DETERMINE,PETSC_DETERMINE,1, > PETSC_NULL, 0, PETSC_NULL, &lumpedMP);CHKERRQ(ierr); > But I would think there might be a even better (faster/less memory) way? > > I thought about the BDiag Matrix. I tried > ierr=MatCreateMPIBDiag(comm,localM,globalSize,globalSize,PETSC_NULL,1, > PETSC_NULL,PETSC_NULL,&lumpedM);CHKERRQ(ierr); > but not sure if it is done correctly. I am a little suspicious about those > pointers I set as NULL. > > If the above two are both correct, is there a better one? > > Thank you very much. > -- > Shi Jin, PhD > > > ________________________________ > Special deal for Yahoo! users & friends - No Cost. Get a month of > Blockbuster Total Access now -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From jinzishuai at yahoo.com Mon Mar 31 17:05:13 2008 From: jinzishuai at yahoo.com (Shi Jin) Date: Mon, 31 Mar 2008 15:05:13 -0700 (PDT) Subject: Mannually specify a diagonal matrix as a preconditioner? Message-ID: <383333.62452.qm@web36201.mail.mud.yahoo.com> You do not need another matrix to do lumping. Just give the mass matrix in both arguments, and then use -pc_type jacobi -pc_jacobi_rowsum Matt Thank you Matt. However, I want to do more than a row sum, which is perfectly fine for first order elements. For higher order elements, I have to use a special quadrature rule to construct the lumped diagonal matrix. With this in mind, could you please answer my two questions again? Thank you very much. Shi On Mon, Mar 31, 2008 at 4:04 PM, Shi Jin wrote: > > Hi there, > > I am trying to solve a mass matrix linear system by KSPSolve. Right now, I > am passing the mass matrix itself (let's call it M) to KSPSetOperators() as > the Pmat argument. In order to speed up the convergence, I have constructed > the lumped mass matrix (named lumpedM). For a linear finite element, this is > simply a diagonal matrix with entries equal to the sum of the row on M. It > is a common practice to replace M with lumpedM to have faster convergence > without losing the order of accuracy. > > What I want to do is to still solve the M matrix but use lumpedM to > precondition it. This way hopefully the number of iterations would be > greatly reduced. In Petsc code, I tried > > ierr = KSPSetOperators( solMP, M, lumpedM, SAME_PRECONDITIONER); > > However, instead of giving faster convergence, it actually takes more > iterations to convergence than the regular one. Therefore, I wonder if > setting lumpedM as Pmat is the correct way to do it. Could you please > advice? I think right now lumpedM is taken as the input to compute the > preconditioning matrix, using whatever method is specified by -pc_type . > What I really want to do is to simply set lumpedM as the precondition > matrix, without spending time to compute anything. > Thank you very much. > > Shi > > > > > > -- > Shi Jin, PhD > > > ________________________________ > Like movies? Here's a limited-time offer: Blockbuster Total Access for one > month at no cost. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener ____________________________________________________________________________________ Like movies? Here's a limited-time offer: Blockbuster Total Access for one month at no cost. http://tc.deals.yahoo.com/tc/blockbuster/text4.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 31 17:29:30 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 31 Mar 2008 17:29:30 -0500 Subject: Mannually specify a diagonal matrix as a preconditioner? In-Reply-To: <383333.62452.qm@web36201.mail.mud.yahoo.com> References: <383333.62452.qm@web36201.mail.mud.yahoo.com> Message-ID: On Mon, Mar 31, 2008 at 5:05 PM, Shi Jin wrote: > > > You do not need another matrix to do lumping. Just give the mass matrix in > both arguments, and then use > > -pc_type jacobi -pc_jacobi_rowsum > > Matt > > Thank you Matt. > However, I want to do more than a row sum, which is perfectly fine for first > order elements. For higher order elements, I have to use a special > quadrature rule to construct the lumped diagonal matrix. > > With this in mind, could you please answer my two questions again? Thank you > very much. Then my answer is the same as my second answer. To first order, I think the time is completely negligible, so just use an AIJ matrix. If, after measuring the performance, you are unhappy, then put the diagonal matrix in a vector and wrap MatDiagonalScale() in a MatShell. Matt > Shi > > > On Mon, Mar 31, 2008 at 4:04 PM, Shi Jin wrote: > > > > Hi there, > > > > I am trying to solve a mass matrix linear system by KSPSolve. Right now, I > > am passing the mass matrix itself (let's call it M) to KSPSetOperators() > as > > the Pmat argument. In order to speed up the convergence, I have > constructed > > the lumped mass matrix (named lumpedM). For a linear finite element, this > is > > simply a diagonal matrix with entries equal to the sum of the row on M. It > > is a common practice to replace M with lumpedM to have faster convergence > > without losing the order of accuracy. > > > > What I want to do is to still solve the M matrix but use lumpedM to > > precondition it. This way hopefully the number of iterations would be > > greatly reduced. In Petsc code, I tried > > > > ierr = KSPSetOperators( solMP, M, lumpedM, SAME_PRECONDITIONER); > > > > However, instead of giving faster convergence, it actually takes more > > iterations to convergence than the regular one. Therefore, I wonder if > > setting lumpedM as Pmat is the correct way to do it. Could you please > > advice? I think right now lumpedM is taken as the input to compute the > > preconditioning matrix, using whatever method is specified by -pc_type . > > What I really want to do is to simply set lumpedM as the precondition > > matrix, without spending time to compute anything. > > Thank you very much. > > > > Shi > > > > > > > > > > > > -- > > Shi Jin, PhD > > > > > > ________________________________ > > Like movies? Here's a limited-time offer: Blockbuster Total Access for one > > month at no cost. > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > > > ________________________________ > No Cost - Get a month of Blockbuster Total Access now. Sweet deal for Yahoo! > users and friends. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From jinzishuai at yahoo.com Mon Mar 31 17:44:32 2008 From: jinzishuai at yahoo.com (Shi Jin) Date: Mon, 31 Mar 2008 15:44:32 -0700 (PDT) Subject: Mannually specify a diagonal matrix as a preconditioner? Message-ID: <397890.22785.qm@web36207.mail.mud.yahoo.com> > Then my answer is the same as my second answer. To first order, I think the time > is completely negligible, so just use an AIJ matrix. If, after measuring the > performance, you are unhappy, then put the diagonal matrix in a vector and > wrap MatDiagonalScale() in a MatShell. > > Matt Thank you. I think I am OK with AIJ matrix and it is working fine. But I still want to ask the first question about specifying a preconditioning matrix. Right now I am testing a first order element case since I know what it should look like and it should improve performance. But when I do > > > ierr = KSPSetOperators( solMP, M, lumpedM, SAME_PRECONDITIONER); The solver takes more iterations to converge. It this the correct way to specify the lumped matrix? Thank you very much. Shi > > > > > > However, instead of giving faster convergence, it actually takes more > > > iterations to convergence than the regular one. Therefore, I wonder if > > > setting lumpedM as Pmat is the correct way to do it. Could you please > > > advice? I think right now lumpedM is taken as the input to compute the > > > preconditioning matrix, using whatever method is specified by -pc_type . > > > What I really want to do is to simply set lumpedM as the precondition > > > matrix, without spending time to compute anything. > > > Thank you very much. ____________________________________________________________________________________ Like movies? Here's a limited-time offer: Blockbuster Total Access for one month at no cost. http://tc.deals.yahoo.com/tc/blockbuster/text4.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 31 17:51:36 2008 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 31 Mar 2008 17:51:36 -0500 Subject: Mannually specify a diagonal matrix as a preconditioner? In-Reply-To: <397890.22785.qm@web36207.mail.mud.yahoo.com> References: <397890.22785.qm@web36207.mail.mud.yahoo.com> Message-ID: On Mon, Mar 31, 2008 at 5:44 PM, Shi Jin wrote: > > > > Then my answer is the same as my second answer. To first order, I think > the time > > is completely negligible, so just use an AIJ matrix. If, after measuring > the > > performance, you are unhappy, then put the diagonal matrix in a vector and > > wrap MatDiagonalScale() in a MatShell. > > > > Matt > Thank you. I think I am OK with AIJ matrix and it is working fine. > But I still want to ask the first question about specifying a > preconditioning matrix. Right now I am testing a first order element case > since I know what it should look like and it should improve performance. > But when I do > > > > > ierr = KSPSetOperators( solMP, M, lumpedM, SAME_PRECONDITIONER); > > The solver takes more iterations to converge. It this the correct way to > specify the lumped matrix? Has nothing to do with lumping. However, I am worried about the SAME_PRECONDITIONER flag. Is that what you really mean? This will not recalculate the preconditioner (meaning reinvert the diagonal) when the system changes. Also, what PC do you actually use? It should be Jacobi. Matt > Thank you very much. > > Shi > > > > > > > > > However, instead of giving faster convergence, it actually takes more > > > > iterations to convergence than the regular one. Therefore, I wonder if > > > > setting lumpedM as Pmat is the correct way to do it. Could you please > > > > advice? I think right now lumpedM is taken as the input to compute the > > > > preconditioning matrix, using whatever method is specified by -pc_type > . > > > > What I really want to do is to simply set lumpedM as the precondition > > > > matrix, without spending time to compute anything. > > > > Thank you very much. > > > ________________________________ > OMG, Sweet deal for Yahoo! users/friends: Get A Month of Blockbuster Total > Access, No Cost. W00t -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener